Minsky 위에서의 PyTorch 설치, 그리고 MNIST 수행

IBM에서 제공되는 주요 오픈소스 기반 Deep Learning framework toolkit인 PowerAI에는 아직 PyTorch가 포함되어 있지 않습니다.  그러나 요즘 python의 인기를 타고 PyTorch를 사용하는 사례가 점점 늘고 있습니다.

ppc64le 아키텍처 기반의 IBM Minsky 서버에서는 그렇다면 아직 PyTorch를 사용 못하는 것인가 ?  아닙니다.  오픈소스 좋은 것이 무엇이겠습니까 ?  그냥 source에서 빌드하셔서 사용하시면 됩니다.   여기서는 그 과정을 한번 보겠습니다.   제가 가난하여 Minsky가 없는 관계로, Kolon Benit에서 잠깐 빌려주신 Firestone 서버 (POWER8 + K80)에서 빌드하고 테스트했습니다.   여기서는 Ubuntu 16.04.03에 CUDA 8.0.61을 썼습니다.

root@ubuntu:/data/examples/mnist# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

root@ubuntu:/data/examples/mnist# dpkg -l | grep cuda

ii  cuda                                    8.0.61-1                                   ppc64el      CUDA meta-package

ii  cuda-8-0                                8.0.61-1                                   ppc64el      CUDA 8.0 meta-package

먼저 아래 URL에 따라 anaconda를 설치합니다.

http://hwengineer.blogspot.kr/2017/09/minsky-anaconda-installer-full-package.html

다음으로 github에서 source download 합니다.

root@ubuntu:/data# git clone https://github.com/pytorch/pytorch.git
Cloning into 'pytorch'...
remote: Counting objects: 40225, done.
remote: Compressing objects: 100% (39/39), done.
remote: Total 40225 (delta 33), reused 30 (delta 24), pack-reused 40162
Receiving objects: 100% (40225/40225), 15.52 MiB | 6.41 MiB/s, done.
Resolving deltas: 100% (30571/30571), done.
Checking connectivity... done.

root@ubuntu:/data# cd pytorch

Anaconda로부터 conda가 제대로 설치되었는지 확인하고, 그에 따라 CMAKE_PREFIX_PATH를 설정합니다.

root@ubuntu:/data/pytorch# which conda
/opt/anaconda2/bin/conda

root@ubuntu:/data/pytorch# export CMAKE_PREFIX_PATH=/opt/anaconda2

이어서 numpy와 pyyaml 등 PyTorch에 필요한 python package들을 conda 명령으로 설치합니다.  이때 원래의 build instruction (https://github.com/pytorch/pytorch#from-source)에서는 intel에만 있는 mkl도 conda로 설치하라고 나옵니다만, ppc64le에서는 그 대신 openblas를 설치하시면 됩니다.

root@ubuntu:/data/pytorch# conda install numpy pyyaml setuptools cmake cffi openblas
...
Package plan for installation in environment /opt/anaconda2:

The following NEW packages will be INSTALLED:

    bzip2:        1.0.6-3
    certifi:      2016.2.28-py27_0
    cmake:        3.6.3-0

The following packages will be UPDATED:

    anaconda:     4.4.0-np112py27_0  --> custom-py27_0
    astropy:      1.3.2-np112py27_0  --> 2.0.1-np113py27_1
    bottleneck:   1.2.1-np112py27_0  --> 1.2.1-np113py27_1
    conda:        4.3.21-py27_0      --> 4.3.27-py27_0
    h5py:         2.7.0-np112py27_0  --> 2.7.0-np113py27_1
    matplotlib:   2.0.2-np112py27_0  --> 2.0.2-np113py27_0
    numexpr:      2.6.2-np112py27_0  --> 2.6.2-np113py27_1
    numpy:        1.12.1-py27_0      --> 1.13.1-py27_1
    pandas:       0.20.1-np112py27_0 --> 0.20.3-py27_1
    pytables:     3.2.2-np112py27_4  --> 3.4.2-np113py27_0
    pywavelets:   0.5.2-np112py27_0  --> 0.5.2-np113py27_1
    scikit-image: 0.13.0-np112py27_0 --> 0.13.0-np113py27_0
    scikit-learn: 0.18.1-np112py27_1 --> 0.19.0-np113py27_1
    scipy:        0.19.0-np112py27_0 --> 0.19.1-np113py27_1
    setuptools:   27.2.0-py27_0      --> 36.4.0-py27_1
    statsmodels:  0.8.0-np112py27_0  --> 0.8.0-np113py27_1

Proceed ([y]/n)? y

bzip2-1.0.6-3. 100% |################################| Time: 0:00:00  10.23 MB/s
anaconda-custo 100% |################################| Time: 0:00:00  15.66 MB/s
certifi-2016.2 100% |################################| Time: 0:00:01 147.40 kB/s
cmake-3.6.3-0. 100% |################################| Time: 0:00:36 225.32 kB/s
numpy-1.13.1-p 100% |################################| Time: 0:00:04   1.68 MB/s
bottleneck-1.2 100% |################################| Time: 0:00:01 224.70 kB/s
h5py-2.7.0-np1 100% |################################| Time: 0:00:02   1.09 MB/s
numexpr-2.6.2- 100% |################################| Time: 0:00:01 288.12 kB/s
pywavelets-0.5 100% |################################| Time: 0:00:05   1.08 MB/s
scipy-0.19.1-n 100% |################################| Time: 0:01:25 459.82 kB/s
setuptools-36. 100% |################################| Time: 0:00:01 347.78 kB/s
pandas-0.20.3- 100% |################################| Time: 0:00:56 407.34 kB/s
pytables-3.4.2 100% |################################| Time: 0:00:41 168.51 kB/s
scikit-learn-0 100% |################################| Time: 0:01:19 158.86 kB/s
astropy-2.0.1- 100% |################################| Time: 0:00:15 644.67 kB/s
statsmodels-0. 100% |################################| Time: 0:00:44 178.04 kB/s
conda-4.3.27-p 100% |################################| Time: 0:00:00  44.12 MB/s
matplotlib-2.0 100% |################################| Time: 0:00:04   2.51 MB/s
scikit-image-0 100% |################################| Time: 0:02:18 245.94 kB/s


  - https://repo.continuum.io/pkgs/free/noarch
  - https://repo.continuum.io/pkgs/r/linux-ppc64le
  - https://repo.continuum.io/pkgs/r/noarch
  - https://repo.continuum.io/pkgs/pro/linux-ppc64le
  - https://repo.continuum.io/pkgs/pro/noarch


이제 pytorch를 설치할 차례입니다.

root@ubuntu:/data/pytorch# python setup.py install
Could not find /data/pytorch/torch/lib/gloo/CMakeLists.txt
Did you run 'git submodule update --init'?

억, git clone할 때 --recursive를 안 붙여줬기 때문에 이런 error를 겪나 봅니다.  시키는 대로 git submodule 명령을 수행하시면 이 error는 안 생깁니다.

root@ubuntu:/data/pytorch# git submodule update --init
Submodule 'torch/lib/gloo' (https://github.com/facebookincubator/gloo) registered for path 'torch/lib/gloo'
Submodule 'torch/lib/nanopb' (https://github.com/nanopb/nanopb.git) registered for path 'torch/lib/nanopb'
Submodule 'torch/lib/pybind11' (https://github.com/pybind/pybind11) registered for path 'torch/lib/pybind11'
Cloning into 'torch/lib/gloo'...
remote: Counting objects: 1922, done.
remote: Compressing objects: 100% (61/61), done.
remote: Total 1922 (delta 28), reused 64 (delta 24), pack-reused 1837
Receiving objects: 100% (1922/1922), 567.77 KiB | 0 bytes/s, done.
Resolving deltas: 100% (1422/1422), done.
Checking connectivity... done.
Submodule path 'torch/lib/gloo': checked out '7fd607e2852c910f0f1320d2aaa92f1da2291109'
Cloning into 'torch/lib/nanopb'...
remote: Counting objects: 4384, done.
...
Resolving deltas: 100% (6335/6335), done.
Checking connectivity... done.
Submodule path 'torch/lib/pybind11': checked out '9f6a636e547fc70a02fa48436449aad67080698f'


이제 다시 pytorch를 설치합니다.   보시다시피 설치 메시지에서 x86 아키텍처에만 있는 SSE2 extension이 없다든가, mkl_intel이 없다는 등의 경고 메시지가 많이 나옵니다만 대범하게 무시하십시요.

root@ubuntu:/data/pytorch# python setup.py install
running install
running build_deps
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Checking if C linker supports --verbose
-- Checking if C linker supports --verbose - yes
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Checking if CXX linker supports --verbose
-- Checking if CXX linker supports --verbose - yes
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -fopenmp
-- Compiling with OpenMP support
-- Could not find hardware support for NEON on this machine.
-- No OMAP3 processor on this machine.
-- No OMAP4 processor on this machine.
-- Looking for cpuid.h
...
-- Performing Test C_HAS_SSE1_1
-- Performing Test C_HAS_SSE1_1 - Failed
-- Performing Test C_HAS_SSE1_2
-- Performing Test C_HAS_SSE1_2 - Failed
-- Performing Test C_HAS_SSE1_3
...
-- Checking for [mkl_gf_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_gf_lp64: not found
-- Checking for [mkl_gf_lp64 - mkl_intel_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_gf_lp64: not found
...
-- MKL library not found
-- Checking for [openblas]
--   Library openblas: /opt/anaconda2/lib/libopenblas.so
-- Looking for sgemm_
-- Looking for sgemm_ - found
-- Performing Test BLAS_F2C_DOUBLE_WORKS
-- Performing Test BLAS_F2C_DOUBLE_WORKS - Success
-- Performing Test BLAS_F2C_FLOAT_WORKS
-- Performing Test BLAS_F2C_FLOAT_WORKS - Success
-- Performing Test BLAS_USE_CBLAS_DOT
-- Performing Test BLAS_USE_CBLAS_DOT - Success
-- Found a library with BLAS API (open).
-- Looking for cheev_
-- Looking for cheev_ - found
-- Found a library with LAPACK API. (open)
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - found
...
Scanning dependencies of target TH
[ 12%] Building C object CMakeFiles/TH.dir/THSize.c.o
[ 12%] Building C object CMakeFiles/TH.dir/THHalf.c.o
[ 18%] Building C object CMakeFiles/TH.dir/THGeneral.c.o
[ 25%] Building C object CMakeFiles/TH.dir/THAllocator.c.o
[ 31%] Building C object CMakeFiles/TH.dir/THStorage.c.o
[ 37%] Building C object CMakeFiles/TH.dir/THRandom.c.o
[ 43%] Building C object CMakeFiles/TH.dir/THFile.c.o
[ 50%] Building C object CMakeFiles/TH.dir/THTensor.c.o
[ 56%] Building C object CMakeFiles/TH.dir/THDiskFile.c.o
[ 62%] Building C object CMakeFiles/TH.dir/THMemoryFile.c.o
[ 75%] Building C object CMakeFiles/TH.dir/THLogAdd.c.o
[ 75%] Building C object CMakeFiles/TH.dir/THLapack.c.o
[ 81%] Building C object CMakeFiles/TH.dir/THBlas.c.o
[ 87%] Building C object CMakeFiles/TH.dir/THVector.c.o
[ 93%] Building C object CMakeFiles/TH.dir/THAtomic.c.o
...
/data/pytorch/torch/lib/tmp_install/include/THC/THCNumerics.cuh(38): warning: integer conversion resulted in a change of sign
...
Compiling src/reduce_scatter.cu               > /data/pytorch/torch/lib/build/nccl/obj/reduce_scatter.o
ptxas warning : Too big maxrregcount value specified 96, will be ignored
ptxas warning : Too big maxrregcount value specified 96, will be ignored
...
byte-compiling /opt/anaconda2/lib/python2.7/site-packages/torch/utils/data/__init__.py to __init__.pyc
byte-compiling /opt/anaconda2/lib/python2.7/site-packages/torch/utils/data/dataset.py to dataset.pyc
byte-compiling /opt/anaconda2/lib/python2.7/site-packages/torch/utils/data/distributed.py to distributed.pyc
byte-compiling /opt/anaconda2/lib/python2.7/site-packages/torch/_utils.py to _utils.pyc
running install_egg_info
running egg_info
creating torch.egg-info
writing requirements to torch.egg-info/requires.txt
writing torch.egg-info/PKG-INFO
writing top-level names to torch.egg-info/top_level.txt
writing dependency_links to torch.egg-info/dependency_links.txt
writing manifest file 'torch.egg-info/SOURCES.txt'
reading manifest file 'torch.egg-info/SOURCES.txt'
writing manifest file 'torch.egg-info/SOURCES.txt'
Copying torch.egg-info to /opt/anaconda2/lib/python2.7/site-packages/torch-0.2.0+efe91fb-py2.7.egg-info
running install_scripts

보시다시피 warning message들이 나왔을 뿐 결국 잘 compile 됩니다.   이제 python에서 torch를 import하여 간단한 test를 몇가지 해보겠습니다.   저는 프로그래밍에 젬병인지라 그냥 pytorch 홈페이지의 튜토리얼 (http://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#sphx-glr-beginner-blitz-tensor-tutorial-py) 일부를 그대로 수행해보았습니다.

root@ubuntu:/data/pytorch# python
Python 2.7.13 |Anaconda custom (64-bit)| (default, Mar 16 2017, 18:34:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from __future__ import print_function
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "torch/__init__.py", line 53, in <module>
    from torch._C import *
ImportError: No module named _C
>>>

시작하자마자 이게 웬 error인가요 ?  이건 다소 우스운 error입니다.  (https://github.com/pytorch/pytorch/issues/7)  pytorch의 설치 directory (여기서는 /data/pytorch)에는 torch라는 이름의 directory가 있는데, 이 directory에서 import torch를 하면 이 directory name과 중복되어 error가 나는 것입니다.  그냥 다른 directory로 옮기셔서 python을 수행하시면 이 error는 나지 않습니다. 

root@ubuntu:/data/pytorch# ls
build            DLConvertor.h  LICENSE           test            tox.ini
cmake            dlpack.h       README.md         tools
CONTRIBUTING.md  Dockerfile     requirements.txt  torch
DLConvertor.cpp  docs           setup.py          torch.egg-info

이제 다른 아무 directory로 옮겨가서 거기서 python을 수행하겠습니다.

root@ubuntu:/data/pytorch# cd

root@ubuntu:~# python
Python 2.7.13 |Anaconda custom (64-bit)| (default, Mar 16 2017, 18:34:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from __future__ import print_function
>>> import torch

예, 이번에는 이상없이 import됩니다.  Torch에서 사용하는 Tensor 및 rand 함수를 써보겠습니다.

>>> x = torch.Tensor(5, 3)
>>> print(x)

 0.0000e+00  0.0000e+00  0.0000e+00
 0.0000e+00  9.1957e+35  2.2955e-41
 9.2701e+35  2.2955e-41  1.1673e+36
 2.2955e-41  9.2913e+35  2.2955e-41
 0.0000e+00  0.0000e+00  0.0000e+00
[torch.FloatTensor of size 5x3]

>>> x = torch.rand(5, 3)
>>> print(x)

 0.7949  0.8651  0.0330
 0.5913  0.2181  0.9074
 0.7759  0.0349  0.9361
 0.3618  0.9953  0.8532
 0.2193  0.1514  0.6486
[torch.FloatTensor of size 5x3]

>>> print(x.size())
(5L, 3L)

>>> y = torch.rand(5, 3)
>>> print(x + y)

 0.8520  1.0601  0.7188
 0.7161  0.3146  1.0981
 1.4604  1.0081  0.9696
 1.1450  1.7239  1.2189
 0.2487  0.9476  1.6199
[torch.FloatTensor of size 5x3]

>>> print(torch.add(x, y))

 0.8520  1.0601  0.7188
 0.7161  0.3146  1.0981
 1.4604  1.0081  0.9696
 1.1450  1.7239  1.2189
 0.2487  0.9476  1.6199
[torch.FloatTensor of size 5x3]

>>> result = torch.Tensor(5, 3)
>>> torch.add(x, y, out=result)

 0.8520  1.0601  0.7188
 0.7161  0.3146  1.0981
 1.4604  1.0081  0.9696
 1.1450  1.7239  1.2189
 0.2487  0.9476  1.6199
[torch.FloatTensor of size 5x3]

>>> y.add_(x)

 0.8520  1.0601  0.7188
 0.7161  0.3146  1.0981
 1.4604  1.0081  0.9696
 1.1450  1.7239  1.2189
 0.2487  0.9476  1.6199
[torch.FloatTensor of size 5x3]

다 잘 됩니다.  Torch Tensor를 numpy Array로 전환하는 것도 해보겠습니다.

>>> a = torch.ones(5)
>>> print(a)

 1
 1
 1
 1
 1
[torch.FloatTensor of size 5]

>>> b = a.numpy()
>>> print(b)
[ 1.  1.  1.  1.  1.]

>>> a.add_(1)

 2
 2
 2
 2
 2
[torch.FloatTensor of size 5]

>>> print(b)
[ 2.  2.  2.  2.  2.]

다 잘 됩니다.   이제 GPU를 써서 pytorch를 구동해보겠습니다.

>>> if torch.cuda.is_available():
...     x = x.cuda()
...     y = y.cuda()
...     x + y
...

 1.6470  1.9252  0.7518
 1.3074  0.5327  2.0054
 2.2363  1.0430  1.9057
 1.5068  2.7193  2.0721
 0.4680  1.0990  2.2685
[torch.cuda.FloatTensor of size 5x3 (GPU 0)]

역시 잘 됩니다.  저 위의 .cuda()가 구동되는 순간 아래와 같이 GPU를 이미 python이 점거하는 것을 보실 수 있습니다.

Tue Oct 10 00:13:54 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 361.119                Driver Version: 361.119                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 0000:03:00.0     Off |                    0 |
| N/A   44C    P0    58W / 149W |    200MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 0000:04:00.0     Off |                    0 |
| N/A   30C    P8    31W / 149W |      2MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | 0020:03:00.0     Off |                    0 |
| N/A   35C    P8    26W / 149W |      2MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           Off  | 0020:04:00.0     Off |                    0 |
| N/A   30C    P8    29W / 149W |      2MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     40655    C   python                                         198MiB |
+-----------------------------------------------------------------------------+


이제 가장 간단한 deep learning 예제인 MNIST를 pytorch로 수행해보겠습니다.   먼저, 아래와 같이 pytorch에서 제공하는 example source code를 download 받습니다.

root@ubuntu:/data# git clone --recursive https://github.com/pytorch/examples.git
Cloning into 'examples'...
remote: Counting objects: 1461, done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 1461 (delta 1), reused 2 (delta 0), pack-reused 1455
Receiving objects: 100% (1461/1461), 29.95 MiB | 6.56 MiB/s, done.
Resolving deltas: 100% (767/767), done.
Checking connectivity... done.

root@ubuntu:/data# cd examples/mnist

다운받은 requirements.txt 속에는 아래와 같이 torch와 torchvision의 2줄이 들어있습니다.

root@ubuntu:/data/examples/mnist# vi requirements.txt
torch
torchvision

여기에 대해 pip로 install하면 다음과 같이 torchvision이 새로 설치됩니다.

root@ubuntu:/data/examples/mnist# pip install -r requirements.txt
Requirement already satisfied: torch in /opt/anaconda2/lib/python2.7/site-packages (from -r requirements.txt (line 1))
Collecting torchvision (from -r requirements.txt (line 2))
  Downloading torchvision-0.1.9-py2.py3-none-any.whl (43kB)
    100% |████████████████████████████████| 51kB 423kB/s
Requirement already satisfied: pyyaml in /opt/anaconda2/lib/python2.7/site-packages (from torch->-r requirements.txt (line 1))
Requirement already satisfied: numpy in /opt/anaconda2/lib/python2.7/site-packages (from torch->-r requirements.txt (line 1))
Requirement already satisfied: pillow in /opt/anaconda2/lib/python2.7/site-packages (from torchvision->-r requirements.txt (line 2))
Requirement already satisfied: six in /opt/anaconda2/lib/python2.7/site-packages (from torchvision->-r requirements.txt (line 2))
Requirement already satisfied: olefile in /opt/anaconda2/lib/python2.7/site-packages (from pillow->torchvision->-r requirements.txt (line 2))
Installing collected packages: torchvision
Successfully installed torchvision-0.1.9

이제 남은 것은 mnist directory에 들어있는 main.py를 수행하기만 하면 됩니다.

root@ubuntu:/data/examples/mnist# python main.py
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
Processing...
...
Train Epoch: 10 [55040/60000 (92%)]     Loss: 0.148484
Train Epoch: 10 [55680/60000 (93%)]     Loss: 0.215679
Train Epoch: 10 [56320/60000 (94%)]     Loss: 0.122693
Train Epoch: 10 [56960/60000 (95%)]     Loss: 0.120907
Train Epoch: 10 [57600/60000 (96%)]     Loss: 0.153347
Train Epoch: 10 [58240/60000 (97%)]     Loss: 0.100982
Train Epoch: 10 [58880/60000 (98%)]     Loss: 0.272780
Train Epoch: 10 [59520/60000 (99%)]     Loss: 0.079338

Test set: Average loss: 0.0541, Accuracy: 9815/10000 (98%)

10번의 epoch(전체 training dataset을 총 10회 반복 training했다는 뜻) 만에 98.15%의 accuracy를 결과로 냅니다.   이 과정에서는 물론 GPU는 1장만 쓰는데, 대략 30~35%의 사용률을 보입니다.  (이거 P100이 아니라 K80입니다.  착오 없으시기 바랍니다.)


Tue Oct 10 00:23:02 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 361.119                Driver Version: 361.119                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 0000:03:00.0     Off |                    0 |
| N/A   44C    P0    62W / 149W |    380MiB / 11441MiB |     30%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 0000:04:00.0     Off |                    0 |
| N/A   30C    P8    31W / 149W |      2MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | 0020:03:00.0     Off |                    0 |
| N/A   36C    P8    26W / 149W |      2MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           Off  | 0020:04:00.0     Off |                    0 |
| N/A   30C    P8    29W / 149W |      2MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     40735    C   python                                         378MiB |
+-----------------------------------------------------------------------------+