tf_cnn_benchmarks를 이용한 GPU 성능 벤치마크 테스트

Deep learning용 GPU 서버의 성능 측정은 역시 deep learning에 가장 많이 사용되는 tensorflow를 이용한 benchmark test를 돌려보는 것입니다.  구체적으로 어떤 GPU에서는 몇 images/sec의 속도가 나와야 한다는 기준은 존재하지 않습니다.  테스트에 사용되는 neural network의 종류와 사소한 parameter, 그리고 어떤 dataset을 이용하느냐에 따라 그 성능은 천차만별이기 때문입니다.  또 테스트를 위해 labeling된 image dataset을 구하는 것도 쉽지 않습니다.

그런데 이런 어려운 문제를 극복하고 GPU를 이용한 tensorflow를 이용한 benchmark test를 수행하는 방법이 있습니다.  Tensorflow github에 공개된 tf_cnn_benchmarks를 이용하는 것입니다.  이 테스트는 더 이상 update되고 있지는 않지만 다음과 같은 장점이 있어서 아직도 널리 쓰이고 있습니다.

1) 사용방법이 간단

2) GPU 개수를 조절해가며 테스트 가능

3) CPU만으로도 테스트하여 CPU 대비 GPU 성능도 평가 가능

4) Test용 image dataset을 별도로 준비하지 않아도 python code에서 합성하여 사용

5) 성능 평가 수치를 images/sec로 간단명료하게 제시

이를 POWER9 processor를 장착한 IBM POWER 서버에서 수행하기 위해서는 먼저 다음과 같이 anaconda package를 download 받아 설치합니다.

[cecuser@p1226-kvm1 ~]$ wget https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-ppc64le.sh

[cecuser@p1226-kvm1 ~]$ chmod a+x Anaconda3-2020.07-Linux-ppc64le.sh

[cecuser@p1226-kvm1 ~]$ ./Anaconda3-2020.07-Linux-ppc64le.sh

Do you accept the license terms? [yes|no]

[no] >>> yes

[/home/cecuser/anaconda3] >>>

Do you wish the installer to initialize Anaconda3

by running conda init? [yes|no]

[no] >>> yes

Anaconda 설치가 끝난 뒤 profile을 수행하면 conda environment가 가동됩니다.

[cecuser@p1226-kvm1 ~]$ . ~/.bashrc

(base) [cecuser@p1226-kvm1 ~]$

이제 IBM OPEN-CE (구 명칭 IBM PowerAI 또는 Watson Machine Learning Community Edition), 즉 IBM ppc64le 환경에서 쉽게 deep leanring 관련 open source conda package를 설치할 수 있도록 해주는 conda channel을 등록합니다.

(base) [cecuser@p1226-kvm1 ~]$ conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

이 tf_cnn_benchmarks는 python 3.6 환경에서 수행하므로 다음과 같이 python 3.6용 conda virtual environment를 생성합니다.  여기서는 이름을 wmlce_env로 정했습니다.

(base) [cecuser@p1226-kvm1 ~]$ conda create --name wmlce_env python=3.6

방금 만든 wmlce_env 환경을 activate 합니다.  

(base) [cecuser@p1226-kvm1 ~]$ conda activate wmlce_env

이제 prompt 맨 앞부분이 (wmlce_env)로 바뀐 것을 보실 수 있습니다.  이제 OPEN-CE를 설치합니다.  구명칭인 PowerAI라는 이름의 패키지를 설치하시면 됩니다.  네트워크 환경에 따라 시간이 꽤 걸릴 수 있습니다.  (30분 ~ 몇 시간)

(wmlce_env) [cecuser@p1226-kvm1 ~]$ conda install powerai

그러고 난 뒤, 다음과 같이 tensorflow의 하위 github인 benchmarks를 clone 합니다.

(wmlce_env) [cecuser@p1226-kvm1 ~]$ git clone https://github.com/tensorflow/benchmarks

이제 그 중 tf_cnn_benchmarks로 들어갑니다.

(wmlce_env) [cecuser@p1226-kvm1 ~]$ cd benchmarks/scripts/tf_cnn_benchmarks

이 directory에 들어있는 tf_cnn_benchmarks.py를 수행하면 됩니다.  이때 다음과 같은 parameter를 주고 수행하면 됩니다.

먼저, GPU를 1개 이용한 테스트입니다.   아래 테스트는 NVIDIA P100 GPU를 이용한 것입니다만 공식적인 수치는 아니며 환경에 따라 또 다른 수치가 나올 수 있으니 참고용으로만 쓰십시요.

(wmlce_env) [cecuser@p1226-kvm1 tf_cnn_benchmarks]$ python tf_cnn_benchmarks.py --num_gpus=1 --batch_size=32 --model=resnet50 --variable_update=parameter_server --data_format=NHWC --num_batches=640

Done warm up

Step    Img/sec total_loss

1       images/sec: 224.9 +/- 0.0 (jitter = 0.0)        8.108

10      images/sec: 225.9 +/- 0.2 (jitter = 0.6)        8.122

20      images/sec: 226.0 +/- 0.2 (jitter = 0.4)        7.983

...

620     images/sec: 221.4 +/- 0.6 (jitter = 0.7)        7.772

630     images/sec: 221.4 +/- 0.6 (jitter = 0.7)        7.676

640     images/sec: 221.5 +/- 0.6 (jitter = 0.7)        7.779

----------------------------------------------------------------

total images/sec: 221.40

만약 GPU 2장을 쓰고 싶으시면 아래와 같이 --num_gpus=1 대신 --num_gpus=2를 쓰시면 됩니다.

(wmlce_env) [cecuser@p1226-kvm1 tf_cnn_benchmarks]$ python tf_cnn_benchmarks.py --num_gpus=2 --batch_size=32 --model=resnet50 --variable_update=parameter_server --data_format=NHWC --num_batches=640

GPU의 성능이 제대로 나오고 있는 것인지 보는 가장 직관적이고 좋은 방법은 이 동일한 테스트를 CPU로 돌려보는 것입니다.  아래와 같이 --device=CPU를 쓰시면 CPU로만 수행이 가능합니다.  역시 CPU 성능 및 개수와 밀접한 상관이 있으므로, 아래 수치는 참고로만 쓰십시요.

(wmlce_env) [cecuser@p1226-kvm1 tf_cnn_benchmarks]$ python tf_cnn_benchmarks.py --device=CPU --batch_size=32 --model=resnet50 --variable_update=parameter_server --data_format=NHWC --num_batches=640

Done warm up

Step    Img/sec total_loss

1       images/sec: 3.0 +/- 0.0 (jitter = 0.0)  8.108

10      images/sec: 3.0 +/- 0.0 (jitter = 0.0)  8.122

20      images/sec: 2.9 +/- 0.0 (jitter = 0.0)  7.983

...

620     images/sec: 2.9 +/- 0.0 (jitter = 0.0)  7.750

630     images/sec: 2.9 +/- 0.0 (jitter = 0.0)  7.667

640     images/sec: 2.9 +/- 0.0 (jitter = 0.0)  7.805

----------------------------------------------------------------

total images/sec: 2.91

----------------------------------------------------------------

참고로 위 GPU 테스트를 수행할 떄의 nvidia-smi tool을 보면 아래와 같이 GPU 사용률을 보여줍니다.  

Tue Oct 20 04:23:57 2020

+-----------------------------------------------------------------------------+

| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |

|-------------------------------+----------------------+----------------------+

| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |

| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |

|===============================+======================+======================|

|   0  Tesla P100-SXM2...  On   | 00000001:00:01.0 Off |                    0 |

| N/A   51C    P0   240W / 300W |  15677MiB / 16280MiB |     96%      Default |

+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+

| Processes:                                                       GPU Memory |

|  GPU       PID   Type   Process name                             Usage      |

|=============================================================================|

|    0     16243      C   python                                     15667MiB |

+-----------------------------------------------------------------------------+

tf_cnn_benchmarks.py를 이용한 Tensorflow Large Model Support (LMS)의 demo

IBM Watson Machine Learning Community Edition (WML-CE) 1.6.2 속에 포함된  Tensorflow 1.15를 이용하여 large model support (LMS)에 대한 demo를 해보는 방법입니다.

** 아래 환경은 IBM CECC cloud에서 제공되는 가상화 환경의 P100 GPU를 이용했기 때문에, training 성능 자체는 떨어진다는 점을 인지해주시기 바랍니다.


가장 간단한 것은 WML-CE 속에 들어있는 tf_cnn_benchmarks suite를 이용하는 것입니다.

먼저 아래 명령어를 이용하여 tf_cnn_benchmarks suite를 원하는 directory에 copy합니다.  (이건 optional step이고, 그냥 해당 directory로 직접 찾아 들어가도 됩니다.)

(wmlce_162) [cecuser@p1290-kvm1 ~]$ ./anaconda3/envs/wmlce_162/bin/install_tf_cnn_benchmarks .

(wmlce_162) [cecuser@p1290-kvm1 ~]$ cd tf_cnn_benchmarks

(wmlce_162) [cecuser@p1290-kvm1 tf_cnn_benchmarks]$ ls
all_reduce_benchmark.py                   cnn_util.py                 mlperf_test.py     test_data
all_reduce_benchmark_test.py              cnn_util_test.py            models             test_util.py
allreduce.py                              coco_metric.py              platforms          tf_cnn_benchmarks.py
allreduce_test.py                         constants.py                preprocessing.py   variable_mgr.py
batch_allreduce.py                        convnet_builder.py          __pycache__        variable_mgr_util.py
benchmark_cnn_distributed_test.py         datasets.py                 README.md          variable_mgr_util_test.py
benchmark_cnn_distributed_test_runner.py  flags.py                    run_tests.py
benchmark_cnn.py                          leading_indicators_test.py  ssd_constants.py
benchmark_cnn_test.py                     mlperf.py                   ssd_dataloader.py


다만 여기서 benchmark_cnn.py 에서 일부 source code를 수정해야 합니다.  이는 source code 안에 들어있는 LMS 관련 parameter인 lms_swapout_threshold 관련 bug 때문입니다.   원래 값인 -1을 그대로 놔두면 원래는 auto-tuning이 되어야 하는데, TF 버전 등과의 호환 문제로 거기서 에러가 나므로, 일단은 그냥 1로 수정합니다.

(wmlce_162) [cecuser@p1290-kvm1 tf_cnn_benchmarks]$ vi benchmark_cnn.py

...

#flags.DEFINE_integer('lms_swapout_threshold', -1,

flags.DEFINE_integer('lms_swapout_threshold', 1,

...

그렇게 한 뒤 아래와 같이 tf_cnn_benchmarks.py를 수행해 봅니다.   여기서는 batch_size를 150으로 주고 해봅니다.  일단 잘 수행되는 것을 보실 수 있습니다.

(wmlce_162) [cecuser@p1290-kvm1 tf_cnn_benchmarks]$ python tf_cnn_benchmarks.py --batch_size=150 --num_batches=100 --model=resnet50  --num_gpus=1 --display_every=10
...
I0612 04:14:43.269922 140736229690240 session_manager.py:502] Done running local_init_op.
Running warm up
2020-06-12 04:14:45.534222: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-06-12 04:14:45.728889: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 255.8 +/- 0.0 (jitter = 0.0)        7.820
10      images/sec: 255.8 +/- 0.1 (jitter = 0.4)        8.082
20      images/sec: 255.7 +/- 0.1 (jitter = 0.4)        7.856
30      images/sec: 255.6 +/- 0.1 (jitter = 0.3)        7.832
40      images/sec: 255.5 +/- 0.0 (jitter = 0.3)        7.879
50      images/sec: 255.5 +/- 0.0 (jitter = 0.2)        7.701
60      images/sec: 255.5 +/- 0.0 (jitter = 0.2)        7.918
70      images/sec: 255.5 +/- 0.0 (jitter = 0.2)        7.845
80      images/sec: 255.4 +/- 0.0 (jitter = 0.2)        7.750
90      images/sec: 255.3 +/- 0.0 (jitter = 0.2)        7.806
100     images/sec: 255.3 +/- 0.0 (jitter = 0.3)        7.856
----------------------------------------------------------------
total images/sec: 255.22
----------------------------------------------------------------


이때 OS에서 nmon tool을 이용해서 관찰해보면, host RAM의 메모리 사용량이 5GB 정도에 불과하고 tf_cnn_benchmarks의 data size도 32GB 정도, res set size도 2.1GB 정도에 불과한 것을 보실 수 있습니다.


x Memory and Swap qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqx
x PageSize:64KB  RAM-Memory  Swap-Space       High-Memory     Low-Memory         x
x Total (MB)        63392.2      4094.5       - not in use   - not in use        x
x Free  (MB)        58734.8      3518.5                                          x
x Free Percent         92.7%       85.9%                                         x

x Top Processes Procs=365-mode=3-1=Base 3=Perf 4=Size 5=I/O[RootOnly] u=Argsqqqqqx
x  PID    %CPU  Size   Res   Res   Res   Res Shared   Faults  Command            x
x          Used    KB   Set  Text  Data   Lib    KB  Min  Maj                    x
x   13671  42.1 32142m 2153m 3200  2866m    0 633216  12    0 tf_cnn_benchmar    x



이제 batch_size를 200으로 늘려 보겠습니다.  그러면 16GB에 불과한 P100 GPU의 메모리가 꽉 차서 결국 Out-Of-Memory(OOM) error가 발생합니다.

(wmlce_162) [cecuser@p1290-kvm1 tf_cnn_benchmarks]$ python tf_cnn_benchmarks.py --batch_size=200 --num_batches=100 --model=resnet50  --num_gpus=1 --display_every=10
...
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[200,2048,7,7] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node tower_0/v/cg/resnet_v113/conv46/conv2d/Conv2D (defined at /home/cecuser/anaconda3/envs/wmlce_162/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[main_fetch_group/_566]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[200,2048,7,7] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node tower_0/v/cg/resnet_v113/conv46/conv2d/Conv2D (defined at /home/cecuser/anaconda3/envs/wmlce_162/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.
...


하지만 동일한 batch_size를 그대로 주더라도, LMS를 적용하여 --lms=True 옵션을 주고 수행해보면 (비록 많이 느려졌지만) error 없이 수행되는 것을 보실 수 있습니다.


(wmlce_162) [cecuser@p1290-kvm1 tf_cnn_benchmarks]$ python tf_cnn_benchmarks.py --batch_size=200 --num_batches=100 --model=resnet50  --num_gpus=1 --display_every=10 --lms=True
....
I0612 04:27:06.439511 140735558208384 lms.py:1275] [LMS][0] [Simulator] Available memory for simulation: -0.09 GiB (memory ratio: 0.9)
I0612 04:27:06.439677 140735558208384 lms.py:1275] [LMS][0] [Simulator] Available CPU memory for simulation: 64.0 GiB
I0612 04:27:06.440271 140735558208384 lms.py:1275] [LMS][0] LMS will use the latest parameter set found by Simulator for the best performance. However, if you encounter an out-of-memory error, please manually use the previous parameter set found by Simulator.
I0612 04:27:06.440359 140735558208384 lms.py:1275] [LMS][0]   sync_mode: 3 (Synchronous memory copy between host and device)
I0612 04:27:06.440439 140735558208384 lms.py:1275] [LMS][0]   swapout_threshold: 1
I0612 04:27:06.440520 140735558208384 lms.py:1275] [LMS][0]   swapin_ahead: -1 (ignored since sync_mode is 3)
I0612 04:27:06.440600 140735558208384 lms.py:1275] [LMS][0]   swapin_groupby: -1 (ignored since sync_mode is 3)
I0612 04:27:06.869183 140735558208384 lms.py:1275] [LMS][0] Added 425 operations to the model (180 swap-out operations (20.33 GiB) and 245 swap-in operations (31.36 GiB))
I0612 04:27:06.869335 140735558208384 lms.py:1275] [LMS][0] Editing model for LMS, took: 799.3814945220947 ms
...
Running warm up
2020-06-12 04:27:15.098435: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-06-12 04:27:15.371592: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 21.1 +/- 0.0 (jitter = 0.0) 7.988
10      images/sec: 21.1 +/- 0.0 (jitter = 0.1) 7.900
20      images/sec: 21.1 +/- 0.0 (jitter = 0.1) 7.914
30      images/sec: 21.1 +/- 0.0 (jitter = 0.0) 8.043
40      images/sec: 21.1 +/- 0.0 (jitter = 0.0) 7.880
50      images/sec: 21.1 +/- 0.0 (jitter = 0.0) 7.903
60      images/sec: 21.1 +/- 0.0 (jitter = 0.0) 7.889
70      images/sec: 21.1 +/- 0.0 (jitter = 0.0) 7.770
80      images/sec: 21.1 +/- 0.0 (jitter = 0.0) 7.906
90      images/sec: 21.1 +/- 0.0 (jitter = 0.1) 7.813
100     images/sec: 21.1 +/- 0.0 (jitter = 0.1) 7.824
----------------------------------------------------------------
total images/sec: 21.13
----------------------------------------------------------------


이때 host의 OS를 관찰해보면 host RAM 사용량이 22GB 정도로 대폭 늘었고, tf_cnn_benchmarks의 data size도 50GB 정도, res set size도 19GB 정도로 대폭 늘어난 것을 보실 수 있습니다.


x Memory and Swap qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqx
x PageSize:64KB  RAM-Memory  Swap-Space       High-Memory     Low-Memory         x
x Total (MB)        63392.2      4094.5       - not in use   - not in use        x
x Free  (MB)        41505.6      3490.5                                          x
x Free Percent         65.5%       85.2%                                         x

x Top Processes Procs=365-mode=3-1=Base 3=Perf 4=Size 5=I/O[RootOnly] u=Argsqqqqqx
x  PID    %CPU  Size   Res   Res   Res   Res Shared   Faults  Command            x
x          Used    KB   Set  Text  Data   Lib    KB  Min  Maj                    x
x   13427  10.4 49577m19322m 3200  3399m    0 176710   0    0 tf_cnn_benchmar    x

WML-CE (Watson Machine Learning Community Edition, 구 PowerAI) 1.6.2 설치

IBM POWER 아키텍처 (POWER8/9, 즉 ppc64le)에서 tensorflow나 caffe 등 각종 deep learning framework을 제공해주던 무료 toolkit인 기존 PowerAI는 이미 다들 아시는 바와 같이 이름을 Watson Machine Learning Community Edition (WML-CE)로 변경했습니다.  물론 여전히 무료입니다만, 기존처럼 *.rpm이나 *.deb의 형태로 제공하지 않고 아예 별도의 conda channel을 만들어서 conda에서 설치하도록 하고 있습니다.  따라서, Anaconda가 prerequsite이며, 2020년 2월 초 현재 최신 버전인 1.6.2는 Ananconda 2019.07을 prerequisite으로 하고 있습니다.  아예 모든 것이 설치된 docker image 형태로도 제공됩니다.

자세한 원본 manual은 아래 link를 참조하시면 됩니다.

https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.2/navigation/wmlce_planning.html
https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.2/navigation/wmlce_install.html

여기서는 ppc64le Ubuntu 18.04, Python 3.7.5 환경에서 WML-CE 1.6.2를 설치해보겠습니다.

cecuser@p1234-kvm1:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:        18.04
Codename:       bionic

먼저 Anaconda 2019.07 버전을 download 받아 설치합니다.

cecuser@p1234-kvm1:~$ wget https://repo.continuum.io/archive/Anaconda3-2019.07-Linux-ppc64le.sh

cecuser@p1234-kvm1:~$ chmod a+x Anaconda3-2019.07-Linux-ppc64le.sh

cecuser@p1234-kvm1:~$ ./Anaconda3-2019.07-Linux-ppc64le.sh

설치가 끝나면 ~/.bashrc를 수행하여  conda init을 수행합니다.

cecuser@p1234-kvm1:~$ . ~/.bashrc

이제 IBM이 제공하는 WML-CE를 위한 conda channel을 conda에 추가합니다.

(base) cecuser@p1234-kvm1:~$ conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

Conda 가상 환경을 생성하여 거기에 WML-CE를 설치하기를 권장하므로, 먼저 python 3.7.5 환경으로 wmlce_env라는 이름의 virtual env를 만듭니다.

(base) cecuser@p1234-kvm1:~$ conda create --name wmlce_env python=3.7.5
...
The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-ppc64le::_libgcc_mutex-0.1-main
  ca-certificates    pkgs/main/linux-ppc64le::ca-certificates-2020.1.1-0
  certifi            pkgs/main/linux-ppc64le::certifi-2019.11.28-py37_0
  libedit            pkgs/main/linux-ppc64le::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-ppc64le::libffi-3.2.1-hf62a594_5
  libgcc-ng          pkgs/main/linux-ppc64le::libgcc-ng-8.2.0-h822a55f_1
  libstdcxx-ng       pkgs/main/linux-ppc64le::libstdcxx-ng-8.2.0-h822a55f_1
  ncurses            pkgs/main/linux-ppc64le::ncurses-6.1-he6710b0_1
  openssl            pkgs/main/linux-ppc64le::openssl-1.1.1d-h7b6447c_3
  pip                pkgs/main/linux-ppc64le::pip-20.0.2-py37_1
  python             pkgs/main/linux-ppc64le::python-3.7.5-h4134adf_0
  readline           pkgs/main/linux-ppc64le::readline-7.0-h7b6447c_5
  setuptools         pkgs/main/linux-ppc64le::setuptools-45.1.0-py37_0
  sqlite             pkgs/main/linux-ppc64le::sqlite-3.30.1-h7b6447c_0
  tk                 pkgs/main/linux-ppc64le::tk-8.6.8-hbc83047_0
  wheel              pkgs/main/linux-ppc64le::wheel-0.34.1-py37_0
  xz                 pkgs/main/linux-ppc64le::xz-5.2.4-h14c3975_4
  zlib               pkgs/main/linux-ppc64le::zlib-1.2.11-h7b6447c_3


Proceed ([y]/n)? y
...
Downloading and Extracting Packages
ca-certificates-2020 | 125 KB    | ##################################### | 100%
setuptools-45.1.0    | 511 KB    | ##################################### | 100%
pip-20.0.2           | 1.7 MB    | ##################################### | 100%
sqlite-3.30.1        | 2.3 MB    | ##################################### | 100%
wheel-0.34.1         | 50 KB     | ##################################### | 100%
python-3.7.5         | 32.5 MB   | ##################################### | 100%
openssl-1.1.1d       | 3.8 MB    | ##################################### | 100%
certifi-2019.11.28   | 156 KB    | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate wmlce_env
#
# To deactivate an active environment, use
#
#     $ conda deactivate

이제 conda 가상환경인 wmlce_env를 활성화합니다.

(base) cecuser@p1234-kvm1:~$ conda activate wmlce_env

이제 WML-CE를 설치합니다.  Conda package 이름은 여전히 PowerAI로 되어 있는 점에 유의하십시요.  아래와 같이 하면 tensorflow와 caffe2, pytorch 등 WML-CE에서 지원하는 모든 deep learning framework이 한꺼번에 다 설치됩니다.  혹시 WML-CE 전체를 설치하지 않고 가령 PyTorch만 설치하려 할 때는 그냥 conda install pytorch 라고 하시면 됩니다.

아래의 명령어로 어느어느 package들이 설치되는지 보여드리기 위해 긴 ouput을 일부러 다 옮겨 붙였습니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda install powerai
....
The following NEW packages will be INSTALLED:

  _py-xgboost-mutex  ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_py-xgboost-mutex-1.0-gpu_590.g8a21f75
  _pytorch_select    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_pytorch_select-2.0-gpu_20238.g1faf942
  _tflow_select      ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_tflow_select-2.1.0-gpu_840.g50de12c
  absl-py            pkgs/main/linux-ppc64le::absl-py-0.7.1-py37_0
  apex               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::apex-0.1.0_1.6.2-py37_596.g1eb5c77
  asn1crypto         pkgs/main/linux-ppc64le::asn1crypto-1.3.0-py37_0
  astor              pkgs/main/linux-ppc64le::astor-0.7.1-py37_0
  atomicwrites       pkgs/main/linux-ppc64le::atomicwrites-1.3.0-py37_1
  attrs              pkgs/main/noarch::attrs-19.3.0-py_0
  blas               pkgs/main/linux-ppc64le::blas-1.0-openblas
  bokeh              pkgs/main/linux-ppc64le::bokeh-1.4.0-py37_0
  boost              pkgs/main/linux-ppc64le::boost-1.67.0-py37_4
  bzip2              pkgs/main/linux-ppc64le::bzip2-1.0.8-h7b6447c_0
  c-ares             pkgs/main/linux-ppc64le::c-ares-1.15.0-h7b6447c_1001
  caffe              ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::caffe-1.0_1.6.2-5184.g7b10df4
  caffe-base         ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::caffe-base-1.0_1.6.2-gpu_py37_5184.g7b10df4
  cairo              pkgs/main/linux-ppc64le::cairo-1.14.12-h8948797_3
  cffi               pkgs/main/linux-ppc64le::cffi-1.12.3-py37h2e261b9_0
  chardet            pkgs/main/linux-ppc64le::chardet-3.0.4-py37_1003
  click              pkgs/main/linux-ppc64le::click-7.0-py37_0
  cloudpickle        pkgs/main/noarch::cloudpickle-1.2.2-py_0
  coverage           pkgs/main/linux-ppc64le::coverage-5.0-py37h7b6447c_0
  cryptography       pkgs/main/linux-ppc64le::cryptography-2.8-py37h1ba5d50_0
  cudatoolkit        ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudatoolkit-10.1.243-616.gc122b8b
  cudnn              ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudnn-7.6.3_10.1-590.g5627c5e
  cycler             pkgs/main/linux-ppc64le::cycler-0.10.0-py37_0
  cytoolz            pkgs/main/linux-ppc64le::cytoolz-0.10.1-py37h7b6447c_0
  dask               pkgs/main/noarch::dask-2.3.0-py_0
  dask-core          pkgs/main/noarch::dask-core-2.3.0-py_0
  dask-cuda          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-cuda-0.9.1-py37_573.g9af8baa
  dask-xgboost       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-xgboost-0.1.7-py37_579.g8a31cf5
  ddl                ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::ddl-1.5.0-py37_1287.gc90c6f2
  ddl-tensorflow     ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::ddl-tensorflow-1.5.0-py37_1007.g8dbb51d
  decorator          pkgs/main/noarch::decorator-4.4.1-py_0
  distributed        pkgs/main/noarch::distributed-2.3.2-py_1
  ffmpeg             pkgs/main/linux-ppc64le::ffmpeg-4.0-hcdf2ecd_0
  fontconfig         pkgs/main/linux-ppc64le::fontconfig-2.13.0-h9420a91_0
  freeglut           pkgs/main/linux-ppc64le::freeglut-3.0.0-hf484d3e_5
  freetype           pkgs/main/linux-ppc64le::freetype-2.9.1-h8a8886c_0
  fsspec             pkgs/main/noarch::fsspec-0.6.2-py_0
  future             pkgs/main/linux-ppc64le::future-0.17.1-py37_0
  gast               pkgs/main/linux-ppc64le::gast-0.2.2-py37_0
  gflags             ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::gflags-2.2.2-1624.g17209b3
  glib               pkgs/main/linux-ppc64le::glib-2.63.1-h5a9c865_0
  glog               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::glog-0.3.5-1613.gd054598
  google-pasta       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::google-pasta-0.1.6-py37_564.g04df2d9
  graphite2          pkgs/main/linux-ppc64le::graphite2-1.3.13-h23475e2_0
  graphsurgeon       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::graphsurgeon-0.4.1-py37_612.gb2bf6b9
  grpcio             pkgs/main/linux-ppc64le::grpcio-1.16.1-py37hf8bcb03_1
  h5py               pkgs/main/linux-ppc64le::h5py-2.8.0-py37h8d01980_0
  harfbuzz           pkgs/main/linux-ppc64le::harfbuzz-1.8.8-hffaf4a1_0
  hdf5               pkgs/main/linux-ppc64le::hdf5-1.10.2-hba1933b_1
  heapdict           pkgs/main/noarch::heapdict-1.0.1-py_0
  hypothesis         pkgs/main/linux-ppc64le::hypothesis-3.59.1-py37h39e3cac_0
  icu                pkgs/main/linux-ppc64le::icu-58.2-h64fc554_1
  idna               pkgs/main/linux-ppc64le::idna-2.8-py37_0
  imageio            pkgs/main/linux-ppc64le::imageio-2.6.1-py37_0
  importlib_metadata pkgs/main/linux-ppc64le::importlib_metadata-1.4.0-py37_0
  jasper             pkgs/main/linux-ppc64le::jasper-2.0.14-h07fcdf6_1
  jinja2             pkgs/main/noarch::jinja2-2.10.3-py_0
  joblib             pkgs/main/linux-ppc64le::joblib-0.13.2-py37_0
  jpeg               pkgs/main/linux-ppc64le::jpeg-9b-hcb7ba68_2
  keras-applications pkgs/main/noarch::keras-applications-1.0.8-py_0
  keras-preprocessi~ pkgs/main/noarch::keras-preprocessing-1.1.0-py_1
  kiwisolver         pkgs/main/linux-ppc64le::kiwisolver-1.1.0-py37he6710b0_0
  leveldb            pkgs/main/linux-ppc64le::leveldb-1.20-hf484d3e_1
  libboost           pkgs/main/linux-ppc64le::libboost-1.67.0-h46d08c1_4
  libgfortran-ng     pkgs/main/linux-ppc64le::libgfortran-ng-7.3.0-h822a55f_1
  libglu             pkgs/main/linux-ppc64le::libglu-9.0.0-hf484d3e_1
  libopenblas        pkgs/main/linux-ppc64le::libopenblas-0.3.6-h5a2b251_1
  libopencv          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libopencv-3.4.7-725.g92aa195
  libopus            pkgs/main/linux-ppc64le::libopus-1.3-h7b6447c_0
  libpng             pkgs/main/linux-ppc64le::libpng-1.6.37-hbc83047_0
  libprotobuf        ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libprotobuf-3.8.0-577.g45759bb
  libtiff            pkgs/main/linux-ppc64le::libtiff-4.1.0-h2733197_0
  libuuid            pkgs/main/linux-ppc64le::libuuid-1.0.3-h1bed415_2
  libvpx             pkgs/main/linux-ppc64le::libvpx-1.7.0-hf484d3e_0
  libxcb             pkgs/main/linux-ppc64le::libxcb-1.13-h1bed415_0
  libxgboost-base    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libxgboost-base-0.90-gpu_590.g8a21f75
  libxml2            pkgs/main/linux-ppc64le::libxml2-2.9.9-hea5a465_1
  llvmlite           pkgs/main/linux-ppc64le::llvmlite-0.29.0-py37hd408876_0
  lmdb               pkgs/main/linux-ppc64le::lmdb-0.9.22-hf484d3e_1
  locket             pkgs/main/linux-ppc64le::locket-0.2.0-py37_1
  markdown           pkgs/main/linux-ppc64le::markdown-3.1.1-py37_0
  markupsafe         pkgs/main/linux-ppc64le::markupsafe-1.1.1-py37h7b6447c_0
  matplotlib         pkgs/main/linux-ppc64le::matplotlib-3.1.2-py37_1
  matplotlib-base    pkgs/main/linux-ppc64le::matplotlib-base-3.1.2-py37h4fdacc2_1
  mock               pkgs/main/linux-ppc64le::mock-2.0.0-py37_0
  more-itertools     pkgs/main/noarch::more-itertools-8.0.2-py_0
  msgpack-python     pkgs/main/linux-ppc64le::msgpack-python-0.6.1-py37hfd86e86_1
  nccl               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::nccl-2.4.8-586.gdba67b7
  networkx           pkgs/main/linux-ppc64le::networkx-2.2-py37_1
  ninja              pkgs/main/linux-ppc64le::ninja-1.9.0-py37hfd86e86_0
  nomkl              pkgs/main/linux-ppc64le::nomkl-3.0-0
  numactl            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::numactl-2.0.12-573.gdf5dc62
  numba              pkgs/main/linux-ppc64le::numba-0.45.1-py37h962f231_0
  numpy              pkgs/main/linux-ppc64le::numpy-1.16.6-py37h30dfecb_0
  numpy-base         pkgs/main/linux-ppc64le::numpy-base-1.16.6-py37h2f8d375_0
  olefile            pkgs/main/linux-ppc64le::olefile-0.46-py37_0
  onnx               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::onnx-1.5.0-py37_614.gd049fd7
  openblas           pkgs/main/linux-ppc64le::openblas-0.3.6-1
  openblas-devel     pkgs/main/linux-ppc64le::openblas-devel-0.3.6-1
  opencv             ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::opencv-3.4.7-py37_725.g92aa195
  packaging          pkgs/main/noarch::packaging-20.1-py_0
  pai4sk             ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pai4sk-1.5.0-py37_1071.g5abf42e
  pandas             pkgs/main/linux-ppc64le::pandas-1.0.0-py37h0573a6f_0
  partd              pkgs/main/noarch::partd-1.1.0-py_0
  pbr                pkgs/main/noarch::pbr-5.4.4-py_0
  pciutils           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pciutils-3.6.2-571.g2316d13
  pcre               pkgs/main/linux-ppc64le::pcre-8.43-he6710b0_0
  pillow             pkgs/main/linux-ppc64le::pillow-6.2.1-py37h0d2faf8_0
  pixman             pkgs/main/linux-ppc64le::pixman-0.34.0-h1f8d8dc_3
  pluggy             pkgs/main/linux-ppc64le::pluggy-0.13.1-py37_0
  powerai            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-1.6.2-615.g1dade79
  powerai-license    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-license-1.6.2-716.g7081e12
  powerai-release    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-release-1.6.2-572.gb216c2c
  powerai-tools      ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-tools-1.6.2-565.g97f2c3f
  protobuf           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::protobuf-3.8.0-py37_587.gab45ad3
  psutil             pkgs/main/linux-ppc64le::psutil-5.5.0-py37h7b6447c_0
  py                 pkgs/main/noarch::py-1.8.1-py_0
  py-boost           pkgs/main/linux-ppc64le::py-boost-1.67.0-py37h04863e7_4
  py-opencv          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-opencv-3.4.7-py37_725.g92aa195
  py-xgboost-base    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-xgboost-base-0.90-gpu_py37_590.g8a21f75
  py-xgboost-gpu     ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-xgboost-gpu-0.90-590.g8a21f75
  pycparser          pkgs/main/linux-ppc64le::pycparser-2.19-py37_0
  pyopenssl          pkgs/main/linux-ppc64le::pyopenssl-19.1.0-py37_0
  pyparsing          pkgs/main/noarch::pyparsing-2.4.6-py_0
  pysocks            pkgs/main/linux-ppc64le::pysocks-1.7.1-py37_0
  pytest             pkgs/main/linux-ppc64le::pytest-4.4.2-py37_0
  python-dateutil    pkgs/main/noarch::python-dateutil-2.8.1-py_0
  python-lmdb        pkgs/main/linux-ppc64le::python-lmdb-0.94-py37h14c3975_0
  pytorch            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pytorch-1.2.0-20238.g1faf942
  pytorch-base       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pytorch-base-1.2.0-gpu_py37_20238.g1faf942
  pytz               pkgs/main/noarch::pytz-2019.3-py_0
  pywavelets         pkgs/main/linux-ppc64le::pywavelets-1.1.1-py37h7b6447c_0
  pyyaml             pkgs/main/linux-ppc64le::pyyaml-5.1.2-py37h7b6447c_0
  requests           pkgs/main/linux-ppc64le::requests-2.22.0-py37_1
  scikit-image       pkgs/main/linux-ppc64le::scikit-image-0.15.0-py37he6710b0_0
  scikit-learn       pkgs/main/linux-ppc64le::scikit-learn-0.21.3-py37h22eb022_0
  scipy              pkgs/main/linux-ppc64le::scipy-1.3.1-py37he2b7bc3_0
  simsearch          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::simsearch-1.1.0-py37_764.g7c5f6cf
  six                pkgs/main/linux-ppc64le::six-1.12.0-py37_0
  snapml-spark       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::snapml-spark-1.4.0-py37_942.gc873569
  snappy             pkgs/main/linux-ppc64le::snappy-1.1.7-h1532aa0_3
  sortedcontainers   pkgs/main/linux-ppc64le::sortedcontainers-2.1.0-py37_0
  spectrum-mpi       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::spectrum-mpi-10.03-622.gfc88b70
  tabulate           pkgs/main/linux-ppc64le::tabulate-0.8.2-py37_0
  tblib              pkgs/main/noarch::tblib-1.6.0-py_0
  tensorboard        ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorboard-1.15.0-py37_ab7f72a_3645.gf4f525e
  tensorflow         ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-1.15.0-gpu_py37_841.g50de12c
  tensorflow-base    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-base-1.15.0-gpu_py37_590d6ee_64210.g4a039ec
  tensorflow-estima~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-estimator-1.15.1-py37_a5f60ce_1351.g50de12c
  tensorflow-gpu     ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-gpu-1.15.0-841.g50de12c
  tensorflow-large-~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-large-model-support-2.0.2-py37_970.gfa57a9e
  tensorflow-probab~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-probability-0.8.0-py37_b959b26_2686.g50de12c
  tensorflow-servin~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-serving-api-1.15.0-py37_748217e_5094.g89559ef
  tensorrt           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorrt-6.0.1.5-py37_612.gb2bf6b9
  termcolor          pkgs/main/linux-ppc64le::termcolor-1.1.0-py37_1
  tf_cnn_benchmarks  ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tf_cnn_benchmarks-1.15-gpu_py37_1374.g5e94b18
  toolz              pkgs/main/noarch::toolz-0.10.0-py_0
  torchtext          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::torchtext-0.4.0-py37_578.g5bf3960
  torchvision-base   ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::torchvision-base-0.4.0-gpu_py37_593.g80f339d
  tornado            pkgs/main/linux-ppc64le::tornado-6.0.3-py37h7b6447c_0
  tqdm               pkgs/main/noarch::tqdm-4.32.1-py_0
  typing             pkgs/main/linux-ppc64le::typing-3.6.4-py37_0
  uff                ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::uff-0.6.5-py37_612.gb2bf6b9
  urllib3            pkgs/main/linux-ppc64le::urllib3-1.25.8-py37_0
  werkzeug           pkgs/main/noarch::werkzeug-0.15.4-py_0
  wrapt              pkgs/main/linux-ppc64le::wrapt-1.11.2-py37h7b6447c_0
  yaml               pkgs/main/linux-ppc64le::yaml-0.1.7-h1bed415_2
  zict               pkgs/main/noarch::zict-1.0.0-py_0
  zipp               pkgs/main/noarch::zipp-0.6.0-py_0
  zstd               pkgs/main/linux-ppc64le::zstd-1.3.7-h0b5b093_0


Proceed ([y]/n)? y
...

아래와 같이 설치된 package들을 각각 확인하시면 됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep tensorflow
ddl-tensorflow            1.5.0           py37_1007.g8dbb51d    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow                1.15.0          gpu_py37_841.g50de12c    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-base           1.15.0          gpu_py37_590d6ee_64210.g4a039ec    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-estimator      1.15.1          py37_a5f60ce_1351.g50de12c    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-gpu            1.15.0             841.g50de12c    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-large-model-support 2.0.2           py37_970.gfa57a9e    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-probability    0.8.0           py37_b959b26_2686.g50de12c    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-serving-api    1.15.0          py37_748217e_5094.g89559ef    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep caffe
caffe                     1.0_1.6.2         5184.g7b10df4    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
caffe-base                1.0_1.6.2       gpu_py37_5184.g7b10df4    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep pytorch
_pytorch_select           2.0             gpu_20238.g1faf942    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
pytorch                   1.2.0            20238.g1faf942    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
pytorch-base              1.2.0           gpu_py37_20238.g1faf942    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda


WML-CE에서는 NCCL와 CUDNN 등 base facility도 함께 제공되어 설치됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep -i nccl
nccl                      2.4.8              586.gdba67b7    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep -i dnn
cudnn                     7.6.3_10.1         590.g5627c5e    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

아래와 같이 python에서 import를 해보셔도 됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ python
Python 3.7.5 (default, Oct 25 2019, 16:29:01)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> import tensorflow as tf
2020-02-03 21:31:31.095570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
>>>


다만 "conda install powerai" 만으로는 RAPIDS까지 설치되지는 않기 때문에, 아래와 같이 별도로 설치하셔야 합니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda install powerai-rapids
...
The following NEW packages will be INSTALLED:

  arrow-cpp          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::arrow-cpp-0.15.1-py37_603.g702c836
  boost-cpp          pkgs/main/linux-ppc64le::boost-cpp-1.67.0-h14c3975_4
  brotli             pkgs/main/linux-ppc64le::brotli-1.0.6-he6710b0_0
  cudf               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudf-0.9.0-cuda10.1_py37_626.gddcad2d
  cuml               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cuml-0.9.1-cuda10.1_py37_605.gfe9e07b
  cupy               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cupy-6.2.0-py37_567.g0f1e2ef
  cython             pkgs/main/linux-ppc64le::cython-0.29.14-py37he6710b0_0
  dask-cudf          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-cudf-0.9.0-py37_575.g0416adf
  dlpack             ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dlpack-0.2-562.g28dffd9
  double-conversion  ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::double-conversion-3.1.5-564.g4b43169
  fastavro           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::fastavro-0.22.4-py37_562.g9525976
  fastrlock          pkgs/main/linux-ppc64le::fastrlock-0.4-py37he6710b0_0
  grpc-cpp           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::grpc-cpp-1.23.0-568.g4f71a06
  libcudf            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libcudf-0.9.0-cuda10.1_609.g113236a
  libcuml            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libcuml-0.9.1-cuda10.1_576.ga304a0a
  libevent           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libevent-2.1.8-561.ge1d98f7
  libnvstrings       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libnvstrings-0.9.0-cuda10.1_570.ga04797c
  librmm             ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::librmm-0.9.0-cuda10.1_567.gff1b1a1
  lz4-c              pkgs/main/linux-ppc64le::lz4-c-1.8.1.2-h14c3975_0
  nvstrings          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::nvstrings-0.9.0-cuda10.1_py37_580.gdbb6546
  parquet-cpp        ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::parquet-cpp-1.5.1-579.g6eecc60
  powerai-rapids     ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-rapids-1.6.2-560.ga7c5a47
  pyarrow            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pyarrow-0.15.1-py37_609.g3a6717a
  re2                ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::re2-2019.08.01-561.gef92448
  rmm                ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::rmm-0.9.0-cuda10.1_py37_569.g04c75fb
  thrift-cpp         ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::thrift-cpp-0.12.0-580.gf96fa62
  uriparser          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::uriparser-0.9.3-561.g7465fef

The following packages will be DOWNGRADED:

  pandas                               1.0.0-py37h0573a6f_0 --> 0.24.2-py37he6710b0_0


Proceed ([y]/n)? y
...


만약 GPU가 없는 시스템에서 tensorflow나 pytorch 등을 사용하시고자 할 때는, 아래와 같이 CPU-only 버전의 WML-CE를 설치하시면 됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda install powerai-cpu