IBM PowerAI (Watson ML Community Edition)이 설치된 Ubuntu ppc64le 기반의 docker image 만들기

 

먼저 다음 링크를 참조하여 ppc64le (IBM POWER9) nvidia-docker2 환경에서 Ubuntu 기반의 docker image를 만듭니다.  

http://hwengineer.blogspot.com/2019/05/ppc64le-ibm-power9-nvidia-docker2.html

참고로 ppc64le (IBM POWER9)에서의 CUDA 설치는 NVIDIA CUDA download page의 안내와 같이 아래처럼 진행하시면 됩니다.

# wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/cuda-repo-ubuntu1804_10.1.105-1_ppc64el.deb

# dpkg -i cuda-repo-ubuntu1804_10.1.105-1_ppc64el.deb

# apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/7fa2af80.pub

# apt-get update

# apt-get install cuda

또는 이미 만들어둔 docker image를 다음과 같이 pull 해와도 됩니다.

# docker pull bsyu/ubuntu18.04_cuda10-1_ppc64le:v0.1

이렇게 pull 받아온 docker image를 확인합니다.

root@unigpu:/files/docker# docker images

REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE

bsyu/ubuntu18.04_cuda10-1_ppc64le   v0.1                ef8dd4d654e7        2 hours ago         6.33GB

ubuntu                              18.04               ecc8dc2e4170        4 weeks ago         106MB

이 image를 다음과 같이 구동합니다.  

root@unigpu:/files/docker# docker run --runtime=nvidia -ti --rm bsyu/ubuntu18.04_cuda10-1_ppc64lel:v0.1 bash

이제 그 image 속에서 다음과 같이 IBM PowerAI (IBM Watson ML Community Edition)을 설치합니다.  이는 IBM이 마련한 conda channel을 등록하고 거기에서 conda install 명령을 수행하는 방식으로 설치됩니다.

# conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

# conda create --name wmlce_env python=3.6

# conda activate wmlce_env

# apt-get install openssh-server

# conda install powerai     

위와 같이 conda install powerai 명령을 내리면 tensorflow 뿐만 아니라 caffe, pytorch 등이 모두 설치됩니다.  가령 Tensorflow 1.14만 설치하고자 한다면 위 명령 대신 conda install tensorflow=1.14를 수행하시면 됩니다.

powerai 전체 package 설치는 network 사정에 따라 1~2시간이 걸리기도 합니다.  설치가 완료되면 다음과 같이 docker commit하여 docker image를 저장합니다.

# docker ps -a

CONTAINER ID        IMAGE                                   COMMAND             CREATED             STATUS              PORTS               NAMES

45fb663f025c        bsyu/ubuntu18.04_cuda10-1_ppc64le:v0.1   "bash"              42 seconds ago      Up 39 seconds                           elastic_gates

이어서 v0.2 등의 새로운 tag로 commit 하시면 됩니다.

[root@ac922 docker]# docker commit 45fb663f025c bsyu/ubuntu18.04_cuda10-1_ppc64le:v0.2

아래는 그렇게 만들어진 docker image들의 사용예입니다.   제가 만든 그런 image들은 https://hub.docker.com/u/bsyu 에 올려져 있습니다.

root@unigpu:~# docker run --runtime=nvidia -ti --rm bsyu/ubuntu18.04_cuda10-1_tf1.15_pytorch1.2_ppc64le:latest 

(wmlce_env) root@bdbf11e90094:/# python

Python 3.6.10 |Anaconda, Inc.| (default, Mar 26 2020, 00:22:27)

[GCC 7.3.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import torch

>>> import tensorflow as tf

2020-05-29 04:53:09.637141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1

>>> sess=tf.Session()

2020-05-29 04:53:20.795027: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1

2020-05-29 04:53:20.825918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:

name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53

pciBusID: 0004:04:00.0

2020-05-29 04:53:20.827162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties:

name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53

pciBusID: 0004:05:00.0

2020-05-29 04:53:20.828421: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 2 with properties:

name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53

pciBusID: 0035:03:00.0

2020-05-29 04:53:20.829655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 3 with properties:

name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53

pciBusID: 0035:04:00.0

tf_cnn_benchmarks를 이용한 GPU 성능 벤치마크 테스트

Deep learning용 GPU 서버의 성능 측정은 역시 deep learning에 가장 많이 사용되는 tensorflow를 이용한 benchmark test를 돌려보는 것입니다.  구체적으로 어떤 GPU에서는 몇 images/sec의 속도가 나와야 한다는 기준은 존재하지 않습니다.  테스트에 사용되는 neural network의 종류와 사소한 parameter, 그리고 어떤 dataset을 이용하느냐에 따라 그 성능은 천차만별이기 때문입니다.  또 테스트를 위해 labeling된 image dataset을 구하는 것도 쉽지 않습니다.

그런데 이런 어려운 문제를 극복하고 GPU를 이용한 tensorflow를 이용한 benchmark test를 수행하는 방법이 있습니다.  Tensorflow github에 공개된 tf_cnn_benchmarks를 이용하는 것입니다.  이 테스트는 더 이상 update되고 있지는 않지만 다음과 같은 장점이 있어서 아직도 널리 쓰이고 있습니다.

1) 사용방법이 간단

2) GPU 개수를 조절해가며 테스트 가능

3) CPU만으로도 테스트하여 CPU 대비 GPU 성능도 평가 가능

4) Test용 image dataset을 별도로 준비하지 않아도 python code에서 합성하여 사용

5) 성능 평가 수치를 images/sec로 간단명료하게 제시

이를 POWER9 processor를 장착한 IBM POWER 서버에서 수행하기 위해서는 먼저 다음과 같이 anaconda package를 download 받아 설치합니다.

[cecuser@p1226-kvm1 ~]$ wget https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-ppc64le.sh

[cecuser@p1226-kvm1 ~]$ chmod a+x Anaconda3-2020.07-Linux-ppc64le.sh

[cecuser@p1226-kvm1 ~]$ ./Anaconda3-2020.07-Linux-ppc64le.sh

Do you accept the license terms? [yes|no]

[no] >>> yes

[/home/cecuser/anaconda3] >>>

Do you wish the installer to initialize Anaconda3

by running conda init? [yes|no]

[no] >>> yes

Anaconda 설치가 끝난 뒤 profile을 수행하면 conda environment가 가동됩니다.

[cecuser@p1226-kvm1 ~]$ . ~/.bashrc

(base) [cecuser@p1226-kvm1 ~]$

이제 IBM OPEN-CE (구 명칭 IBM PowerAI 또는 Watson Machine Learning Community Edition), 즉 IBM ppc64le 환경에서 쉽게 deep leanring 관련 open source conda package를 설치할 수 있도록 해주는 conda channel을 등록합니다.

(base) [cecuser@p1226-kvm1 ~]$ conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

이 tf_cnn_benchmarks는 python 3.6 환경에서 수행하므로 다음과 같이 python 3.6용 conda virtual environment를 생성합니다.  여기서는 이름을 wmlce_env로 정했습니다.

(base) [cecuser@p1226-kvm1 ~]$ conda create --name wmlce_env python=3.6

방금 만든 wmlce_env 환경을 activate 합니다.  

(base) [cecuser@p1226-kvm1 ~]$ conda activate wmlce_env

이제 prompt 맨 앞부분이 (wmlce_env)로 바뀐 것을 보실 수 있습니다.  이제 OPEN-CE를 설치합니다.  구명칭인 PowerAI라는 이름의 패키지를 설치하시면 됩니다.  네트워크 환경에 따라 시간이 꽤 걸릴 수 있습니다.  (30분 ~ 몇 시간)

(wmlce_env) [cecuser@p1226-kvm1 ~]$ conda install powerai

그러고 난 뒤, 다음과 같이 tensorflow의 하위 github인 benchmarks를 clone 합니다.

(wmlce_env) [cecuser@p1226-kvm1 ~]$ git clone https://github.com/tensorflow/benchmarks

이제 그 중 tf_cnn_benchmarks로 들어갑니다.

(wmlce_env) [cecuser@p1226-kvm1 ~]$ cd benchmarks/scripts/tf_cnn_benchmarks

이 directory에 들어있는 tf_cnn_benchmarks.py를 수행하면 됩니다.  이때 다음과 같은 parameter를 주고 수행하면 됩니다.

먼저, GPU를 1개 이용한 테스트입니다.   아래 테스트는 NVIDIA P100 GPU를 이용한 것입니다만 공식적인 수치는 아니며 환경에 따라 또 다른 수치가 나올 수 있으니 참고용으로만 쓰십시요.

(wmlce_env) [cecuser@p1226-kvm1 tf_cnn_benchmarks]$ python tf_cnn_benchmarks.py --num_gpus=1 --batch_size=32 --model=resnet50 --variable_update=parameter_server --data_format=NHWC --num_batches=640

Done warm up

Step    Img/sec total_loss

1       images/sec: 224.9 +/- 0.0 (jitter = 0.0)        8.108

10      images/sec: 225.9 +/- 0.2 (jitter = 0.6)        8.122

20      images/sec: 226.0 +/- 0.2 (jitter = 0.4)        7.983

...

620     images/sec: 221.4 +/- 0.6 (jitter = 0.7)        7.772

630     images/sec: 221.4 +/- 0.6 (jitter = 0.7)        7.676

640     images/sec: 221.5 +/- 0.6 (jitter = 0.7)        7.779

----------------------------------------------------------------

total images/sec: 221.40

만약 GPU 2장을 쓰고 싶으시면 아래와 같이 --num_gpus=1 대신 --num_gpus=2를 쓰시면 됩니다.

(wmlce_env) [cecuser@p1226-kvm1 tf_cnn_benchmarks]$ python tf_cnn_benchmarks.py --num_gpus=2 --batch_size=32 --model=resnet50 --variable_update=parameter_server --data_format=NHWC --num_batches=640

GPU의 성능이 제대로 나오고 있는 것인지 보는 가장 직관적이고 좋은 방법은 이 동일한 테스트를 CPU로 돌려보는 것입니다.  아래와 같이 --device=CPU를 쓰시면 CPU로만 수행이 가능합니다.  역시 CPU 성능 및 개수와 밀접한 상관이 있으므로, 아래 수치는 참고로만 쓰십시요.

(wmlce_env) [cecuser@p1226-kvm1 tf_cnn_benchmarks]$ python tf_cnn_benchmarks.py --device=CPU --batch_size=32 --model=resnet50 --variable_update=parameter_server --data_format=NHWC --num_batches=640

Done warm up

Step    Img/sec total_loss

1       images/sec: 3.0 +/- 0.0 (jitter = 0.0)  8.108

10      images/sec: 3.0 +/- 0.0 (jitter = 0.0)  8.122

20      images/sec: 2.9 +/- 0.0 (jitter = 0.0)  7.983

...

620     images/sec: 2.9 +/- 0.0 (jitter = 0.0)  7.750

630     images/sec: 2.9 +/- 0.0 (jitter = 0.0)  7.667

640     images/sec: 2.9 +/- 0.0 (jitter = 0.0)  7.805

----------------------------------------------------------------

total images/sec: 2.91

----------------------------------------------------------------

참고로 위 GPU 테스트를 수행할 떄의 nvidia-smi tool을 보면 아래와 같이 GPU 사용률을 보여줍니다.  

Tue Oct 20 04:23:57 2020

+-----------------------------------------------------------------------------+

| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |

|-------------------------------+----------------------+----------------------+

| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |

| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |

|===============================+======================+======================|

|   0  Tesla P100-SXM2...  On   | 00000001:00:01.0 Off |                    0 |

| N/A   51C    P0   240W / 300W |  15677MiB / 16280MiB |     96%      Default |

+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+

| Processes:                                                       GPU Memory |

|  GPU       PID   Type   Process name                             Usage      |

|=============================================================================|

|    0     16243      C   python                                     15667MiB |

+-----------------------------------------------------------------------------+

WML-CE (Watson Machine Learning Community Edition, 구 PowerAI) 1.6.2 설치

IBM POWER 아키텍처 (POWER8/9, 즉 ppc64le)에서 tensorflow나 caffe 등 각종 deep learning framework을 제공해주던 무료 toolkit인 기존 PowerAI는 이미 다들 아시는 바와 같이 이름을 Watson Machine Learning Community Edition (WML-CE)로 변경했습니다.  물론 여전히 무료입니다만, 기존처럼 *.rpm이나 *.deb의 형태로 제공하지 않고 아예 별도의 conda channel을 만들어서 conda에서 설치하도록 하고 있습니다.  따라서, Anaconda가 prerequsite이며, 2020년 2월 초 현재 최신 버전인 1.6.2는 Ananconda 2019.07을 prerequisite으로 하고 있습니다.  아예 모든 것이 설치된 docker image 형태로도 제공됩니다.

자세한 원본 manual은 아래 link를 참조하시면 됩니다.

https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.2/navigation/wmlce_planning.html
https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.2/navigation/wmlce_install.html

여기서는 ppc64le Ubuntu 18.04, Python 3.7.5 환경에서 WML-CE 1.6.2를 설치해보겠습니다.

cecuser@p1234-kvm1:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:        18.04
Codename:       bionic

먼저 Anaconda 2019.07 버전을 download 받아 설치합니다.

cecuser@p1234-kvm1:~$ wget https://repo.continuum.io/archive/Anaconda3-2019.07-Linux-ppc64le.sh

cecuser@p1234-kvm1:~$ chmod a+x Anaconda3-2019.07-Linux-ppc64le.sh

cecuser@p1234-kvm1:~$ ./Anaconda3-2019.07-Linux-ppc64le.sh

설치가 끝나면 ~/.bashrc를 수행하여  conda init을 수행합니다.

cecuser@p1234-kvm1:~$ . ~/.bashrc

이제 IBM이 제공하는 WML-CE를 위한 conda channel을 conda에 추가합니다.

(base) cecuser@p1234-kvm1:~$ conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

Conda 가상 환경을 생성하여 거기에 WML-CE를 설치하기를 권장하므로, 먼저 python 3.7.5 환경으로 wmlce_env라는 이름의 virtual env를 만듭니다.

(base) cecuser@p1234-kvm1:~$ conda create --name wmlce_env python=3.7.5
...
The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-ppc64le::_libgcc_mutex-0.1-main
  ca-certificates    pkgs/main/linux-ppc64le::ca-certificates-2020.1.1-0
  certifi            pkgs/main/linux-ppc64le::certifi-2019.11.28-py37_0
  libedit            pkgs/main/linux-ppc64le::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-ppc64le::libffi-3.2.1-hf62a594_5
  libgcc-ng          pkgs/main/linux-ppc64le::libgcc-ng-8.2.0-h822a55f_1
  libstdcxx-ng       pkgs/main/linux-ppc64le::libstdcxx-ng-8.2.0-h822a55f_1
  ncurses            pkgs/main/linux-ppc64le::ncurses-6.1-he6710b0_1
  openssl            pkgs/main/linux-ppc64le::openssl-1.1.1d-h7b6447c_3
  pip                pkgs/main/linux-ppc64le::pip-20.0.2-py37_1
  python             pkgs/main/linux-ppc64le::python-3.7.5-h4134adf_0
  readline           pkgs/main/linux-ppc64le::readline-7.0-h7b6447c_5
  setuptools         pkgs/main/linux-ppc64le::setuptools-45.1.0-py37_0
  sqlite             pkgs/main/linux-ppc64le::sqlite-3.30.1-h7b6447c_0
  tk                 pkgs/main/linux-ppc64le::tk-8.6.8-hbc83047_0
  wheel              pkgs/main/linux-ppc64le::wheel-0.34.1-py37_0
  xz                 pkgs/main/linux-ppc64le::xz-5.2.4-h14c3975_4
  zlib               pkgs/main/linux-ppc64le::zlib-1.2.11-h7b6447c_3


Proceed ([y]/n)? y
...
Downloading and Extracting Packages
ca-certificates-2020 | 125 KB    | ##################################### | 100%
setuptools-45.1.0    | 511 KB    | ##################################### | 100%
pip-20.0.2           | 1.7 MB    | ##################################### | 100%
sqlite-3.30.1        | 2.3 MB    | ##################################### | 100%
wheel-0.34.1         | 50 KB     | ##################################### | 100%
python-3.7.5         | 32.5 MB   | ##################################### | 100%
openssl-1.1.1d       | 3.8 MB    | ##################################### | 100%
certifi-2019.11.28   | 156 KB    | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate wmlce_env
#
# To deactivate an active environment, use
#
#     $ conda deactivate

이제 conda 가상환경인 wmlce_env를 활성화합니다.

(base) cecuser@p1234-kvm1:~$ conda activate wmlce_env

이제 WML-CE를 설치합니다.  Conda package 이름은 여전히 PowerAI로 되어 있는 점에 유의하십시요.  아래와 같이 하면 tensorflow와 caffe2, pytorch 등 WML-CE에서 지원하는 모든 deep learning framework이 한꺼번에 다 설치됩니다.  혹시 WML-CE 전체를 설치하지 않고 가령 PyTorch만 설치하려 할 때는 그냥 conda install pytorch 라고 하시면 됩니다.

아래의 명령어로 어느어느 package들이 설치되는지 보여드리기 위해 긴 ouput을 일부러 다 옮겨 붙였습니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda install powerai
....
The following NEW packages will be INSTALLED:

  _py-xgboost-mutex  ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_py-xgboost-mutex-1.0-gpu_590.g8a21f75
  _pytorch_select    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_pytorch_select-2.0-gpu_20238.g1faf942
  _tflow_select      ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_tflow_select-2.1.0-gpu_840.g50de12c
  absl-py            pkgs/main/linux-ppc64le::absl-py-0.7.1-py37_0
  apex               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::apex-0.1.0_1.6.2-py37_596.g1eb5c77
  asn1crypto         pkgs/main/linux-ppc64le::asn1crypto-1.3.0-py37_0
  astor              pkgs/main/linux-ppc64le::astor-0.7.1-py37_0
  atomicwrites       pkgs/main/linux-ppc64le::atomicwrites-1.3.0-py37_1
  attrs              pkgs/main/noarch::attrs-19.3.0-py_0
  blas               pkgs/main/linux-ppc64le::blas-1.0-openblas
  bokeh              pkgs/main/linux-ppc64le::bokeh-1.4.0-py37_0
  boost              pkgs/main/linux-ppc64le::boost-1.67.0-py37_4
  bzip2              pkgs/main/linux-ppc64le::bzip2-1.0.8-h7b6447c_0
  c-ares             pkgs/main/linux-ppc64le::c-ares-1.15.0-h7b6447c_1001
  caffe              ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::caffe-1.0_1.6.2-5184.g7b10df4
  caffe-base         ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::caffe-base-1.0_1.6.2-gpu_py37_5184.g7b10df4
  cairo              pkgs/main/linux-ppc64le::cairo-1.14.12-h8948797_3
  cffi               pkgs/main/linux-ppc64le::cffi-1.12.3-py37h2e261b9_0
  chardet            pkgs/main/linux-ppc64le::chardet-3.0.4-py37_1003
  click              pkgs/main/linux-ppc64le::click-7.0-py37_0
  cloudpickle        pkgs/main/noarch::cloudpickle-1.2.2-py_0
  coverage           pkgs/main/linux-ppc64le::coverage-5.0-py37h7b6447c_0
  cryptography       pkgs/main/linux-ppc64le::cryptography-2.8-py37h1ba5d50_0
  cudatoolkit        ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudatoolkit-10.1.243-616.gc122b8b
  cudnn              ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudnn-7.6.3_10.1-590.g5627c5e
  cycler             pkgs/main/linux-ppc64le::cycler-0.10.0-py37_0
  cytoolz            pkgs/main/linux-ppc64le::cytoolz-0.10.1-py37h7b6447c_0
  dask               pkgs/main/noarch::dask-2.3.0-py_0
  dask-core          pkgs/main/noarch::dask-core-2.3.0-py_0
  dask-cuda          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-cuda-0.9.1-py37_573.g9af8baa
  dask-xgboost       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-xgboost-0.1.7-py37_579.g8a31cf5
  ddl                ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::ddl-1.5.0-py37_1287.gc90c6f2
  ddl-tensorflow     ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::ddl-tensorflow-1.5.0-py37_1007.g8dbb51d
  decorator          pkgs/main/noarch::decorator-4.4.1-py_0
  distributed        pkgs/main/noarch::distributed-2.3.2-py_1
  ffmpeg             pkgs/main/linux-ppc64le::ffmpeg-4.0-hcdf2ecd_0
  fontconfig         pkgs/main/linux-ppc64le::fontconfig-2.13.0-h9420a91_0
  freeglut           pkgs/main/linux-ppc64le::freeglut-3.0.0-hf484d3e_5
  freetype           pkgs/main/linux-ppc64le::freetype-2.9.1-h8a8886c_0
  fsspec             pkgs/main/noarch::fsspec-0.6.2-py_0
  future             pkgs/main/linux-ppc64le::future-0.17.1-py37_0
  gast               pkgs/main/linux-ppc64le::gast-0.2.2-py37_0
  gflags             ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::gflags-2.2.2-1624.g17209b3
  glib               pkgs/main/linux-ppc64le::glib-2.63.1-h5a9c865_0
  glog               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::glog-0.3.5-1613.gd054598
  google-pasta       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::google-pasta-0.1.6-py37_564.g04df2d9
  graphite2          pkgs/main/linux-ppc64le::graphite2-1.3.13-h23475e2_0
  graphsurgeon       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::graphsurgeon-0.4.1-py37_612.gb2bf6b9
  grpcio             pkgs/main/linux-ppc64le::grpcio-1.16.1-py37hf8bcb03_1
  h5py               pkgs/main/linux-ppc64le::h5py-2.8.0-py37h8d01980_0
  harfbuzz           pkgs/main/linux-ppc64le::harfbuzz-1.8.8-hffaf4a1_0
  hdf5               pkgs/main/linux-ppc64le::hdf5-1.10.2-hba1933b_1
  heapdict           pkgs/main/noarch::heapdict-1.0.1-py_0
  hypothesis         pkgs/main/linux-ppc64le::hypothesis-3.59.1-py37h39e3cac_0
  icu                pkgs/main/linux-ppc64le::icu-58.2-h64fc554_1
  idna               pkgs/main/linux-ppc64le::idna-2.8-py37_0
  imageio            pkgs/main/linux-ppc64le::imageio-2.6.1-py37_0
  importlib_metadata pkgs/main/linux-ppc64le::importlib_metadata-1.4.0-py37_0
  jasper             pkgs/main/linux-ppc64le::jasper-2.0.14-h07fcdf6_1
  jinja2             pkgs/main/noarch::jinja2-2.10.3-py_0
  joblib             pkgs/main/linux-ppc64le::joblib-0.13.2-py37_0
  jpeg               pkgs/main/linux-ppc64le::jpeg-9b-hcb7ba68_2
  keras-applications pkgs/main/noarch::keras-applications-1.0.8-py_0
  keras-preprocessi~ pkgs/main/noarch::keras-preprocessing-1.1.0-py_1
  kiwisolver         pkgs/main/linux-ppc64le::kiwisolver-1.1.0-py37he6710b0_0
  leveldb            pkgs/main/linux-ppc64le::leveldb-1.20-hf484d3e_1
  libboost           pkgs/main/linux-ppc64le::libboost-1.67.0-h46d08c1_4
  libgfortran-ng     pkgs/main/linux-ppc64le::libgfortran-ng-7.3.0-h822a55f_1
  libglu             pkgs/main/linux-ppc64le::libglu-9.0.0-hf484d3e_1
  libopenblas        pkgs/main/linux-ppc64le::libopenblas-0.3.6-h5a2b251_1
  libopencv          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libopencv-3.4.7-725.g92aa195
  libopus            pkgs/main/linux-ppc64le::libopus-1.3-h7b6447c_0
  libpng             pkgs/main/linux-ppc64le::libpng-1.6.37-hbc83047_0
  libprotobuf        ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libprotobuf-3.8.0-577.g45759bb
  libtiff            pkgs/main/linux-ppc64le::libtiff-4.1.0-h2733197_0
  libuuid            pkgs/main/linux-ppc64le::libuuid-1.0.3-h1bed415_2
  libvpx             pkgs/main/linux-ppc64le::libvpx-1.7.0-hf484d3e_0
  libxcb             pkgs/main/linux-ppc64le::libxcb-1.13-h1bed415_0
  libxgboost-base    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libxgboost-base-0.90-gpu_590.g8a21f75
  libxml2            pkgs/main/linux-ppc64le::libxml2-2.9.9-hea5a465_1
  llvmlite           pkgs/main/linux-ppc64le::llvmlite-0.29.0-py37hd408876_0
  lmdb               pkgs/main/linux-ppc64le::lmdb-0.9.22-hf484d3e_1
  locket             pkgs/main/linux-ppc64le::locket-0.2.0-py37_1
  markdown           pkgs/main/linux-ppc64le::markdown-3.1.1-py37_0
  markupsafe         pkgs/main/linux-ppc64le::markupsafe-1.1.1-py37h7b6447c_0
  matplotlib         pkgs/main/linux-ppc64le::matplotlib-3.1.2-py37_1
  matplotlib-base    pkgs/main/linux-ppc64le::matplotlib-base-3.1.2-py37h4fdacc2_1
  mock               pkgs/main/linux-ppc64le::mock-2.0.0-py37_0
  more-itertools     pkgs/main/noarch::more-itertools-8.0.2-py_0
  msgpack-python     pkgs/main/linux-ppc64le::msgpack-python-0.6.1-py37hfd86e86_1
  nccl               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::nccl-2.4.8-586.gdba67b7
  networkx           pkgs/main/linux-ppc64le::networkx-2.2-py37_1
  ninja              pkgs/main/linux-ppc64le::ninja-1.9.0-py37hfd86e86_0
  nomkl              pkgs/main/linux-ppc64le::nomkl-3.0-0
  numactl            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::numactl-2.0.12-573.gdf5dc62
  numba              pkgs/main/linux-ppc64le::numba-0.45.1-py37h962f231_0
  numpy              pkgs/main/linux-ppc64le::numpy-1.16.6-py37h30dfecb_0
  numpy-base         pkgs/main/linux-ppc64le::numpy-base-1.16.6-py37h2f8d375_0
  olefile            pkgs/main/linux-ppc64le::olefile-0.46-py37_0
  onnx               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::onnx-1.5.0-py37_614.gd049fd7
  openblas           pkgs/main/linux-ppc64le::openblas-0.3.6-1
  openblas-devel     pkgs/main/linux-ppc64le::openblas-devel-0.3.6-1
  opencv             ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::opencv-3.4.7-py37_725.g92aa195
  packaging          pkgs/main/noarch::packaging-20.1-py_0
  pai4sk             ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pai4sk-1.5.0-py37_1071.g5abf42e
  pandas             pkgs/main/linux-ppc64le::pandas-1.0.0-py37h0573a6f_0
  partd              pkgs/main/noarch::partd-1.1.0-py_0
  pbr                pkgs/main/noarch::pbr-5.4.4-py_0
  pciutils           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pciutils-3.6.2-571.g2316d13
  pcre               pkgs/main/linux-ppc64le::pcre-8.43-he6710b0_0
  pillow             pkgs/main/linux-ppc64le::pillow-6.2.1-py37h0d2faf8_0
  pixman             pkgs/main/linux-ppc64le::pixman-0.34.0-h1f8d8dc_3
  pluggy             pkgs/main/linux-ppc64le::pluggy-0.13.1-py37_0
  powerai            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-1.6.2-615.g1dade79
  powerai-license    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-license-1.6.2-716.g7081e12
  powerai-release    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-release-1.6.2-572.gb216c2c
  powerai-tools      ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-tools-1.6.2-565.g97f2c3f
  protobuf           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::protobuf-3.8.0-py37_587.gab45ad3
  psutil             pkgs/main/linux-ppc64le::psutil-5.5.0-py37h7b6447c_0
  py                 pkgs/main/noarch::py-1.8.1-py_0
  py-boost           pkgs/main/linux-ppc64le::py-boost-1.67.0-py37h04863e7_4
  py-opencv          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-opencv-3.4.7-py37_725.g92aa195
  py-xgboost-base    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-xgboost-base-0.90-gpu_py37_590.g8a21f75
  py-xgboost-gpu     ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-xgboost-gpu-0.90-590.g8a21f75
  pycparser          pkgs/main/linux-ppc64le::pycparser-2.19-py37_0
  pyopenssl          pkgs/main/linux-ppc64le::pyopenssl-19.1.0-py37_0
  pyparsing          pkgs/main/noarch::pyparsing-2.4.6-py_0
  pysocks            pkgs/main/linux-ppc64le::pysocks-1.7.1-py37_0
  pytest             pkgs/main/linux-ppc64le::pytest-4.4.2-py37_0
  python-dateutil    pkgs/main/noarch::python-dateutil-2.8.1-py_0
  python-lmdb        pkgs/main/linux-ppc64le::python-lmdb-0.94-py37h14c3975_0
  pytorch            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pytorch-1.2.0-20238.g1faf942
  pytorch-base       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pytorch-base-1.2.0-gpu_py37_20238.g1faf942
  pytz               pkgs/main/noarch::pytz-2019.3-py_0
  pywavelets         pkgs/main/linux-ppc64le::pywavelets-1.1.1-py37h7b6447c_0
  pyyaml             pkgs/main/linux-ppc64le::pyyaml-5.1.2-py37h7b6447c_0
  requests           pkgs/main/linux-ppc64le::requests-2.22.0-py37_1
  scikit-image       pkgs/main/linux-ppc64le::scikit-image-0.15.0-py37he6710b0_0
  scikit-learn       pkgs/main/linux-ppc64le::scikit-learn-0.21.3-py37h22eb022_0
  scipy              pkgs/main/linux-ppc64le::scipy-1.3.1-py37he2b7bc3_0
  simsearch          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::simsearch-1.1.0-py37_764.g7c5f6cf
  six                pkgs/main/linux-ppc64le::six-1.12.0-py37_0
  snapml-spark       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::snapml-spark-1.4.0-py37_942.gc873569
  snappy             pkgs/main/linux-ppc64le::snappy-1.1.7-h1532aa0_3
  sortedcontainers   pkgs/main/linux-ppc64le::sortedcontainers-2.1.0-py37_0
  spectrum-mpi       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::spectrum-mpi-10.03-622.gfc88b70
  tabulate           pkgs/main/linux-ppc64le::tabulate-0.8.2-py37_0
  tblib              pkgs/main/noarch::tblib-1.6.0-py_0
  tensorboard        ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorboard-1.15.0-py37_ab7f72a_3645.gf4f525e
  tensorflow         ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-1.15.0-gpu_py37_841.g50de12c
  tensorflow-base    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-base-1.15.0-gpu_py37_590d6ee_64210.g4a039ec
  tensorflow-estima~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-estimator-1.15.1-py37_a5f60ce_1351.g50de12c
  tensorflow-gpu     ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-gpu-1.15.0-841.g50de12c
  tensorflow-large-~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-large-model-support-2.0.2-py37_970.gfa57a9e
  tensorflow-probab~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-probability-0.8.0-py37_b959b26_2686.g50de12c
  tensorflow-servin~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-serving-api-1.15.0-py37_748217e_5094.g89559ef
  tensorrt           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorrt-6.0.1.5-py37_612.gb2bf6b9
  termcolor          pkgs/main/linux-ppc64le::termcolor-1.1.0-py37_1
  tf_cnn_benchmarks  ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tf_cnn_benchmarks-1.15-gpu_py37_1374.g5e94b18
  toolz              pkgs/main/noarch::toolz-0.10.0-py_0
  torchtext          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::torchtext-0.4.0-py37_578.g5bf3960
  torchvision-base   ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::torchvision-base-0.4.0-gpu_py37_593.g80f339d
  tornado            pkgs/main/linux-ppc64le::tornado-6.0.3-py37h7b6447c_0
  tqdm               pkgs/main/noarch::tqdm-4.32.1-py_0
  typing             pkgs/main/linux-ppc64le::typing-3.6.4-py37_0
  uff                ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::uff-0.6.5-py37_612.gb2bf6b9
  urllib3            pkgs/main/linux-ppc64le::urllib3-1.25.8-py37_0
  werkzeug           pkgs/main/noarch::werkzeug-0.15.4-py_0
  wrapt              pkgs/main/linux-ppc64le::wrapt-1.11.2-py37h7b6447c_0
  yaml               pkgs/main/linux-ppc64le::yaml-0.1.7-h1bed415_2
  zict               pkgs/main/noarch::zict-1.0.0-py_0
  zipp               pkgs/main/noarch::zipp-0.6.0-py_0
  zstd               pkgs/main/linux-ppc64le::zstd-1.3.7-h0b5b093_0


Proceed ([y]/n)? y
...

아래와 같이 설치된 package들을 각각 확인하시면 됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep tensorflow
ddl-tensorflow            1.5.0           py37_1007.g8dbb51d    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow                1.15.0          gpu_py37_841.g50de12c    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-base           1.15.0          gpu_py37_590d6ee_64210.g4a039ec    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-estimator      1.15.1          py37_a5f60ce_1351.g50de12c    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-gpu            1.15.0             841.g50de12c    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-large-model-support 2.0.2           py37_970.gfa57a9e    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-probability    0.8.0           py37_b959b26_2686.g50de12c    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-serving-api    1.15.0          py37_748217e_5094.g89559ef    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep caffe
caffe                     1.0_1.6.2         5184.g7b10df4    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
caffe-base                1.0_1.6.2       gpu_py37_5184.g7b10df4    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep pytorch
_pytorch_select           2.0             gpu_20238.g1faf942    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
pytorch                   1.2.0            20238.g1faf942    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
pytorch-base              1.2.0           gpu_py37_20238.g1faf942    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda


WML-CE에서는 NCCL와 CUDNN 등 base facility도 함께 제공되어 설치됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep -i nccl
nccl                      2.4.8              586.gdba67b7    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep -i dnn
cudnn                     7.6.3_10.1         590.g5627c5e    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

아래와 같이 python에서 import를 해보셔도 됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ python
Python 3.7.5 (default, Oct 25 2019, 16:29:01)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> import tensorflow as tf
2020-02-03 21:31:31.095570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
>>>


다만 "conda install powerai" 만으로는 RAPIDS까지 설치되지는 않기 때문에, 아래와 같이 별도로 설치하셔야 합니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda install powerai-rapids
...
The following NEW packages will be INSTALLED:

  arrow-cpp          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::arrow-cpp-0.15.1-py37_603.g702c836
  boost-cpp          pkgs/main/linux-ppc64le::boost-cpp-1.67.0-h14c3975_4
  brotli             pkgs/main/linux-ppc64le::brotli-1.0.6-he6710b0_0
  cudf               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudf-0.9.0-cuda10.1_py37_626.gddcad2d
  cuml               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cuml-0.9.1-cuda10.1_py37_605.gfe9e07b
  cupy               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cupy-6.2.0-py37_567.g0f1e2ef
  cython             pkgs/main/linux-ppc64le::cython-0.29.14-py37he6710b0_0
  dask-cudf          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-cudf-0.9.0-py37_575.g0416adf
  dlpack             ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dlpack-0.2-562.g28dffd9
  double-conversion  ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::double-conversion-3.1.5-564.g4b43169
  fastavro           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::fastavro-0.22.4-py37_562.g9525976
  fastrlock          pkgs/main/linux-ppc64le::fastrlock-0.4-py37he6710b0_0
  grpc-cpp           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::grpc-cpp-1.23.0-568.g4f71a06
  libcudf            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libcudf-0.9.0-cuda10.1_609.g113236a
  libcuml            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libcuml-0.9.1-cuda10.1_576.ga304a0a
  libevent           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libevent-2.1.8-561.ge1d98f7
  libnvstrings       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libnvstrings-0.9.0-cuda10.1_570.ga04797c
  librmm             ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::librmm-0.9.0-cuda10.1_567.gff1b1a1
  lz4-c              pkgs/main/linux-ppc64le::lz4-c-1.8.1.2-h14c3975_0
  nvstrings          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::nvstrings-0.9.0-cuda10.1_py37_580.gdbb6546
  parquet-cpp        ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::parquet-cpp-1.5.1-579.g6eecc60
  powerai-rapids     ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-rapids-1.6.2-560.ga7c5a47
  pyarrow            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pyarrow-0.15.1-py37_609.g3a6717a
  re2                ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::re2-2019.08.01-561.gef92448
  rmm                ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::rmm-0.9.0-cuda10.1_py37_569.g04c75fb
  thrift-cpp         ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::thrift-cpp-0.12.0-580.gf96fa62
  uriparser          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::uriparser-0.9.3-561.g7465fef

The following packages will be DOWNGRADED:

  pandas                               1.0.0-py37h0573a6f_0 --> 0.24.2-py37he6710b0_0


Proceed ([y]/n)? y
...


만약 GPU가 없는 시스템에서 tensorflow나 pytorch 등을 사용하시고자 할 때는, 아래와 같이 CPU-only 버전의 WML-CE를 설치하시면 됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda install powerai-cpu

WLM-CE 설치시 "failed with repodata from current_repodata.json" error에 대한 work-around

최신 버전의 Anaconda를 설치하고 거기에 포함된 conda를 이용해서 Watson Machine Learning Community Edition (WML-CE, 예전 이름 PowerAI)를 설치할 때 다음과 같이 error가 나는 경우가 있습니다. 


(base) [user607@p607-kvm2 ~]$ conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/


(base) [user607@p607-kvm2 ~]$ conda info

     active environment : base
    active env location : /home/user607/anaconda3
            shell level : 1
       user config file : /home/user607/.condarc
 populated config files : /home/user607/.condarc
          conda version : 4.7.12
    conda-build version : 3.18.9
         python version : 3.7.4.final.0
       virtual packages :
       base environment : /home/user607/anaconda3  (writable)
           channel URLs : https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le
                          https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/noarch
                          https://repo.anaconda.com/pkgs/main/linux-ppc64le
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-ppc64le
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/user607/anaconda3/pkgs
                          /home/user607/.conda/pkgs
       envs directories : /home/user607/anaconda3/envs
                          /home/user607/.conda/envs
               platform : linux-ppc64le
             user-agent : conda/4.7.12 requests/2.22.0 CPython/3.7.4 Linux/4.18.0-80.el8.ppc64le rhel/8.0 glibc/2.28
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

(base) [user607@p607-kvm2 ~]$ conda install powerai
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \ failed


제 개인적인 경험으로 요약하면 아직 python 3.7 이상에서는 WLM-CE가 error를 내는 경우가 종종 있는 것 같습니다.  가장 좋은 work-around는 python 3.6.8의 virtual env를 만들어서 그것으로 사용하시는 것입니다.   다음과 같이 매우 간단하고, WLM-CE도 잘 설치됩니다.


(base) [user607@p607-kvm2 ~]$ conda create -n py368 python=3.6.8

(base) [user607@p607-kvm2 ~]$ conda activate py368

(py368) [user607@p607-kvm2 ~]$ conda install powerai
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/user607/anaconda3/envs/py368

  added / updated specs:
    - powerai


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _py-xgboost-mutex-1.0      | gpu_590.g8a21f75           7 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    _pytorch_select-2.0        |gpu_20238.g1faf942           3 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    _tflow_select-2.1.0        | gpu_840.g50de12c           3 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    absl-py-0.7.1              |           py36_0         157 KB
    apex-0.1.0_1.6.2           |py36_596.g1eb5c77         1.6 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    asn1crypto-1.2.0           |           py36_0         161 KB
    astor-0.7.1                |           py36_0          43 KB
    atomicwrites-1.3.0         |           py36_1          12 KB
    attrs-19.3.0               |             py_0          39 KB
    bokeh-1.3.4                |           py36_0         4.0 MB
    boost-1.67.0               |           py36_4          11 KB
    c-ares-1.15.0              |    h7b6447c_1001         107 KB
    caffe-1.0_1.6.2            |    5184.g7b10df4           3 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    caffe-base-1.0_1.6.2       |gpu_py36_5184.g7b10df4        27.6 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    cffi-1.12.3                |   py36h2e261b9_0         236 KB
    chardet-3.0.4              |        py36_1003         197 KB
    click-7.0                  |           py36_0         118 KB
    coverage-4.5.4             |   py36h7b6447c_0         227 KB
    cryptography-2.8           |   py36h1ba5d50_0         579 KB
    cudatoolkit-10.1.243       |     616.gc122b8b       510.3 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    cudnn-7.6.3_10.1           |     590.g5627c5e       472.5 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    cycler-0.10.0              |           py36_0          13 KB
    cytoolz-0.10.0             |   py36h7b6447c_0         388 KB
    dask-2.3.0                 |             py_0          12 KB
    dask-core-2.3.0            |             py_0         577 KB
    dask-cuda-0.9.1            |py36_573.g9af8baa          25 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    dask-xgboost-0.1.7         |py36_579.g8a31cf5          23 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    ddl-1.5.0                  |py36_1287.gc90c6f2         775 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    ddl-tensorflow-1.5.0       |py36_1007.g8dbb51d         2.4 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    decorator-4.4.1            |             py_0          13 KB
    distributed-2.3.2          |             py_1         374 KB
    ffmpeg-4.0                 |       hcdf2ecd_0        66.2 MB
    freeglut-3.0.0             |       hf484d3e_5         308 KB
    future-0.17.1              |           py36_0         698 KB
    gast-0.2.2                 |           py36_0         138 KB
    gflags-2.2.2               |    1624.g17209b3         238 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    glog-0.3.5                 |    1613.gd054598         161 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    google-pasta-0.1.6         |py36_564.g04df2d9          82 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    graphite2-1.3.13           |       h23475e2_0         109 KB
    graphsurgeon-0.4.1         |py36_612.gb2bf6b9          27 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    grpcio-1.16.1              |   py36hf8bcb03_1         1.1 MB
    h5py-2.8.0                 |   py36h8d01980_0         962 KB
    harfbuzz-1.8.8             |       hffaf4a1_0        1001 KB
    hypothesis-3.59.1          |   py36h39e3cac_0         352 KB
    idna-2.8                   |           py36_0         133 KB
    imageio-2.6.1              |           py36_0         3.4 MB
    importlib_metadata-0.23    |           py36_0          43 KB
    jasper-2.0.14              |       h07fcdf6_1         1.4 MB
    joblib-0.13.2              |           py36_0         365 KB
    keras-applications-1.0.8   |             py_0          33 KB
    keras-preprocessing-1.1.0  |             py_1          36 KB
    kiwisolver-1.1.0           |   py36he6710b0_0         101 KB
    leveldb-1.20               |       hf484d3e_1         394 KB
    libboost-1.67.0            |       h46d08c1_4        22.1 MB
    libglu-9.0.0               |       hf484d3e_1         635 KB
    libopencv-3.4.7            |     725.g92aa195        17.1 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    libopus-1.3                |       h7b6447c_0         832 KB
    libprotobuf-3.8.0          |     577.g45759bb         6.6 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    libvpx-1.7.0               |       hf484d3e_0         2.6 MB
    libxgboost-base-0.90       | gpu_590.g8a21f75       168.4 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    llvmlite-0.29.0            |   py36hd408876_0        16.6 MB
    lmdb-0.9.22                |       hf484d3e_1         926 KB
    locket-0.2.0               |           py36_1           8 KB
    markdown-3.1.1             |           py36_0         113 KB
    markupsafe-1.1.1           |   py36h7b6447c_0          30 KB
    matplotlib-3.1.1           |   py36h5429711_0         6.7 MB
    mock-2.0.0                 |           py36_0         104 KB
    more-itertools-7.2.0       |           py36_0          99 KB
    msgpack-python-0.6.1       |   py36hfd86e86_1          95 KB
    nccl-2.4.8                 |     586.gdba67b7       137.4 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    networkx-2.2               |           py36_1         2.0 MB
    ninja-1.9.0                |   py36hfd86e86_0         1.9 MB
    numactl-2.0.12             |     573.gdf5dc62         139 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    numba-0.45.1               |   py36h962f231_0         3.2 MB
    numpy-1.16.5               |   py36h99e49ec_0          49 KB
    numpy-base-1.16.5          |   py36h2f8d375_0         4.5 MB
    olefile-0.46               |           py36_0          48 KB
    onnx-1.5.0                 |py36_614.gd049fd7         2.9 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    opencv-3.4.7               |py36_725.g92aa195           4 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    pai4sk-1.5.0               |py36_1067.gc6c35a2         7.6 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    pandas-0.25.2              |   py36he6710b0_0        10.7 MB
    pbr-5.4.3                  |             py_0          78 KB
    pciutils-3.6.2             |     571.g2316d13         324 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    pillow-6.2.0               |   py36h0d2faf8_0         693 KB
    pluggy-0.13.0              |           py36_0          31 KB
    powerai-1.6.2              |     615.g1dade79           3 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    powerai-license-1.6.2      |     716.g7081e12         6.4 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    powerai-release-1.6.2      |     572.gb216c2c           3 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    powerai-tools-1.6.2        |     565.g97f2c3f           7 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    protobuf-3.8.0             |py36_587.gab45ad3         699 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    psutil-5.5.0               |   py36h7b6447c_0         317 KB
    py-1.8.0                   |           py36_0         140 KB
    py-boost-1.67.0            |   py36h04863e7_4         358 KB
    py-opencv-3.4.7            |py36_725.g92aa195         1.5 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    py-xgboost-base-0.90       |gpu_py36_590.g8a21f75        84.7 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    py-xgboost-gpu-0.90        |     590.g8a21f75           3 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    pycparser-2.19             |           py36_0         173 KB
    pyopenssl-19.0.0           |           py36_0          81 KB
    pysocks-1.7.1              |           py36_0          30 KB
    pytest-4.4.2               |           py36_0         358 KB
    python-dateutil-2.8.0      |           py36_0         281 KB
    python-lmdb-0.94           |   py36h14c3975_0         142 KB
    pytorch-1.2.0              |   20238.g1faf942           3 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    pytorch-base-1.2.0         |gpu_py36_20238.g1faf942       517.1 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    pywavelets-1.1.1           |   py36h7b6447c_0         4.4 MB
    pyyaml-5.1.2               |   py36h7b6447c_0         174 KB
    requests-2.22.0            |           py36_0          89 KB
    scikit-image-0.15.0        |   py36he6710b0_0        28.2 MB
    scikit-learn-0.21.3        |   py36h22eb022_0         4.6 MB
    scipy-1.3.0                |   py36he2b7bc3_0        19.3 MB
    simsearch-1.1.0            |py36_764.g7c5f6cf        26.0 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    six-1.12.0                 |           py36_0          22 KB
    snapml-spark-1.4.0         |py36_942.gc873569         1.6 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    sortedcontainers-2.1.0     |           py36_0          45 KB
    spectrum-mpi-10.03         |     622.gfc88b70        22.3 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    tabulate-0.8.2             |           py36_0          36 KB
    tbb-2019.8                 |       hfd86e86_0         998 KB
    tensorboard-1.15.0         |py36_3634.g50de12c         3.8 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    tensorflow-1.15.0          |gpu_py36_841.g50de12c           4 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    tensorflow-base-1.15.0     |gpu_py36_590d6ee_64210.g4a039ec       571.9 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    tensorflow-estimator-1.15.1|py36_a5f60ce_1351.g50de12c         665 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    tensorflow-gpu-1.15.0      |     841.g50de12c           3 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    tensorflow-large-model-support-2.0.2|py36_970.gfa57a9e          60 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    tensorflow-probability-0.8.0|py36_b959b26_2686.g50de12c         2.1 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    tensorflow-serving-api-1.15.0|py36_748217e_5094.g89559ef          36 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    tensorrt-6.0.1.5           |py36_612.gb2bf6b9       437.3 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    termcolor-1.1.0            |           py36_1           7 KB
    tf_cnn_benchmarks-1.15     |gpu_py36_1374.g5e94b18         189 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    torchtext-0.4.0            |py36_578.g5bf3960         1.1 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    torchvision-base-0.4.0     |gpu_py36_593.g80f339d         3.0 MB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    tornado-6.0.3              |   py36h7b6447c_0         642 KB
    tqdm-4.32.1                |             py_0          48 KB
    typing-3.6.4               |           py36_0          44 KB
    uff-0.6.5                  |py36_612.gb2bf6b9          79 KB  https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
    urllib3-1.24.2             |           py36_0         152 KB
    werkzeug-0.15.4            |             py_0         262 KB
    wrapt-1.11.2               |   py36h7b6447c_0          48 KB
    ------------------------------------------------------------
                                           Total:        3.18 GB

The following NEW packages will be INSTALLED:

  _py-xgboost-mutex  ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_py-xgboost-mutex-1.0-gpu_590.g8a21f75
  _pytorch_select    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_pytorch_select-2.0-gpu_20238.g1faf942
  _tflow_select      ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_tflow_select-2.1.0-gpu_840.g50de12c
  absl-py            pkgs/main/linux-ppc64le::absl-py-0.7.1-py36_0
  apex               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::apex-0.1.0_1.6.2-py36_596.g1eb5c77
  asn1crypto         pkgs/main/linux-ppc64le::asn1crypto-1.2.0-py36_0
  astor              pkgs/main/linux-ppc64le::astor-0.7.1-py36_0
  atomicwrites       pkgs/main/linux-ppc64le::atomicwrites-1.3.0-py36_1
  attrs              pkgs/main/noarch::attrs-19.3.0-py_0
  blas               pkgs/main/linux-ppc64le::blas-1.0-openblas
  bokeh              pkgs/main/linux-ppc64le::bokeh-1.3.4-py36_0
  boost              pkgs/main/linux-ppc64le::boost-1.67.0-py36_4
  bzip2              pkgs/main/linux-ppc64le::bzip2-1.0.8-h7b6447c_0
  c-ares             pkgs/main/linux-ppc64le::c-ares-1.15.0-h7b6447c_1001
  caffe              ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::caffe-1.0_1.6.2-5184.g7b10df4
  caffe-base         ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::caffe-base-1.0_1.6.2-gpu_py36_5184.g7b10df4
  cairo              pkgs/main/linux-ppc64le::cairo-1.14.12-h8948797_3
  cffi               pkgs/main/linux-ppc64le::cffi-1.12.3-py36h2e261b9_0
  chardet            pkgs/main/linux-ppc64le::chardet-3.0.4-py36_1003
  click              pkgs/main/linux-ppc64le::click-7.0-py36_0
  cloudpickle        pkgs/main/noarch::cloudpickle-1.2.2-py_0
  coverage           pkgs/main/linux-ppc64le::coverage-4.5.4-py36h7b6447c_0
  cryptography       pkgs/main/linux-ppc64le::cryptography-2.8-py36h1ba5d50_0
  cudatoolkit        ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudatoolkit-10.1.243-616.gc122b8b
  cudnn              ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudnn-7.6.3_10.1-590.g5627c5e
  cycler             pkgs/main/linux-ppc64le::cycler-0.10.0-py36_0
  cytoolz            pkgs/main/linux-ppc64le::cytoolz-0.10.0-py36h7b6447c_0
  dask               pkgs/main/noarch::dask-2.3.0-py_0
  dask-core          pkgs/main/noarch::dask-core-2.3.0-py_0
  dask-cuda          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-cuda-0.9.1-py36_573.g9af8baa
  dask-xgboost       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-xgboost-0.1.7-py36_579.g8a31cf5
  ddl                ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::ddl-1.5.0-py36_1287.gc90c6f2
  ddl-tensorflow     ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::ddl-tensorflow-1.5.0-py36_1007.g8dbb51d
  decorator          pkgs/main/noarch::decorator-4.4.1-py_0
  distributed        pkgs/main/noarch::distributed-2.3.2-py_1
  ffmpeg             pkgs/main/linux-ppc64le::ffmpeg-4.0-hcdf2ecd_0
  fontconfig         pkgs/main/linux-ppc64le::fontconfig-2.13.0-h9420a91_0
  freeglut           pkgs/main/linux-ppc64le::freeglut-3.0.0-hf484d3e_5
  freetype           pkgs/main/linux-ppc64le::freetype-2.9.1-h8a8886c_0
  fsspec             pkgs/main/noarch::fsspec-0.5.2-py_0
  future             pkgs/main/linux-ppc64le::future-0.17.1-py36_0
  gast               pkgs/main/linux-ppc64le::gast-0.2.2-py36_0
  gflags             ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::gflags-2.2.2-1624.g17209b3
  glib               pkgs/main/linux-ppc64le::glib-2.56.2-hd408876_0
  glog               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::glog-0.3.5-1613.gd054598
  google-pasta       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::google-pasta-0.1.6-py36_564.g04df2d9
  graphite2          pkgs/main/linux-ppc64le::graphite2-1.3.13-h23475e2_0
  graphsurgeon       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::graphsurgeon-0.4.1-py36_612.gb2bf6b9
  grpcio             pkgs/main/linux-ppc64le::grpcio-1.16.1-py36hf8bcb03_1
  h5py               pkgs/main/linux-ppc64le::h5py-2.8.0-py36h8d01980_0
  harfbuzz           pkgs/main/linux-ppc64le::harfbuzz-1.8.8-hffaf4a1_0
  hdf5               pkgs/main/linux-ppc64le::hdf5-1.10.2-hba1933b_1
  heapdict           pkgs/main/noarch::heapdict-1.0.1-py_0
  hypothesis         pkgs/main/linux-ppc64le::hypothesis-3.59.1-py36h39e3cac_0
  icu                pkgs/main/linux-ppc64le::icu-58.2-h64fc554_1
  idna               pkgs/main/linux-ppc64le::idna-2.8-py36_0
  imageio            pkgs/main/linux-ppc64le::imageio-2.6.1-py36_0
  importlib_metadata pkgs/main/linux-ppc64le::importlib_metadata-0.23-py36_0
  jasper             pkgs/main/linux-ppc64le::jasper-2.0.14-h07fcdf6_1
  jinja2             pkgs/main/noarch::jinja2-2.10.3-py_0
  joblib             pkgs/main/linux-ppc64le::joblib-0.13.2-py36_0
  jpeg               pkgs/main/linux-ppc64le::jpeg-9b-hcb7ba68_2
  keras-applications pkgs/main/noarch::keras-applications-1.0.8-py_0
  keras-preprocessi~ pkgs/main/noarch::keras-preprocessing-1.1.0-py_1
  kiwisolver         pkgs/main/linux-ppc64le::kiwisolver-1.1.0-py36he6710b0_0
  leveldb            pkgs/main/linux-ppc64le::leveldb-1.20-hf484d3e_1
  libboost           pkgs/main/linux-ppc64le::libboost-1.67.0-h46d08c1_4
  libgfortran-ng     pkgs/main/linux-ppc64le::libgfortran-ng-7.3.0-h822a55f_1
  libglu             pkgs/main/linux-ppc64le::libglu-9.0.0-hf484d3e_1
  libopenblas        pkgs/main/linux-ppc64le::libopenblas-0.3.6-h5a2b251_1
  libopencv          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libopencv-3.4.7-725.g92aa195
  libopus            pkgs/main/linux-ppc64le::libopus-1.3-h7b6447c_0
  libpng             pkgs/main/linux-ppc64le::libpng-1.6.37-hbc83047_0
  libprotobuf        ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libprotobuf-3.8.0-577.g45759bb
  libtiff            pkgs/main/linux-ppc64le::libtiff-4.0.10-h2733197_2
  libuuid            pkgs/main/linux-ppc64le::libuuid-1.0.3-h1bed415_2
  libvpx             pkgs/main/linux-ppc64le::libvpx-1.7.0-hf484d3e_0
  libxcb             pkgs/main/linux-ppc64le::libxcb-1.13-h1bed415_0
  libxgboost-base    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libxgboost-base-0.90-gpu_590.g8a21f75
  libxml2            pkgs/main/linux-ppc64le::libxml2-2.9.9-hea5a465_1
  llvmlite           pkgs/main/linux-ppc64le::llvmlite-0.29.0-py36hd408876_0
  lmdb               pkgs/main/linux-ppc64le::lmdb-0.9.22-hf484d3e_1
  locket             pkgs/main/linux-ppc64le::locket-0.2.0-py36_1
  markdown           pkgs/main/linux-ppc64le::markdown-3.1.1-py36_0
  markupsafe         pkgs/main/linux-ppc64le::markupsafe-1.1.1-py36h7b6447c_0
  matplotlib         pkgs/main/linux-ppc64le::matplotlib-3.1.1-py36h5429711_0
  mock               pkgs/main/linux-ppc64le::mock-2.0.0-py36_0
  more-itertools     pkgs/main/linux-ppc64le::more-itertools-7.2.0-py36_0
  msgpack-python     pkgs/main/linux-ppc64le::msgpack-python-0.6.1-py36hfd86e86_1
  nccl               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::nccl-2.4.8-586.gdba67b7
  networkx           pkgs/main/linux-ppc64le::networkx-2.2-py36_1
  ninja              pkgs/main/linux-ppc64le::ninja-1.9.0-py36hfd86e86_0
  nomkl              pkgs/main/linux-ppc64le::nomkl-3.0-0
  numactl            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::numactl-2.0.12-573.gdf5dc62
  numba              pkgs/main/linux-ppc64le::numba-0.45.1-py36h962f231_0
  numpy              pkgs/main/linux-ppc64le::numpy-1.16.5-py36h99e49ec_0
  numpy-base         pkgs/main/linux-ppc64le::numpy-base-1.16.5-py36h2f8d375_0
  olefile            pkgs/main/linux-ppc64le::olefile-0.46-py36_0
  onnx               ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::onnx-1.5.0-py36_614.gd049fd7
  openblas           pkgs/main/linux-ppc64le::openblas-0.3.6-1
  openblas-devel     pkgs/main/linux-ppc64le::openblas-devel-0.3.6-1
  opencv             ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::opencv-3.4.7-py36_725.g92aa195
  packaging          pkgs/main/noarch::packaging-19.2-py_0
  pai4sk             ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pai4sk-1.5.0-py36_1067.gc6c35a2
  pandas             pkgs/main/linux-ppc64le::pandas-0.25.2-py36he6710b0_0
  partd              pkgs/main/noarch::partd-1.0.0-py_0
  pbr                pkgs/main/noarch::pbr-5.4.3-py_0
  pciutils           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pciutils-3.6.2-571.g2316d13
  pcre               pkgs/main/linux-ppc64le::pcre-8.43-he6710b0_0
  pillow             pkgs/main/linux-ppc64le::pillow-6.2.0-py36h0d2faf8_0
  pixman             pkgs/main/linux-ppc64le::pixman-0.34.0-h1f8d8dc_3
  pluggy             pkgs/main/linux-ppc64le::pluggy-0.13.0-py36_0
  powerai            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-1.6.2-615.g1dade79
  powerai-license    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-license-1.6.2-716.g7081e12
  powerai-release    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-release-1.6.2-572.gb216c2c
  powerai-tools      ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-tools-1.6.2-565.g97f2c3f
  protobuf           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::protobuf-3.8.0-py36_587.gab45ad3
  psutil             pkgs/main/linux-ppc64le::psutil-5.5.0-py36h7b6447c_0
  py                 pkgs/main/linux-ppc64le::py-1.8.0-py36_0
  py-boost           pkgs/main/linux-ppc64le::py-boost-1.67.0-py36h04863e7_4
  py-opencv          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-opencv-3.4.7-py36_725.g92aa195
  py-xgboost-base    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-xgboost-base-0.90-gpu_py36_590.g8a21f75
  py-xgboost-gpu     ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-xgboost-gpu-0.90-590.g8a21f75
  pycparser          pkgs/main/linux-ppc64le::pycparser-2.19-py36_0
  pyopenssl          pkgs/main/linux-ppc64le::pyopenssl-19.0.0-py36_0
  pyparsing          pkgs/main/noarch::pyparsing-2.4.2-py_0
  pysocks            pkgs/main/linux-ppc64le::pysocks-1.7.1-py36_0
  pytest             pkgs/main/linux-ppc64le::pytest-4.4.2-py36_0
  python-dateutil    pkgs/main/linux-ppc64le::python-dateutil-2.8.0-py36_0
  python-lmdb        pkgs/main/linux-ppc64le::python-lmdb-0.94-py36h14c3975_0
  pytorch            ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pytorch-1.2.0-20238.g1faf942
  pytorch-base       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pytorch-base-1.2.0-gpu_py36_20238.g1faf942
  pytz               pkgs/main/noarch::pytz-2019.3-py_0
  pywavelets         pkgs/main/linux-ppc64le::pywavelets-1.1.1-py36h7b6447c_0
  pyyaml             pkgs/main/linux-ppc64le::pyyaml-5.1.2-py36h7b6447c_0
  requests           pkgs/main/linux-ppc64le::requests-2.22.0-py36_0
  scikit-image       pkgs/main/linux-ppc64le::scikit-image-0.15.0-py36he6710b0_0
  scikit-learn       pkgs/main/linux-ppc64le::scikit-learn-0.21.3-py36h22eb022_0
  scipy              pkgs/main/linux-ppc64le::scipy-1.3.0-py36he2b7bc3_0
  simsearch          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::simsearch-1.1.0-py36_764.g7c5f6cf
  six                pkgs/main/linux-ppc64le::six-1.12.0-py36_0
  snapml-spark       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::snapml-spark-1.4.0-py36_942.gc873569
  snappy             pkgs/main/linux-ppc64le::snappy-1.1.7-h1532aa0_3
  sortedcontainers   pkgs/main/linux-ppc64le::sortedcontainers-2.1.0-py36_0
  spectrum-mpi       ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::spectrum-mpi-10.03-622.gfc88b70
  tabulate           pkgs/main/linux-ppc64le::tabulate-0.8.2-py36_0
  tbb                pkgs/main/linux-ppc64le::tbb-2019.8-hfd86e86_0
  tblib              pkgs/main/noarch::tblib-1.4.0-py_0
  tensorboard        ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorboard-1.15.0-py36_3634.g50de12c
  tensorflow         ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-1.15.0-gpu_py36_841.g50de12c
  tensorflow-base    ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-base-1.15.0-gpu_py36_590d6ee_64210.g4a039ec
  tensorflow-estima~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-estimator-1.15.1-py36_a5f60ce_1351.g50de12c
  tensorflow-gpu     ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-gpu-1.15.0-841.g50de12c
  tensorflow-large-~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-large-model-support-2.0.2-py36_970.gfa57a9e
  tensorflow-probab~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-probability-0.8.0-py36_b959b26_2686.g50de12c
  tensorflow-servin~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-serving-api-1.15.0-py36_748217e_5094.g89559ef
  tensorrt           ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorrt-6.0.1.5-py36_612.gb2bf6b9
  termcolor          pkgs/main/linux-ppc64le::termcolor-1.1.0-py36_1
  tf_cnn_benchmarks  ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tf_cnn_benchmarks-1.15-gpu_py36_1374.g5e94b18
  toolz              pkgs/main/noarch::toolz-0.10.0-py_0
  torchtext          ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::torchtext-0.4.0-py36_578.g5bf3960
  torchvision-base   ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::torchvision-base-0.4.0-gpu_py36_593.g80f339d
  tornado            pkgs/main/linux-ppc64le::tornado-6.0.3-py36h7b6447c_0
  tqdm               pkgs/main/noarch::tqdm-4.32.1-py_0
  typing             pkgs/main/linux-ppc64le::typing-3.6.4-py36_0
  uff                ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::uff-0.6.5-py36_612.gb2bf6b9
  urllib3            pkgs/main/linux-ppc64le::urllib3-1.24.2-py36_0
  werkzeug           pkgs/main/noarch::werkzeug-0.15.4-py_0
  wrapt              pkgs/main/linux-ppc64le::wrapt-1.11.2-py36h7b6447c_0
  yaml               pkgs/main/linux-ppc64le::yaml-0.1.7-h1bed415_2
  zict               pkgs/main/noarch::zict-1.0.0-py_0
  zipp               pkgs/main/noarch::zipp-0.6.0-py_0
  zstd               pkgs/main/linux-ppc64le::zstd-1.3.7-h0b5b093_0


Proceed ([y]/n)? y
...