HW 엔지니어를 위한 Deep Learning: Anaconda

레이블이 Anaconda인 게시물을 표시합니다. 모든 게시물 표시

2020년 2월 4일 화요일

WML-CE (Watson Machine Learning Community Edition, 구 PowerAI) 1.6.2 설치

IBM POWER 아키텍처 (POWER8/9, 즉 ppc64le)에서 tensorflow나 caffe 등 각종 deep learning framework을 제공해주던 무료 toolkit인 기존 PowerAI는 이미 다들 아시는 바와 같이 이름을 Watson Machine Learning Community Edition (WML-CE)로 변경했습니다. 물론 여전히 무료입니다만, 기존처럼 *.rpm이나 *.deb의 형태로 제공하지 않고 아예 별도의 conda channel을 만들어서 conda에서 설치하도록 하고 있습니다. 따라서, Anaconda가 prerequsite이며, 2020년 2월 초 현재 최신 버전인 1.6.2는 Ananconda 2019.07을 prerequisite으로 하고 있습니다. 아예 모든 것이 설치된 docker image 형태로도 제공됩니다.

자세한 원본 manual은 아래 link를 참조하시면 됩니다.

https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.2/navigation/wmlce_planning.html
https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.2/navigation/wmlce_install.html

여기서는 ppc64le Ubuntu 18.04, Python 3.7.5 환경에서 WML-CE 1.6.2를 설치해보겠습니다.

cecuser@p1234-kvm1:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

먼저 Anaconda 2019.07 버전을 download 받아 설치합니다.

cecuser@p1234-kvm1:~$ wget https://repo.continuum.io/archive/Anaconda3-2019.07-Linux-ppc64le.sh

cecuser@p1234-kvm1:~$ chmod a+x Anaconda3-2019.07-Linux-ppc64le.sh

cecuser@p1234-kvm1:~$ ./Anaconda3-2019.07-Linux-ppc64le.sh

설치가 끝나면 ~/.bashrc를 수행하여 conda init을 수행합니다.

cecuser@p1234-kvm1:~$ . ~/.bashrc

이제 IBM이 제공하는 WML-CE를 위한 conda channel을 conda에 추가합니다.

(base) cecuser@p1234-kvm1:~$ conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

Conda 가상 환경을 생성하여 거기에 WML-CE를 설치하기를 권장하므로, 먼저 python 3.7.5 환경으로 wmlce_env라는 이름의 virtual env를 만듭니다.

(base) cecuser@p1234-kvm1:~$ conda create --name wmlce_env python=3.7.5
...
The following NEW packages will be INSTALLED:

_libgcc_mutex pkgs/main/linux-ppc64le::_libgcc_mutex-0.1-main
ca-certificates pkgs/main/linux-ppc64le::ca-certificates-2020.1.1-0
certifi pkgs/main/linux-ppc64le::certifi-2019.11.28-py37_0
libedit pkgs/main/linux-ppc64le::libedit-3.1.20181209-hc058e9b_0
libffi pkgs/main/linux-ppc64le::libffi-3.2.1-hf62a594_5
libgcc-ng pkgs/main/linux-ppc64le::libgcc-ng-8.2.0-h822a55f_1
libstdcxx-ng pkgs/main/linux-ppc64le::libstdcxx-ng-8.2.0-h822a55f_1
ncurses pkgs/main/linux-ppc64le::ncurses-6.1-he6710b0_1
openssl pkgs/main/linux-ppc64le::openssl-1.1.1d-h7b6447c_3
pip pkgs/main/linux-ppc64le::pip-20.0.2-py37_1
python pkgs/main/linux-ppc64le::python-3.7.5-h4134adf_0
readline pkgs/main/linux-ppc64le::readline-7.0-h7b6447c_5
setuptools pkgs/main/linux-ppc64le::setuptools-45.1.0-py37_0
sqlite pkgs/main/linux-ppc64le::sqlite-3.30.1-h7b6447c_0
tk pkgs/main/linux-ppc64le::tk-8.6.8-hbc83047_0
wheel pkgs/main/linux-ppc64le::wheel-0.34.1-py37_0
xz pkgs/main/linux-ppc64le::xz-5.2.4-h14c3975_4
zlib pkgs/main/linux-ppc64le::zlib-1.2.11-h7b6447c_3

Proceed ([y]/n)? y
...
Downloading and Extracting Packages
ca-certificates-2020 | 125 KB | ##################################### | 100%
setuptools-45.1.0 | 511 KB | ##################################### | 100%
pip-20.0.2 | 1.7 MB | ##################################### | 100%
sqlite-3.30.1 | 2.3 MB | ##################################### | 100%
wheel-0.34.1 | 50 KB | ##################################### | 100%
python-3.7.5 | 32.5 MB | ##################################### | 100%
openssl-1.1.1d | 3.8 MB | ##################################### | 100%
certifi-2019.11.28 | 156 KB | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate wmlce_env
#
# To deactivate an active environment, use
#
# $ conda deactivate

이제 conda 가상환경인 wmlce_env를 활성화합니다.

(base) cecuser@p1234-kvm1:~$ conda activate wmlce_env

이제 WML-CE를 설치합니다. Conda package 이름은 여전히 PowerAI로 되어 있는 점에 유의하십시요. 아래와 같이 하면 tensorflow와 caffe2, pytorch 등 WML-CE에서 지원하는 모든 deep learning framework이 한꺼번에 다 설치됩니다. 혹시 WML-CE 전체를 설치하지 않고 가령 PyTorch만 설치하려 할 때는 그냥 conda install pytorch 라고 하시면 됩니다.

아래의 명령어로 어느어느 package들이 설치되는지 보여드리기 위해 긴 ouput을 일부러 다 옮겨 붙였습니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda install powerai
....
The following NEW packages will be INSTALLED:

_py-xgboost-mutex ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_py-xgboost-mutex-1.0-gpu_590.g8a21f75
_pytorch_select ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_pytorch_select-2.0-gpu_20238.g1faf942
_tflow_select ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_tflow_select-2.1.0-gpu_840.g50de12c
absl-py pkgs/main/linux-ppc64le::absl-py-0.7.1-py37_0
apex ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::apex-0.1.0_1.6.2-py37_596.g1eb5c77
asn1crypto pkgs/main/linux-ppc64le::asn1crypto-1.3.0-py37_0
astor pkgs/main/linux-ppc64le::astor-0.7.1-py37_0
atomicwrites pkgs/main/linux-ppc64le::atomicwrites-1.3.0-py37_1
attrs pkgs/main/noarch::attrs-19.3.0-py_0
blas pkgs/main/linux-ppc64le::blas-1.0-openblas
bokeh pkgs/main/linux-ppc64le::bokeh-1.4.0-py37_0
boost pkgs/main/linux-ppc64le::boost-1.67.0-py37_4
bzip2 pkgs/main/linux-ppc64le::bzip2-1.0.8-h7b6447c_0
c-ares pkgs/main/linux-ppc64le::c-ares-1.15.0-h7b6447c_1001
caffe ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::caffe-1.0_1.6.2-5184.g7b10df4
caffe-base ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::caffe-base-1.0_1.6.2-gpu_py37_5184.g7b10df4
cairo pkgs/main/linux-ppc64le::cairo-1.14.12-h8948797_3
cffi pkgs/main/linux-ppc64le::cffi-1.12.3-py37h2e261b9_0
chardet pkgs/main/linux-ppc64le::chardet-3.0.4-py37_1003
click pkgs/main/linux-ppc64le::click-7.0-py37_0
cloudpickle pkgs/main/noarch::cloudpickle-1.2.2-py_0
coverage pkgs/main/linux-ppc64le::coverage-5.0-py37h7b6447c_0
cryptography pkgs/main/linux-ppc64le::cryptography-2.8-py37h1ba5d50_0
cudatoolkit ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudatoolkit-10.1.243-616.gc122b8b
cudnn ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudnn-7.6.3_10.1-590.g5627c5e
cycler pkgs/main/linux-ppc64le::cycler-0.10.0-py37_0
cytoolz pkgs/main/linux-ppc64le::cytoolz-0.10.1-py37h7b6447c_0
dask pkgs/main/noarch::dask-2.3.0-py_0
dask-core pkgs/main/noarch::dask-core-2.3.0-py_0
dask-cuda ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-cuda-0.9.1-py37_573.g9af8baa
dask-xgboost ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-xgboost-0.1.7-py37_579.g8a31cf5
ddl ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::ddl-1.5.0-py37_1287.gc90c6f2
ddl-tensorflow ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::ddl-tensorflow-1.5.0-py37_1007.g8dbb51d
decorator pkgs/main/noarch::decorator-4.4.1-py_0
distributed pkgs/main/noarch::distributed-2.3.2-py_1
ffmpeg pkgs/main/linux-ppc64le::ffmpeg-4.0-hcdf2ecd_0
fontconfig pkgs/main/linux-ppc64le::fontconfig-2.13.0-h9420a91_0
freeglut pkgs/main/linux-ppc64le::freeglut-3.0.0-hf484d3e_5
freetype pkgs/main/linux-ppc64le::freetype-2.9.1-h8a8886c_0
fsspec pkgs/main/noarch::fsspec-0.6.2-py_0
future pkgs/main/linux-ppc64le::future-0.17.1-py37_0
gast pkgs/main/linux-ppc64le::gast-0.2.2-py37_0
gflags ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::gflags-2.2.2-1624.g17209b3
glib pkgs/main/linux-ppc64le::glib-2.63.1-h5a9c865_0
glog ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::glog-0.3.5-1613.gd054598
google-pasta ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::google-pasta-0.1.6-py37_564.g04df2d9
graphite2 pkgs/main/linux-ppc64le::graphite2-1.3.13-h23475e2_0
graphsurgeon ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::graphsurgeon-0.4.1-py37_612.gb2bf6b9
grpcio pkgs/main/linux-ppc64le::grpcio-1.16.1-py37hf8bcb03_1
h5py pkgs/main/linux-ppc64le::h5py-2.8.0-py37h8d01980_0
harfbuzz pkgs/main/linux-ppc64le::harfbuzz-1.8.8-hffaf4a1_0
hdf5 pkgs/main/linux-ppc64le::hdf5-1.10.2-hba1933b_1
heapdict pkgs/main/noarch::heapdict-1.0.1-py_0
hypothesis pkgs/main/linux-ppc64le::hypothesis-3.59.1-py37h39e3cac_0
icu pkgs/main/linux-ppc64le::icu-58.2-h64fc554_1
idna pkgs/main/linux-ppc64le::idna-2.8-py37_0
imageio pkgs/main/linux-ppc64le::imageio-2.6.1-py37_0
importlib_metadata pkgs/main/linux-ppc64le::importlib_metadata-1.4.0-py37_0
jasper pkgs/main/linux-ppc64le::jasper-2.0.14-h07fcdf6_1
jinja2 pkgs/main/noarch::jinja2-2.10.3-py_0
joblib pkgs/main/linux-ppc64le::joblib-0.13.2-py37_0
jpeg pkgs/main/linux-ppc64le::jpeg-9b-hcb7ba68_2
keras-applications pkgs/main/noarch::keras-applications-1.0.8-py_0
keras-preprocessi~ pkgs/main/noarch::keras-preprocessing-1.1.0-py_1
kiwisolver pkgs/main/linux-ppc64le::kiwisolver-1.1.0-py37he6710b0_0
leveldb pkgs/main/linux-ppc64le::leveldb-1.20-hf484d3e_1
libboost pkgs/main/linux-ppc64le::libboost-1.67.0-h46d08c1_4
libgfortran-ng pkgs/main/linux-ppc64le::libgfortran-ng-7.3.0-h822a55f_1
libglu pkgs/main/linux-ppc64le::libglu-9.0.0-hf484d3e_1
libopenblas pkgs/main/linux-ppc64le::libopenblas-0.3.6-h5a2b251_1
libopencv ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libopencv-3.4.7-725.g92aa195
libopus pkgs/main/linux-ppc64le::libopus-1.3-h7b6447c_0
libpng pkgs/main/linux-ppc64le::libpng-1.6.37-hbc83047_0
libprotobuf ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libprotobuf-3.8.0-577.g45759bb
libtiff pkgs/main/linux-ppc64le::libtiff-4.1.0-h2733197_0
libuuid pkgs/main/linux-ppc64le::libuuid-1.0.3-h1bed415_2
libvpx pkgs/main/linux-ppc64le::libvpx-1.7.0-hf484d3e_0
libxcb pkgs/main/linux-ppc64le::libxcb-1.13-h1bed415_0
libxgboost-base ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libxgboost-base-0.90-gpu_590.g8a21f75
libxml2 pkgs/main/linux-ppc64le::libxml2-2.9.9-hea5a465_1
llvmlite pkgs/main/linux-ppc64le::llvmlite-0.29.0-py37hd408876_0
lmdb pkgs/main/linux-ppc64le::lmdb-0.9.22-hf484d3e_1
locket pkgs/main/linux-ppc64le::locket-0.2.0-py37_1
markdown pkgs/main/linux-ppc64le::markdown-3.1.1-py37_0
markupsafe pkgs/main/linux-ppc64le::markupsafe-1.1.1-py37h7b6447c_0
matplotlib pkgs/main/linux-ppc64le::matplotlib-3.1.2-py37_1
matplotlib-base pkgs/main/linux-ppc64le::matplotlib-base-3.1.2-py37h4fdacc2_1
mock pkgs/main/linux-ppc64le::mock-2.0.0-py37_0
more-itertools pkgs/main/noarch::more-itertools-8.0.2-py_0
msgpack-python pkgs/main/linux-ppc64le::msgpack-python-0.6.1-py37hfd86e86_1
nccl ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::nccl-2.4.8-586.gdba67b7
networkx pkgs/main/linux-ppc64le::networkx-2.2-py37_1
ninja pkgs/main/linux-ppc64le::ninja-1.9.0-py37hfd86e86_0
nomkl pkgs/main/linux-ppc64le::nomkl-3.0-0
numactl ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::numactl-2.0.12-573.gdf5dc62
numba pkgs/main/linux-ppc64le::numba-0.45.1-py37h962f231_0
numpy pkgs/main/linux-ppc64le::numpy-1.16.6-py37h30dfecb_0
numpy-base pkgs/main/linux-ppc64le::numpy-base-1.16.6-py37h2f8d375_0
olefile pkgs/main/linux-ppc64le::olefile-0.46-py37_0
onnx ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::onnx-1.5.0-py37_614.gd049fd7
openblas pkgs/main/linux-ppc64le::openblas-0.3.6-1
openblas-devel pkgs/main/linux-ppc64le::openblas-devel-0.3.6-1
opencv ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::opencv-3.4.7-py37_725.g92aa195
packaging pkgs/main/noarch::packaging-20.1-py_0
pai4sk ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pai4sk-1.5.0-py37_1071.g5abf42e
pandas pkgs/main/linux-ppc64le::pandas-1.0.0-py37h0573a6f_0
partd pkgs/main/noarch::partd-1.1.0-py_0
pbr pkgs/main/noarch::pbr-5.4.4-py_0
pciutils ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pciutils-3.6.2-571.g2316d13
pcre pkgs/main/linux-ppc64le::pcre-8.43-he6710b0_0
pillow pkgs/main/linux-ppc64le::pillow-6.2.1-py37h0d2faf8_0
pixman pkgs/main/linux-ppc64le::pixman-0.34.0-h1f8d8dc_3
pluggy pkgs/main/linux-ppc64le::pluggy-0.13.1-py37_0
powerai ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-1.6.2-615.g1dade79
powerai-license ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-license-1.6.2-716.g7081e12
powerai-release ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-release-1.6.2-572.gb216c2c
powerai-tools ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-tools-1.6.2-565.g97f2c3f
protobuf ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::protobuf-3.8.0-py37_587.gab45ad3
psutil pkgs/main/linux-ppc64le::psutil-5.5.0-py37h7b6447c_0
py pkgs/main/noarch::py-1.8.1-py_0
py-boost pkgs/main/linux-ppc64le::py-boost-1.67.0-py37h04863e7_4
py-opencv ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-opencv-3.4.7-py37_725.g92aa195
py-xgboost-base ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-xgboost-base-0.90-gpu_py37_590.g8a21f75
py-xgboost-gpu ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-xgboost-gpu-0.90-590.g8a21f75
pycparser pkgs/main/linux-ppc64le::pycparser-2.19-py37_0
pyopenssl pkgs/main/linux-ppc64le::pyopenssl-19.1.0-py37_0
pyparsing pkgs/main/noarch::pyparsing-2.4.6-py_0
pysocks pkgs/main/linux-ppc64le::pysocks-1.7.1-py37_0
pytest pkgs/main/linux-ppc64le::pytest-4.4.2-py37_0
python-dateutil pkgs/main/noarch::python-dateutil-2.8.1-py_0
python-lmdb pkgs/main/linux-ppc64le::python-lmdb-0.94-py37h14c3975_0
pytorch ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pytorch-1.2.0-20238.g1faf942
pytorch-base ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pytorch-base-1.2.0-gpu_py37_20238.g1faf942
pytz pkgs/main/noarch::pytz-2019.3-py_0
pywavelets pkgs/main/linux-ppc64le::pywavelets-1.1.1-py37h7b6447c_0
pyyaml pkgs/main/linux-ppc64le::pyyaml-5.1.2-py37h7b6447c_0
requests pkgs/main/linux-ppc64le::requests-2.22.0-py37_1
scikit-image pkgs/main/linux-ppc64le::scikit-image-0.15.0-py37he6710b0_0
scikit-learn pkgs/main/linux-ppc64le::scikit-learn-0.21.3-py37h22eb022_0
scipy pkgs/main/linux-ppc64le::scipy-1.3.1-py37he2b7bc3_0
simsearch ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::simsearch-1.1.0-py37_764.g7c5f6cf
six pkgs/main/linux-ppc64le::six-1.12.0-py37_0
snapml-spark ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::snapml-spark-1.4.0-py37_942.gc873569
snappy pkgs/main/linux-ppc64le::snappy-1.1.7-h1532aa0_3
sortedcontainers pkgs/main/linux-ppc64le::sortedcontainers-2.1.0-py37_0
spectrum-mpi ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::spectrum-mpi-10.03-622.gfc88b70
tabulate pkgs/main/linux-ppc64le::tabulate-0.8.2-py37_0
tblib pkgs/main/noarch::tblib-1.6.0-py_0
tensorboard ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorboard-1.15.0-py37_ab7f72a_3645.gf4f525e
tensorflow ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-1.15.0-gpu_py37_841.g50de12c
tensorflow-base ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-base-1.15.0-gpu_py37_590d6ee_64210.g4a039ec
tensorflow-estima~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-estimator-1.15.1-py37_a5f60ce_1351.g50de12c
tensorflow-gpu ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-gpu-1.15.0-841.g50de12c
tensorflow-large-~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-large-model-support-2.0.2-py37_970.gfa57a9e
tensorflow-probab~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-probability-0.8.0-py37_b959b26_2686.g50de12c
tensorflow-servin~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-serving-api-1.15.0-py37_748217e_5094.g89559ef
tensorrt ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorrt-6.0.1.5-py37_612.gb2bf6b9
termcolor pkgs/main/linux-ppc64le::termcolor-1.1.0-py37_1
tf_cnn_benchmarks ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tf_cnn_benchmarks-1.15-gpu_py37_1374.g5e94b18
toolz pkgs/main/noarch::toolz-0.10.0-py_0
torchtext ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::torchtext-0.4.0-py37_578.g5bf3960
torchvision-base ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::torchvision-base-0.4.0-gpu_py37_593.g80f339d
tornado pkgs/main/linux-ppc64le::tornado-6.0.3-py37h7b6447c_0
tqdm pkgs/main/noarch::tqdm-4.32.1-py_0
typing pkgs/main/linux-ppc64le::typing-3.6.4-py37_0
uff ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::uff-0.6.5-py37_612.gb2bf6b9
urllib3 pkgs/main/linux-ppc64le::urllib3-1.25.8-py37_0
werkzeug pkgs/main/noarch::werkzeug-0.15.4-py_0
wrapt pkgs/main/linux-ppc64le::wrapt-1.11.2-py37h7b6447c_0
yaml pkgs/main/linux-ppc64le::yaml-0.1.7-h1bed415_2
zict pkgs/main/noarch::zict-1.0.0-py_0
zipp pkgs/main/noarch::zipp-0.6.0-py_0
zstd pkgs/main/linux-ppc64le::zstd-1.3.7-h0b5b093_0

Proceed ([y]/n)? y
...

아래와 같이 설치된 package들을 각각 확인하시면 됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep tensorflow
ddl-tensorflow 1.5.0 py37_1007.g8dbb51d https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow 1.15.0 gpu_py37_841.g50de12c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-base 1.15.0 gpu_py37_590d6ee_64210.g4a039ec https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-estimator 1.15.1 py37_a5f60ce_1351.g50de12c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-gpu 1.15.0 841.g50de12c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-large-model-support 2.0.2 py37_970.gfa57a9e https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-probability 0.8.0 py37_b959b26_2686.g50de12c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-serving-api 1.15.0 py37_748217e_5094.g89559ef https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep caffe
caffe 1.0_1.6.2 5184.g7b10df4 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
caffe-base 1.0_1.6.2 gpu_py37_5184.g7b10df4 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep pytorch
_pytorch_select 2.0 gpu_20238.g1faf942 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
pytorch 1.2.0 20238.g1faf942 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
pytorch-base 1.2.0 gpu_py37_20238.g1faf942 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

WML-CE에서는 NCCL와 CUDNN 등 base facility도 함께 제공되어 설치됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep -i nccl
nccl 2.4.8 586.gdba67b7 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep -i dnn
cudnn 7.6.3_10.1 590.g5627c5e https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

아래와 같이 python에서 import를 해보셔도 됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ python
Python 3.7.5 (default, Oct 25 2019, 16:29:01)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> import tensorflow as tf
2020-02-03 21:31:31.095570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
>>>

다만 "conda install powerai" 만으로는 RAPIDS까지 설치되지는 않기 때문에, 아래와 같이 별도로 설치하셔야 합니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda install powerai-rapids
...
The following NEW packages will be INSTALLED:

arrow-cpp ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::arrow-cpp-0.15.1-py37_603.g702c836
boost-cpp pkgs/main/linux-ppc64le::boost-cpp-1.67.0-h14c3975_4
brotli pkgs/main/linux-ppc64le::brotli-1.0.6-he6710b0_0
cudf ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudf-0.9.0-cuda10.1_py37_626.gddcad2d
cuml ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cuml-0.9.1-cuda10.1_py37_605.gfe9e07b
cupy ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cupy-6.2.0-py37_567.g0f1e2ef
cython pkgs/main/linux-ppc64le::cython-0.29.14-py37he6710b0_0
dask-cudf ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-cudf-0.9.0-py37_575.g0416adf
dlpack ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dlpack-0.2-562.g28dffd9
double-conversion ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::double-conversion-3.1.5-564.g4b43169
fastavro ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::fastavro-0.22.4-py37_562.g9525976
fastrlock pkgs/main/linux-ppc64le::fastrlock-0.4-py37he6710b0_0
grpc-cpp ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::grpc-cpp-1.23.0-568.g4f71a06
libcudf ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libcudf-0.9.0-cuda10.1_609.g113236a
libcuml ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libcuml-0.9.1-cuda10.1_576.ga304a0a
libevent ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libevent-2.1.8-561.ge1d98f7
libnvstrings ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libnvstrings-0.9.0-cuda10.1_570.ga04797c
librmm ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::librmm-0.9.0-cuda10.1_567.gff1b1a1
lz4-c pkgs/main/linux-ppc64le::lz4-c-1.8.1.2-h14c3975_0
nvstrings ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::nvstrings-0.9.0-cuda10.1_py37_580.gdbb6546
parquet-cpp ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::parquet-cpp-1.5.1-579.g6eecc60
powerai-rapids ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-rapids-1.6.2-560.ga7c5a47
pyarrow ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pyarrow-0.15.1-py37_609.g3a6717a
re2 ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::re2-2019.08.01-561.gef92448
rmm ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::rmm-0.9.0-cuda10.1_py37_569.g04c75fb
thrift-cpp ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::thrift-cpp-0.12.0-580.gf96fa62
uriparser ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::uriparser-0.9.3-561.g7465fef

The following packages will be DOWNGRADED:

pandas 1.0.0-py37h0573a6f_0 --> 0.24.2-py37he6710b0_0

Proceed ([y]/n)? y
...

만약 GPU가 없는 시스템에서 tensorflow나 pytorch 등을 사용하시고자 할 때는, 아래와 같이 CPU-only 버전의 WML-CE를 설치하시면 됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda install powerai-cpu

2018년 11월 21일 수요일

Redhat ppc64le 환경에서의 boost와 pyarrow build 및 설치

Redhat 7.5의 기본 gcc 버전은 4.8.5입니다. OS의 주요 library들도 모두 gcc 4.8.5로 build되어 있습니다. 그런데 Anaconda의 v5.2부터는 python이 gcc 7.2.0으로 build되어 있습니다.

[bsyu@redhat74 cudf]$ python
Python 3.7.0 (default, Jun 28 2018, 13:02:24)
[GCC 7.2.0] :: Anaconda, Inc. on linux

이로 인해 벌어지는 문제 중 하나가, OS의 기본 gcc/g++을 사용해 build한 python package의 경우 아래처럼 undefined symbol error를 내는 경우가 많다는 것입니다.

ImportError: /home/bsyu/anaconda3/lib/python3.7/site-packages/pyarrow/libparquet.so.12: undefined symbol: _ZN5boost13match_resultsIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEESaINS_9sub_matchISB_EEEE12maybe_assignERKSF_

이는 gcc 5.1부터 CXXABI가 달라졌기 때문입니다. 그에 대한 자세한 내용은 아래에 나와 있습니다.

https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html

이 문제를 해결하려면 gcc 버전을 7.2로 올리는 것이 좋습니다. 그에 대해서는 아래 link를 따라 하시면 됩니다.

http://hwengineer.blogspot.com/2018/05/ppc64le-gcc7-cmake3-source-build.html

이제 gcc 7.2.0을 갖게 되었습니다.

[bsyu@redhat74 files]$ which gcc
/usr/local/bin/gcc

[bsyu@redhat74 files]$ gcc --version
gcc (GCC) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

OS 기본 gcc가 아닌, 새로 만든 gcc 7.2.0을 쓰기 위해서는 PATH와 LD_LIBRARY_PATH를 환경변수에서 /usr/local을 먼저 참조하도록 설정하십시요.

[bsyu@redhat74 files]$ cat ~/.bashrc
...
export PATH=/usr/local/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:$LD_LIBRARY_PATH
export MAKEFLAGS=-j16
export ARROW_BUILD_TYPE=release

이제 anaconda3를 설치하시고...

[bsyu@redhat74 files]$ ./Anaconda3-5.3.0-Linux-ppc64le.sh

[bsyu@redhat74 files]$ which python
~/anaconda3/bin/python

[bsyu@redhat74 files]$ python
Python 3.7.0 (default, Jun 28 2018, 13:02:24)
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.

pyarrow를 설치하기 위해서는 먼저 boost library들도 gcc 7.2로 build 해야 합니다. OS에서 기본 제공되는 것들은 맨 위의 예와 같이 undefined symbol error를 내기 때문입니다.

[bsyu@redhat74 files]$ wget https://dl.bintray.com/boostorg/release/1.68.0/source/boost_1_68_0.tar.gz

[bsyu@redhat74 files]$ tar -zxf boost_1_68_0.tar.gz

여기서 나중에 "pyconfig.h Not Found" error를 피하기 위해서 아래와 같이 pyconfig.h를 살짝 link를 걸어 줘야 합니다.

[bsyu@redhat74 include]$ ln -s python3.7m/pyconfig.h pyconfig.h

[bsyu@redhat74 files]$ cd boost_1_68_0

[bsyu@redhat74 boost_1_68_0]$ ./bootstrap.sh --with-python=/home/bsyu/anaconda3/bin/python --with-python-root=/home/bsyu/anaconda3

[bsyu@redhat74 boost_1_68_0]$ ./b2 -j16

[bsyu@redhat74 boost_1_68_0]$ sudo ./b2 install

이제 boost library들이 설치되었습니다. 이제 pyarrow를 build 합니다.

[bsyu@redhat74 files]$ conda install numpy six setuptools cython pandas pytest cmake flatbuffers rapidjson boost-cpp thrift snappy zlib gflags brotli lz4-c zstd -c conda-forge

(base) [bsyu@redhat74 files]$ which cmake
~/anaconda3/bin/cmake
(base) [bsyu@redhat74 files]$ cmake --version
cmake version 3.12.2

(base) [bsyu@redhat74 files]$ git clone https://github.com/apache/arrow.git

(base) [bsyu@redhat74 files]$ cd arrow

(base) [bsyu@redhat74 arrow]$ export CC=/usr/local/bin/gcc
(base) [bsyu@redhat74 arrow]$ export CXX=/usr/local/bin/g++

(base) [bsyu@redhat74 arrow]$ mkdir cpp/build && cd cpp/build

(base) [bsyu@redhat74 build]$ cmake -DCMAKE_BUILD_TYPE=release -DARROW_PYTHON=on -DARROW_PLASMA=on -DARROW_BUILD_TESTS=OFF -DARROW_PARQUET=ON ..

(base) [bsyu@redhat74 build]$ make -j16

(base) [bsyu@redhat74 build]$ sudo make install

(base) [bsyu@redhat74 build]$ cd ../../python

(base) [bsyu@redhat74 python]$ export ARROW_HOME=/usr/local

(base) [bsyu@redhat74 python]$ MAKEFLAGS=-j16 python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet --inplace

(base) [bsyu@redhat74 python]$ python setup.py build_ext --build-type=release --with-parquet --bundle-arrow-cpp bdist_wheel

위까지 마치면 dist directory 밑에 wheel file이 생성되어 있습니다.

(base) [bsyu@redhat74 python]$ ls -ltr dist
-rw-rw-r-- 1 bsyu bsyu 9645047 Nov 21 10:42 pyarrow-0.11.1.dev264+g7e6bf41.d20181121-cp37-cp37m-linux_ppc64le.whl

이제 이것을 pip로 설치합니다.

(base) [bsyu@redhat74 python]$ pip install ./dist/pyarrow-0.11.1.dev264+g7e6bf41.d20181121-cp37-cp37m-linux_ppc64le.whl

이제 import pyarrow를 해보면 맨 위의 예에서 보이던 undefined symbol error가 없어진 것을 확인하실 수 있습니다.

(base) [bsyu@redhat74 python]$ python
Python 3.7.0 (default, Jun 28 2018, 13:02:24)
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow
>>>

여기서 만든 pyarrow*.whl은 아래 google drive에 올려놓았습니다. python 3.7용입니다.

https://drive.google.com/open?id=16cF8F9A2a8ZtpnpV0PdckJMyotkVYEns

참고로 ldd로 보면, 이제 lib.cpython-37m-powerpc64le-linux-gnu.so가 OS에서 기본 제공하는 library (/usr/lib64/libstdc++.so.6, /usr/lib/libboost_regex.so 등 )가 아닌 /usr/local/lib64/libstdc++.so.6, /usr/local/lib/libboost_regex.so 등을 참조하는 것을 보실 수 있습니다.

(base) [bsyu@redhat74 ~]$ ldd /home/bsyu/anaconda3/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-powerpc64le-linux-gnu.so
linux-vdso64.so.1 => (0x00003fff9f750000)
libarrow.so.12 => /usr/local/lib64/libarrow.so.12 (0x00003fff9efc0000)
libarrow_python.so.12 => /usr/local/lib64/libarrow_python.so.12 (0x00003fff9ee70000)
libparquet.so.12 => /usr/local/lib64/libparquet.so.12 (0x00003fff9ebf0000)
libstdc++.so.6 => /usr/local/lib64/libstdc++.so.6 (0x00003fff9e9c0000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00003fff9e8d0000)
libgcc_s.so.1 => /usr/local/lib64/libgcc_s.so.1 (0x00003fff9e890000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00003fff9e6a0000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00003fff9e670000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00003fff9e630000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00003fff9e5f0000)
libboost_system.so.1.68.0 => /usr/local/lib/libboost_system.so.1.68.0 (0x00003fff9e5c0000)
libboost_filesystem.so.1.68.0 => /usr/local/lib/libboost_filesystem.so.1.68.0 (0x00003fff9e580000)
libboost_regex.so.1.68.0 => /usr/local/lib/libboost_regex.so.1.68.0 (0x00003fff9e440000)
librt.so.1 => /usr/lib64/librt.so.1 (0x00003fff9e410000)
/lib64/ld64.so.2 (0x000000007f930000)
libutil.so.1 => /usr/lib64/libutil.so.1 (0x00003fff9e3e0000)

2018년 2월 28일 수요일

AC922의 기본 setup 및 anaconda virutalenv, tensorflow 등의 설치

먼저, AC922에서 CUDA 버전이 아직 불안정한 관계로 일부 손볼 부분이 있습니다. 혹시 그것이 아직 설정되어 있지 않다면 아래 link를 참조하시어 설정하시기 바랍니다. 설정이 제대로 되었다면 "nvidia-smi -l 3" 명령으로 볼 때 아무 문제가 없을 것입니다. 혹시 GPU 사용%에 UNKNOW ERROR 등이 표시된다면 그 설정이 안 된 것입니다.

https://hwengineer.blogspot.kr/2018/01/ac922-ubuntu-container-image-caffe.html

ppc64le 환경에서 EPEL의 rpm은 다음과 같이 download 받습니다.

[root@ac922 ~]$ wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

이걸 다음과 같이 설치하면 /etc/yum.repos.d/epel.repo 등 epel repository 관련 file들이 설치됩니다.

[root@ac922 ~]# rpm -Uvh epel-release-latest-7.noarch.rpm

[root@ac922 ~]# rpm -ql epel-release-7-11.noarch
/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
/etc/yum.repos.d/epel-testing.repo
/etc/yum.repos.d/epel.repo
/usr/lib/systemd/system-preset/90-epel.preset
/usr/share/doc/epel-release-7
/usr/share/doc/epel-release-7/GPL

/etc/yum.repos.d/epel.repo의 내용을 보면 아래와 같습니다.

[root@ac922 ~]# vi /etc/yum.repos.d/epel.repo
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch/debug
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/7/SRPMS
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

이제 yum update를 수행한 뒤 yum list를 해보면 epel repository에 python34.ppc64le 등이 보입니다. 이걸 설치하시면 됩니다.

[root@ac922 ~]# yum update

[root@ac922 ~]# yum list | grep python34 | more
python34.ppc64le 3.4.5-5.el7 epel
python34-libs.ppc64le 3.4.5-5.el7 epel
libpeas-loader-python34.ppc64le 1.20.0-1.el7 epel
python34-Cython.ppc64le 0.23.5-1.el7 epel
...

[root@ac922 ~]# yum install python34
...
Running transaction
Installing : python34-3.4.5-5.el7.ppc64le 1/2
Installing : python34-libs-3.4.5-5.el7.ppc64le 2/2
Verifying : python34-libs-3.4.5-5.el7.ppc64le 1/2
Verifying : python34-3.4.5-5.el7.ppc64le 2/2

Installed:
python34.ppc64le 0:3.4.5-5.el7

Dependency Installed:
python34-libs.ppc64le 0:3.4.5-5.el7

Complete!

다만 보통 Redhat OS에 포함된 python34를 이용하기보다는 Anaconda를 많이 이용합니다.
Anaconda 설치는 이미 되어 있다고 가정하겠습니다. 안 되어 있다면 다음과 같이 받아서 저 file을 수행하시면 됩니다.

$ wget https://repo.continuum.io/archive/Anaconda2-5.1.0-Linux-ppc64le.sh
$ wget https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-ppc64le.sh

여기서는 Anaconda2를 사용한다고 가정하겠습니다.

[user1@ac922 ~]$ which conda
~/anaconda2/bin/conda

virutalenv는 다음과 같이 x86과 동일하게 생성하시면 됩니다. 여기서는 python 3.6 환경을 만들겠습니다.

[user1@ac922 ~]$ conda create -n env_py36 python=3.6 anaconda
...
isort 4.2.15: ###################################################################################### | 100%
six 1.11.0: ######################################################################################## | 100%
conda-build 3.4.1: ################################################################################# | 100%
numpy 1.13.3: ###################################################################################### | 100%
click 6.7: ######################################################################################### | 100%
send2trash 1.4.2: ################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate env_py36
#
# To deactivate an active environment, use:
# > source deactivate
#

virtualenv의 사용법도 x86에서와 동일합니다. 다만 pip 등의 이름이 pip3가 아니라 그냥 pip일 뿐입니다. pip 버전은 물론 python 3.6에 딸린 버전입니다.

[user1@ac922 ~]$ source activate env_py36

(env_py36) [user1@ac922 ~]$ which python
~/anaconda2/envs/env_py36/bin/python

(env_py36) [user1@ac922 ~]$ which pip
~/anaconda2/envs/env_py36/bin/pip

(env_py36) [user1@ac922 ~]$ pip --version
pip 9.0.1 from /home/user1/anaconda2/envs/env_py36/lib/python3.6/site-packages (python 3.6)

이제 jupyterhub나 sudospawner 등 필요하신 것을 그대로 설치하시면 됩니다.

(env_py36) [user1@ac922 ~]$ pip install jupyterhub
...
Successfully built alembic python-oauth2 Mako python-editor
Installing collected packages: pamela, Mako, python-editor, alembic, python-oauth2, jupyterhub
Successfully installed Mako-1.0.7 alembic-0.9.8 jupyterhub-0.8.1 pamela-0.3.0 python-editor-1.0.3 python-oauth2-1.1.0

(env_py36) [user1@ac922 ~]$ pip install sudospawner
...
Requirement already satisfied: parso==0.1.* in ./anaconda2/envs/env_py36/lib/python3.6/site-packages (from jedi>=0.10->ipython>=4.0.0->ipykernel->notebook->sudospawner)
Requirement already satisfied: wcwidth in ./anaconda2/envs/env_py36/lib/python3.6/site-packages (from prompt_toolkit<2.0.0,>=1.0.4->ipython>=4.0.0->ipykernel->notebook->sudospawner)
Installing collected packages: sudospawner
Successfully installed sudospawner-0.5.1

다만, tensorflow도 그대로 설치하시려 하면 아래처럼 error가 날 겁니다.

(env_py36) [user1@ac922 ~]$ pip install tensorflow
Collecting tensorflow
Could not find a version that satisfies the requirement tensorflow (from versions: )
No matching distribution found for tensorflow

이는 아직 ppc64le 환경에서는 CUDA 9을 위한 tensorflow 1.4.1은 없기 때문입니다. 대신 제가 source에서 빌드해놓은 wheel file이 있습니다. (혹시 필요하실 경우) tensorflow 1.4.1을 source에서 빌드하는 방법은 이 link를 참조하시면 됩니다. 제가 빌드한 tensorflow-1.4.1-cp36-cp36m-linux_ppc64le.whl 파일은 여기서 download 받으시면 됩니다.

(env_py36) [user1@ac922 ~]$ ls files
...
tensorflow-1.4.1-cp36-cp36m-linux_ppc64le.whl

이 wheel 파일을 다음과 같이 pip로 설치하시면 됩니다.

(env_py36) [user1@ac922 ~]$ pip install ./files/tensorflow-1.4.1-cp36-cp36m-linux_ppc64le.whl
Processing ./files/tensorflow-1.4.1-cp36-cp36m-linux_ppc64le.whl
Requirement already satisfied: six>=1.10.0 in ./anaconda2/envs/env_py36/lib/python3.6/site-packages (from tensorflow==1.4.1)
Requirement already satisfied: numpy>=1.12.1 in ./anaconda2/envs/env_py36/lib/python3.6/site-packages (from tensorflow==1.4.1)
Collecting protobuf>=3.3.0 (from tensorflow==1.4.1)
Downloading protobuf-3.5.1-py2.py3-none-any.whl (388kB)
100% |████████████████████████████████| 389kB 1.5MB/s
Collecting tensorflow-tensorboard<0.5.0,>=0.4.0rc1 (from tensorflow==1.4.1)
Downloading tensorflow_tensorboard-0.4.0-py3-none-any.whl (1.7MB)
100% |████████████████████████████████| 1.7MB 629kB/s
Collecting enum34>=1.1.6 (from tensorflow==1.4.1)
Downloading enum34-1.1.6-py3-none-any.whl
Requirement already satisfied: wheel>=0.26 in ./anaconda2/envs/env_py36/lib/python3.6/site-packages (from tensorflow==1.4.1)
Requirement already satisfied: setuptools in ./anaconda2/envs/env_py36/lib/python3.6/site-packages (from protobuf>=3.3.0->tensorflow==1.4.1)
Collecting bleach==1.5.0 (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow==1.4.1)
Downloading bleach-1.5.0-py2.py3-none-any.whl
Collecting html5lib==0.9999999 (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow==1.4.1)
Downloading html5lib-0.9999999.tar.gz (889kB)
100% |████████████████████████████████| 890kB 1.2MB/s
Collecting markdown>=2.6.8 (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow==1.4.1)
Downloading Markdown-2.6.11-py2.py3-none-any.whl (78kB)
100% |████████████████████████████████| 81kB 1.5MB/s
Requirement already satisfied: werkzeug>=0.11.10 in ./anaconda2/envs/env_py36/lib/python3.6/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow==1.4.1)
Building wheels for collected packages: html5lib
Running setup.py bdist_wheel for html5lib ... done
Stored in directory: /home/user1/.cache/pip/wheels/6f/85/6c/56b8e1292c6214c4eb73b9dda50f53e8e977bf65989373c962
Successfully built html5lib
Installing collected packages: protobuf, html5lib, bleach, markdown, tensorflow-tensorboard, enum34, tensorflow
Found existing installation: html5lib 1.0.1
Uninstalling html5lib-1.0.1:
Successfully uninstalled html5lib-1.0.1
Found existing installation: bleach 2.1.2
Uninstalling bleach-2.1.2:
Successfully uninstalled bleach-2.1.2
Successfully installed bleach-1.5.0 enum34-1.1.6 html5lib-0.9999999 markdown-2.6.11 protobuf-3.5.1 tensorflow-1.4.1 tensorflow-tensorboard-0.4.0

(env_py36) [user1@ac922 ~]$ pip list | grep tensorflow
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
tensorflow (1.4.1)
tensorflow-tensorboard (0.4.0)

이 tensorflow의 이름이 tensorflow-gpu가 아니라고 해서 이것이 CPU 버전의 tensorlfow인 것은 아닙니다. Source로부터 build한 것은 tensorlfow-gpu와 tensorflow가 따로 있진 않고, 이것도 GPU를 이용하도록 opt=cuda로 build한 것입니다. 다음과 같이 간단한 test로 tensorlfow가 V100 GPU를 제대로 물고 오는 것을 확인하실 수 있습니다.

(env_py36) [user1@ac922 ~]$ python
Python 3.6.4 |Anaconda, Inc.| (default, Feb 11 2018, 08:19:13)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> sess=tf.Session()
2018-02-28 15:08:07.063120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0004:04:00.0
totalMemory: 15.75GiB freeMemory: 15.33GiB
2018-02-28 15:08:07.432906: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 1 with properties:
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0004:05:00.0
totalMemory: 15.75GiB freeMemory: 15.34GiB
2018-02-28 15:08:07.856193: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 2 with properties:
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0035:03:00.0
totalMemory: 15.75GiB freeMemory: 15.33GiB
2018-02-28 15:08:08.288151: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 3 with properties:
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0035:04:00.0
totalMemory: 15.75GiB freeMemory: 15.34GiB
2018-02-28 15:08:08.292038: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Device peer to peer matrix
2018-02-28 15:08:08.295822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1051] DMA: 0 1 2 3
2018-02-28 15:08:08.295838: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 0: Y Y N N
2018-02-28 15:08:08.295851: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 1: Y Y N N
2018-02-28 15:08:08.295863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 2: N N Y Y
2018-02-28 15:08:08.295874: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 3: N N Y Y
2018-02-28 15:08:08.295894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0004:04:00.0, compute capability: 7.0)
2018-02-28 15:08:08.295909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0004:05:00.0, compute capability: 7.0)
2018-02-28 15:08:08.295921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:2) -> (device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0035:03:00.0, compute capability: 7.0)
2018-02-28 15:08:08.295933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:3) -> (device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0035:04:00.0, compute capability: 7.0)

이때 nvidia-smi로 다음과 같이 tensorflow가 GPU를 점유하는 것을 보실 수 있습니다.

Wed Feb 28 15:08:17 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.36 Driver Version: 387.36 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... On | 00000004:04:00.0 Off | 0 |
| N/A 37C P0 51W / 300W | 15340MiB / 16128MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... On | 00000004:05:00.0 Off | 0 |
| N/A 42C P0 52W / 300W | 15345MiB / 16128MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla V100-SXM2... On | 00000035:03:00.0 Off | 0 |
| N/A 39C P0 51W / 300W | 15343MiB / 16128MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla V100-SXM2... On | 00000035:04:00.0 Off | 0 |
| N/A 43C P0 52W / 300W | 15344MiB / 16128MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 10141 C python 15336MiB |
| 1 10141 C python 15334MiB |
| 2 10141 C python 15332MiB |
| 3 10141 C python 15338MiB |
+-----------------------------------------------------------------------------+

2017년 9월 15일 금요일

Minsky 서버에 Anaconda installer의 full package 설치하기

올 3월까지만 해도 Continuum사의 Anaconda installer가 Minsky의 아키텍처인 ppc64le 환경에서는 full package로는 없었고, Miniconda installer만 있었습니다. 그런데 그 사이에 ppc64le용 Anaconda installer의 full package가 이미 나왔네요.

https://repo.continuum.io/archive/

Filename Size Last Modified MD5
Anaconda2-4.4.0.1-Linux-ppc64le.sh 271.4M 2017-07-26 16:10:02 ce166de6f116acd08cd313f9c55c04d6
Anaconda3-4.4.0.1-Linux-ppc64le.sh 285.6M 2017-07-26 16:08:42 fe7c87abd9fd70dc0cb4f83cc22d336f

설치는 아래와 같이 하시면 됩니다. Python2.7 환경에서는 아래와 같이 Anaconda2-4.4.0.1-Linux-ppc64le.sh를 받으시면 되고, Python3 환경에서는 Anaconda3-4.4.0.1-Linux-ppc64le.sh를 받으시면 됩니다. 일단 둘다 받아서 설치하시는 것이 좋습니다. 여기서는 Anaconda2에 대한 부분만 써놓습니다.

u0017649@sys-89075:~$ wget https://repo.continuum.io/archive/Anaconda2-4.4.0.1-Linux-ppc64le.sh
--2017-09-14 21:42:44-- https://repo.continuum.io/archive/Anaconda2-4.4.0.1-Linux-ppc64le.sh
Resolving repo.continuum.io (repo.continuum.io)... 104.16.18.10, 104.16.19.10, 2400:cb00:2048:1::6810:120a, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.16.18.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 284629257 (271M) [application/x-sh]
Saving to: ‘Anaconda2-4.4.0.1-Linux-ppc64le.sh’

Anaconda2-4.4.0.1-L 100%[===================>] 271.44M 16.7MB/s in 17s

2017-09-14 21:43:01 (16.4 MB/s) - ‘Anaconda2-4.4.0.1-Linux-ppc64le.sh’ saved [284629257/284629257]

여러 user가 함께 쓰는 서버 환경에 Anaconda를 설치할 때는, 해당 user의 home directory에 설치하면 다른 user들이 쓸 수가 없지요. 따라서 일단은 /opt/anaconda2에 설치해놓고 user별로 clone을 해서 사용하시는 것이 좋습니다. 물론 /opt directory에 설치하려면 sudo 권한이 있어야 합니다.

u0017649@sys-89075:~$ chmod u+x Anaconda2-4.4.0.1-Linux-ppc64le.sh

u0017649@sys-89075:~$ sudo ./Anaconda2-4.4.0.1-Linux-ppc64le.sh

Do you approve the license terms? [yes|no]
>>> yes

Anaconda2 will now be installed into this location:
/home/u0017649/anaconda2

- Press ENTER to confirm the location
- Press CTRL-C to abort the installation
- Or specify a different location below

[/home/u0017649/anaconda2] >>> /opt/anaconda2

...
installing: zict-0.1.2-py27_0 ...
installing: zlib-1.2.8-3 ...
installing: anaconda-4.4.0-np112py27_0 ...
installing: conda-4.3.21-py27_0 ...
installing: conda-env-2.6.0-0 ...
Python 2.7.13 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
Do you wish the installer to prepend the Anaconda2 install location
to PATH in your /home/u0017649/.bashrc ? [yes|no]
[no] >>> yes

Prepending PATH=/opt/anaconda2/bin to PATH in /home/u0017649/.bashrc
A backup will be made to: /home/u0017649/.bashrc-anaconda2.bak

For this change to become active, you have to open a new terminal.

Thank you for installing Anaconda2!

Share your notebooks and packages on Anaconda Cloud!
Sign up for free: https://anaconda.org

이제 환경변수를 적용하고, conda install을 해봅니다.

u0017649@sys-89075:~$ . ./.bashrc

u0017649@sys-89075:~$ env | grep PATH
PATH=/opt/anaconda2/bin:/home/u0017649/bin:/home/u0017649/.local/bin:/opt/ibm/xlC/current/bin:/opt/ibm/xlf/current/bin:/opt/at10.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

u0017649@sys-89075:~$ conda install scikit-learn
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /opt/anaconda2:

The following packages will be UPDATED:

scikit-learn: 0.18.1-np112py27_1 --> 0.19.0-np112py27_0

Proceed ([y]/n)? y

CondaIOError: Missing write permissions in: /opt/anaconda2
#
# You don't appear to have the necessary permissions to install packages
# into the install area '/opt/anaconda2'.
# However you can clone this environment into your home directory and
# then make changes to it.
# This may be done using the command:
#
# $ conda create -n my_root --clone="/opt/anaconda2"

보시다시피 /opt/anaconda2에 대해서 일반 user들은 write 권한이 없기 때문에 error가 납니다. 그래서 각 user별로 cloning을 해서 사용하는 것입니다. 아래와 같이 하시면 됩니다.

u0017649@sys-89075:~$ conda create -n anaconda2 --clone="/opt/anaconda2"
Source: /opt/anaconda2
Destination: /home/u0017649/.conda/envs/anaconda2
The following packages cannot be cloned out of the root environment:
- conda-4.3.21-py27_0
- conda-env-2.6.0-0
Packages: 186
Files: 11
#
# To activate this environment, use:
# > source activate anaconda2
#
# To deactivate this environment, use:
# > source deactivate anaconda2
#

이제 cloning 환경으로 들어가서 위에서 실패했던 scikit-learn 패키지의 설치를 다시 해보겠습니다.

u0017649@sys-89075:~$ source activate anaconda2
(anaconda2) u0017649@sys-89075:~$ conda install scikit-learn

이건 잘 됩니다. 이것이 잘 되는 이유는 다음과 같이 각 user의 home directory 밑인 ~/.conda/envs에 cloning을 해놓고 쓰기 때문입니다. 물론 이는 disk 저장공간을 사용하는 작업이므로, 많은 user들이 각자 모두 cloning을 할 경우 disk 공간의 낭비가 꽤 있게 되는 문제가 있긴 합니다. 가령 Anaconda2를 1번 clone할 때마다 약 1.1 GB의 공간이 소모됩니다. 따라서 모든 user가 일괄적으로 이렇게 cloning을 할 필요는 없고, 정말 꼭 자신만의 변경된 package 환경을 가져야 할 경우에만 하실 것을 권고합니다.

(anaconda2) u0017649@sys-89075:~/.conda/envs/anaconda2$ du -sm .
1135 .

2017년 6월 13일 화요일

ppc64le 아키텍처용 anaconda package list에 없는 일부 package의 수동 설치 (xgboost의 사례)

이제 ppc64le 아키텍처에서도 Continuum사의 miniconda가 지원됩니다. 다만 모든 package들이 지금 다 available한 것은 아닙니다. 가령 아래 URL에 가서 확인해보면 몇몇 package들은 ppc64le에서는 지원되지 않는 것을 확인하실 수 있습니다. 대표적인 예가 xgboost 입니다.

https://repo.continuum.io/pkgs/free/linux-ppc64le/

그러나 여기에 포함되어 있지 않다고 해서 ppc64le에서는 정말 사용할 수 없느냐 하면 그건 아닙니다. 매우 간단히 설치가 가능합니다.

먼저, 그냥 그대로 pip로 xgboost를 설치할 때 어떤 error가 벌어지는지 보시지요. 먼저, pip가 anaconda에서 제공하는 pip인지 확인합니다.

u0017496@sys-87576:~$ which pip
/home/u0017496/miniconda3/bin/pip

그 다음에 이 pip를 이용하여 xgboost 설치를 시도해 봅니다.

u0017496@sys-87576:~$ pip install xgboost
...
Using cached xgboost-0.6a2.tar.gz
Complete output from command python setup.py egg_info:
rm -f -rf build build_plugin lib bin *~ */*~ */*/*~ */*/*/*~ */*.o */*/*.o */*/*/*.o xgboost
g++ -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude -Idmlc-core/include -Irabit/include -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
g++ -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude -Idmlc-core/include -Irabit/include -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
g++ -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude -Idmlc-core/include -Irabit/include -fPIC -fopenmp -MM -MT build/common/common.o src/common/common.cc >build/common/common.d
g++ -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude -Idmlc-core/include -Irabit/include -fPIC -fopenmp -MM -MT build/metric/metric.o src/metric/metric.cc >build/metric/metric.d
g++: error: unrecognized command line option ‘-msse2’

결국 intel x86 아키텍처에만 있는 SSE2 instruction 관련 option이 문제인 것을 보실 수 있습니다. 이는 source를 download 받은 뒤 직접 python setup.py를 수행함으로써 간단히 해결 가능합니다.

다음과 같이 pip download 명령으로 xgboost의 source를 download 받습니다.

u0017496@sys-87576:~$ pip download -d "./" xgboost

이것의 압축을 풀고, 관련 Makefile들을 수정합니다. 그냥 -msse2 부분만 빼줘도 되는데, 여기서는 하는 김에 CPU 아키텍처가 POWER8이라는 것을 지정하겠습니다.

u0017496@sys-87576:~$ tar -zxvf xgboost-0.6a2.tar.gz

u0017496@sys-87576:~$ cd xgboost-0.6a2

u0017496@sys-87576:~/xgboost-0.6a2$ vi ./xgboost/Makefile
export CFLAGS= -std=c++0x -Wall -O3 -mcpu=power8 -Wno-unknown-pragmas -funroll-loops -Iinclude $(ADD_CFLAGS) $(PLUGIN_CFLAGS)
#export CFLAGS= -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude $(ADD_CFLAGS) $(PLUGIN_CFLAGS)

u0017496@sys-87576:~/xgboost-0.6a2$ vi ./xgboost/dmlc-core/Makefile
export CFLAGS = -O3 -Wall -mcpu=power8 -Wno-unknown-pragmas -Iinclude -std=c++0x
#export CFLAGS = -O3 -Wall -msse2 -Wno-unknown-pragmas -Iinclude -std=c++0x

u0017496@sys-87576:~/xgboost-0.6a2$ vi ./xgboost/rabit/Makefile
export CFLAGS = -O3 -mcpu=power8 $(WARNFLAGS)
#export CFLAGS = -O3 -msse2 $(WARNFLAGS)

그 다음에 'python setup.py install' 명령을 직접 수행합니다.

u0017496@sys-87576:~/xgboost-0.6a2$ python setup.py install
...
Searching for scipy==0.19.0
Best match: scipy 0.19.0
Adding scipy 0.19.0 to easy-install.pth file

Using /home/u0017496/miniconda3/lib/python3.6/site-packages
Searching for numpy==1.13.0
Best match: numpy 1.13.0
Adding numpy 1.13.0 to easy-install.pth file

Using /home/u0017496/miniconda3/lib/python3.6/site-packages
Finished processing dependencies for xgboost==0.6a2

결과적으로 잘 설치되었습니다. PYTHONPATH로 되어 있는 /home/u0017496/miniconda3/lib/python3.6/site-packages 디렉토리를 확인하면 다음과 같이 해당 directory가 생성된 것을 보실 수 있습니다.

u0017496@sys-87576:~/xgboost-0.6a2$ ls /home/u0017496/miniconda3/lib/python3.6/site-packages | grep xgboost
xgboost-0.6a2-py3.6.egg

또한 conda list 명령으로 보면 pip 명령으로 해당 package가 설치된 것으로 display 되는 것을 확인하실 수 있습니다.

u0017496@sys-87576:~/xgboost-0.6a2$ conda list | grep xgboost
xgboost 0.6a2 <pip>

pip 명령으로 삭제도 정상적으로 됩니다.

u0017496@sys-87576:~/xgboost-0.6a2$ pip uninstall xgboost
Uninstalling xgboost-0.6a2:
/home/u0017496/miniconda3/lib/python3.6/site-packages/xgboost-0.6a2-py3.6.egg

Proceed (y/n)? y

Successfully uninstalled xgboost-0.6a2

2017년 5월 15일 월요일

Minsky 서버에 Continuum 아나콘다(Anaconda) 설치하기 + Tensorflow로 inception v3 training 해보기

Continuum에서 내놓은 아나콘다(Anaconda)는 여태까지는 x86용으로만 존재했으나, 최근 ARM processor와 POWER8 processor를 위한 min-conda를 내놓았습니다. 아래 site에 해당 package를 download 받기 위한 link와 설치 방법이 정리되어 있습니다.

https://www.continuum.io/content/conda-support-raspberry-pi-2-and-power8-le

불행히도 ppc64le를 위한 link는 잘못 지정되어 있어 '404 Not Found'가 나옵니다만, 실제로는 link만 잘못된 것이고 아래와 같이 file은 실제로 존재합니다. 위의 것은 python version2 용이고, 아래 것은 python 3용입니다.

https://repo.continuum.io/miniconda/Miniconda2-4.3.14-Linux-ppc64le.sh
https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh

여기서는 python 3용을 설치해보겠습니다.

u0017496@sys-87250:~$ wget https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
--2017-05-14 21:54:26-- https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
Resolving repo.continuum.io (repo.continuum.io)... 104.16.19.10, 104.16.18.10, 2400:cb00:2048:1::6810:120a, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.16.19.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34765794 (33M) [application/x-sh]
Saving to: ‘Miniconda3-4.3.14-Linux-ppc64le.sh’

Miniconda3-4.3.14-Linux-pp 100%[=====================================>] 33.15M 10.2MB/s in 3.3s

2017-05-17 22:05:18 (10.0 MB/s) - ‘Miniconda3-4.3.14-Linux-ppc64le.sh’ saved [34765794/34765794]

u0017496@sys-87250:~$ chmod a+x Miniconda3-4.3.14-Linux-ppc64le.sh

이제 이 shell script를 수행하면 license 동의 등에 답을 해야 하며, 그 외에도 여러가지 입력 값을 넣어야 합니다. 대부분 그냥 enter를 누르시면 됩니다.

u0017496@sys-87250:~$ ./Miniconda3-4.3.14-Linux-ppc64le.sh

Welcome to Miniconda3 4.3.14 (by Continuum Analytics, Inc.)

In order to continue the installation process, please review the license
agreement.
Please, press ENTER to continue
>>>

(중략)

[/home/u0017496/miniconda3] >>>
PREFIX=/home/u0017496/miniconda3
installing: python-3.6.0-0 ...
installing: cffi-1.9.1-py36_0 ...
installing: conda-env-2.6.0-0 ...
installing: cryptography-1.7.1-py36_0 ...
installing: idna-2.2-py36_0 ...
installing: libffi-3.2.1-1 ...
installing: openssl-1.0.2k-1 ...
installing: pyasn1-0.2.3-py36_0 ...
installing: pycosat-0.6.2-py36_0 ...
installing: pycparser-2.17-py36_0 ...
installing: pyopenssl-16.2.0-py36_0 ...
installing: requests-2.13.0-py36_0 ...
installing: ruamel_yaml-0.11.14-py36_1 ...
installing: setuptools-27.2.0-py36_0 ...
installing: six-1.10.0-py36_0 ...
installing: sqlite-3.13.0-0 ...
installing: xz-5.2.2-1 ...
installing: yaml-0.1.6-0 ...
installing: zlib-1.2.8-3 ...
installing: conda-4.3.14-py36_0 ...
installing: pip-9.0.1-py36_1 ...
installing: wheel-0.29.0-py36_0 ...
Python 3.6.0 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
Do you wish the installer to prepend the Miniconda3 install location
to PATH in your /home/u0017496/.bashrc ? [yes|no]

[no] >>> yes

Prepending PATH=/home/u0017496/miniconda3/bin to PATH in /home/u0017496/.bashrc
A backup will be made to: /home/u0017496/.bashrc-miniconda3.bak

For this change to become active, you have to open a new terminal.

Thank you for installing Miniconda2!

Share your notebooks and packages on Anaconda Cloud!
Sign up for free: https://anaconda.org

이제 mini-conda를 설치했으니 conda 명령을 쓸 수 있어야 합니다. 그러나 보시다시피 conda가 없습니다.

u0017496@sys-87250:~$ which conda

이는 mini-conda 설치시 ~/.bashrc에 conda의 PATH 정보가 자동으로 들어가긴 했지만 .bashrc가 수행되지 않았기 때문에 그런 것입니다. 수행하시면 conda의 PATH가 잡혀 있는 것을 보실 수 있습니다.

u0017496@sys-87250:~$ . ~/.bashrc
u0017496@sys-87250:~$ which conda
/home/u0017496/miniconda3/bin/conda

이제 다음과 같이 ananconda에 포함된 python library들을 보실 수 있습니다.

u0017496@sys-87250:~$ conda list
# packages in environment at /home/u0017496/miniconda3:
#
cffi 1.9.1 py36_0
conda 4.3.14 py36_0
conda-env 2.6.0 0
cryptography 1.7.1 py36_0
idna 2.2 py36_0
libffi 3.2.1 1
openssl 1.0.2k 1
pip 9.0.1 py36_1
pyasn1 0.2.3 py36_0
pycosat 0.6.2 py36_0
pycparser 2.17 py36_0
pyopenssl 16.2.0 py36_0
python 3.6.0 0
requests 2.13.0 py36_0
ruamel_yaml 0.11.14 py36_1
setuptools 27.2.0 py36_0
six 1.10.0 py36_0
sqlite 3.13.0 0
wheel 0.29.0 py36_0
xz 5.2.2 1
yaml 0.1.6 0
zlib 1.2.8 3

현재까지 Continuum에서 빌드해놓은 package들을 모조리 다 설치하는 명령을 다음에 정리했습니다.

u0017496@sys-87250:~$ for i in `conda list | awk '{print $1}' | grep -v \#`
> do
> conda install $i
> done

(중략)

이제 위에서 설치한 패키지 중 pip가 제대로 설치되었는지 conda search로 확인해보겠습니다. 아래와 같이 * 표시가 된 것이 설치된 것입니다.

u0017496@sys-87250:~$ conda search pip
Fetching package metadata .........
pip 7.1.0 py27_0 defaults
7.1.0 py34_0 defaults
7.1.0 py27_1 defaults
7.1.0 py34_1 defaults
7.1.2 py27_0 defaults
7.1.2 py34_0 defaults
8.1.0 py27_0 defaults
8.1.0 py34_0 defaults
8.1.0 py35_0 defaults
8.1.2 py27_0 defaults
8.1.2 py34_0 defaults
8.1.2 py35_0 defaults
9.0.0 py27_0 defaults
9.0.0 py34_0 defaults
9.0.0 py35_0 defaults
9.0.1 py27_1 defaults
9.0.1 py35_1 defaults
* 9.0.1 py36_1 defaults

u0017496@sys-87250:~$ which pip

/home/u0017496/miniconda3/bin/pip

u0017496@sys-87250:~$ pip --version

pip 9.0.1 from /home/u0017496/miniconda3/lib/python3.6/site-packages (python 3.6)

이렇게 conda에서 제공하는 pip로 keras 2.0.4를 설치해보겠습니다.

u0017496@sys-87250:~/miniconda3/lib$ pip install keras==2.0.4
Collecting keras==2.0.4
Downloading Keras-2.0.4.tar.gz (199kB)
100% |████████████████████████████████| 204kB 3.1MB/s
Collecting theano (from keras==2.0.4)
Downloading Theano-0.9.0.tar.gz (3.1MB)
100% |████████████████████████████████| 3.1MB 310kB/s
Collecting pyyaml (from keras==2.0.4)
Downloading PyYAML-3.12.tar.gz (253kB)
100% |████████████████████████████████| 256kB 3.6MB/s
Requirement already satisfied: six in ./python3.6/site-packages (from keras==2.0.4)
Requirement already satisfied: numpy>=1.9.1 in ./python3.6/site-packages (from theano->keras==2.0.4)
Requirement already satisfied: scipy>=0.14 in ./python3.6/site-packages (from theano->keras==2.0.4)
Building wheels for collected packages: keras, theano, pyyaml
Running setup.py bdist_wheel for keras ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/48/82/42/f06a8c03a8f95ada523a81ba723e89f059693e6ad868d09727
Running setup.py bdist_wheel for theano ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/d5/5b/93/433299b86e3e9b25f0f600e4e4ebf18e38eb7534ea518eba13
Running setup.py bdist_wheel for pyyaml ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
Successfully built keras theano pyyaml
Installing collected packages: theano, pyyaml, keras
Successfully installed keras-2.0.4 pyyaml-3.12 theano-0.9.0

또 gensim 2.0.0과 KoNLPy를 설치해보겠습니다.

u0017496@sys-87250:~$ pip install gensim==2.0.0
Collecting gensim==2.0.0
Downloading gensim-2.0.0.tar.gz (14.1MB)
100% |████████████████████████████████| 14.2MB 88kB/s
Requirement already satisfied: numpy>=1.3 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: scipy>=0.7.0 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: six>=1.5.0 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: smart_open>=1.2.1 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: boto>=2.32 in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Requirement already satisfied: bz2file in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Requirement already satisfied: requests in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Building wheels for collected packages: gensim
Running setup.py bdist_wheel for gensim ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/e9/5f/e7/4ff23a3fe4b181b44f37eed5602f179c1cc92a0a34f337e745
Successfully built gensim
Installing collected packages: gensim
Found existing installation: gensim 1.0.1
Uninstalling gensim-1.0.1:
Successfully uninstalled gensim-1.0.1
Successfully installed gensim-2.0.0

u0017496@sys-87250:~$ pip install konlpy

Collecting konlpy

Downloading konlpy-0.4.4-py2.py3-none-any.whl (22.5MB)

100% |████████████████████████████████| 22.5MB 57kB/s

Installing collected packages: konlpy

Successfully installed konlpy-0.4.4

이제 conda 명령을 통해 추가로 numpy와 matplotlib, scipy와 scikit-learn를 설치해보겠습니다. matplotlib의 prerequisite이 numpy이고, scikit-learn의 prerequisite이 scipy라서 그것들은 자동으로 설치되니까, 실제로는 conda 명령은 두번만 쓰면 됩니다.

u0017496@sys-87250:~$ conda install matplotlib
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /home/u0017496/miniconda3:

The following NEW packages will be INSTALLED:

cycler: 0.10.0-py36_0
freetype: 2.5.5-2
libpng: 1.6.27-0
matplotlib: 2.0.2-np112py36_0
numpy: 1.12.1-py36_0
openblas: 0.2.19-0
python-dateutil: 2.6.0-py36_0
pytz: 2017.2-py36_0

Proceed ([y]/n)? y

openblas-0.2.1 100% |###########################################################| Time: 0:00:00 10.21 MB/s
libpng-1.6.27- 100% |###########################################################| Time: 0:00:00 12.75 MB/s
freetype-2.5.5 100% |###########################################################| Time: 0:00:00 10.53 MB/s
numpy-1.12.1-p 100% |###########################################################| Time: 0:00:00 15.12 MB/s
pytz-2017.2-py 100% |###########################################################| Time: 0:00:00 13.25 MB/s
cycler-0.10.0- 100% |###########################################################| Time: 0:00:00 15.61 MB/s
python-dateuti 100% |###########################################################| Time: 0:00:00 6.43 MB/s
matplotlib-2.0 100% |###########################################################| Time: 0:00:00 14.62 MB/s

u0017496@sys-87250:~$ conda install scikit-learn
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /home/u0017496/miniconda3:

The following NEW packages will be INSTALLED:

scikit-learn: 0.18.1-np112py36_1
scipy: 0.19.0-np112py36_0

Proceed ([y]/n)? y

scipy-0.19.0-n 100% |###########################################################| Time: 0:00:02 14.75 MB/s
scikit-learn-0 100% |###########################################################| Time: 0:00:00 15.56 MB/s

이렇게 설치된 것들은 아래와 같이 /home/u0017496/miniconda3/lib/python3.6/site-packages 에 들어갑니다.

u0017496@sys-87250:~$ ls /home/u0017496/miniconda3/lib/python3.6/site-packages/
asn1crypto mpl_toolkits python_dateutil-2.6.0-py3.6.egg-info
asn1crypto-0.22.0-py3.6.egg-info numpy-1.12.1.dist-info pytz
cffi OpenSSL pytz-2017.2-py3.6.egg-info
cffi-1.10.0-py3.6.egg-info packaging README.txt
_cffi_backend.so packaging-16.8-py3.6.egg-info requests
conda pip requests-2.14.2-py3.6.egg-info
conda-4.3.18-py3.6.egg-info pip-9.0.1-py3.6.egg-info ruamel_yaml
conda_env pyasn1 scikit_learn-0.18.1-py3.6.egg-info
cryptography pyasn1-0.2.3-py3.6.egg-info scipy
cryptography-1.8.1-py3.6.egg-info __pycache__ scipy-0.19.0-py3.6.egg-info
cycler-0.10.0-py3.6.egg-info pycosat-0.6.2-py3.6.egg-info setuptools-27.2.0-py3.6.egg
cycler.py pycosat.cpython-36m-powerpc64le-linux-gnu.so setuptools.pth
dateutil pycparser six-1.10.0-py3.6.egg-info
easy-install.pth pycparser-2.17-py3.6.egg-info six.py
idna pylab.py sklearn
idna-2.5-py3.6.egg-info pyOpenSSL-17.0.0-py3.6.egg-info test_pycosat.py
matplotlib pyparsing-2.1.4-py3.6.egg-info wheel
matplotlib-2.0.2-py3.6.egg-info pyparsing.py wheel-0.29.0-py3.6.egg-info

따라서 이것들을 사용하기 위해서는 PYTHONPATH는 다음과 같이 설정하시면 됩니다.

u0017496@sys-87250:~$ export PYTHONPATH=/home/u0017496/miniconda3/lib/python3.6/site-packages:$PYTHONPATH

이제 여기에 (PowerAI에 포함된 tensorflow 말고) conda로 bazel, tensorflow 및 tensorflow-gpu도 설치해보겠습니다.

u0017496@sys-87250:~$ conda install bazel
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /home/u0017496/miniconda3:

The following NEW packages will be INSTALLED:

bazel: 0.4.5-0

Proceed ([y]/n)? y

bazel-0.4.5-0. 100% |#############################################| Time: 0:00:09 13.37 MB/s

u0017496@sys-87250:~$ conda install tensorflow
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /home/u0017496/miniconda3:

The following NEW packages will be INSTALLED:

libprotobuf: 3.2.0-0
protobuf: 3.2.0-py36_0
tensorflow: 1.1.0-np112py36_0
werkzeug: 0.12.2-py36_0

Proceed ([y]/n)? y

libprotobuf-3. 100% |#############################################| Time: 0:00:00 13.84 MB/s
werkzeug-0.12. 100% |#############################################| Time: 0:00:00 18.67 MB/s
protobuf-3.2.0 100% |#############################################| Time: 0:00:00 10.39 MB/s
tensorflow-1.1 100% |#############################################| Time: 0:00:01 15.16 MB/s

u0017496@sys-87250:~$ conda install tensorflow-gpu
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /home/u0017496/miniconda3:

The following NEW packages will be INSTALLED:

cudatoolkit: 8.0-0
cudnn: 6.0.21-0
tensorflow-gpu: 1.1.0-np112py36_0

Proceed ([y]/n)? y

cudatoolkit-8. 100% |#############################################| Time: 0:00:29 11.24 MB/s
cudnn-6.0.21-0 100% |#############################################| Time: 0:00:11 15.97 MB/s
tensorflow-gpu 100% |#############################################| Time: 0:00:06 14.27 MB/s

conda list 명령으로 보면 다음과 같은 것들이 설치된 것을 보실 수 있습니다.

u0017496@sys-87250:~$ conda list
# packages in environment at /home/u0017496/miniconda3:
#
asn1crypto 0.22.0 py36_0
bazel 0.4.5 0
boto 2.46.1 py36_0
bz2file 0.98 py36_0
cffi 1.10.0 py36_0
conda 4.3.18 py36_0
conda-env 2.6.0 0
cryptography 1.8.1 py36_0
cudatoolkit 8.0 0
cudnn 6.0.21 0
cycler 0.10.0 py36_0
freetype 2.5.5 2
gensim 1.0.1 np112py36_0
gensim 2.0.0 <pip>
idna 2.5 py36_0
Keras 2.0.4 <pip>
konlpy 0.4.4 <pip>
libffi 3.2.1 1
libpng 1.6.27 0
libprotobuf 3.2.0 0
matplotlib 2.0.2 np112py36_0
numpy 1.12.1 <pip>
numpy 1.12.1 py36_0
openblas 0.2.19 0
openssl 1.0.2k 2
packaging 16.8 py36_0
pip 9.0.1 py36_1
protobuf 3.2.0 py36_0
pyasn1 0.2.3 py36_0
pycosat 0.6.2 py36_0
pycparser 2.17 py36_0
pyopenssl 17.0.0 py36_0
pyparsing 2.1.4 py36_0
python 3.6.1 2
python-dateutil 2.6.0 py36_0
pytz 2017.2 py36_0
PyYAML 3.12 <pip>
requests 2.14.2 py36_0
ruamel_yaml 0.11.14 py36_1
scikit-learn 0.18.1 np112py36_1
scipy 0.19.0 np112py36_0
setuptools 27.2.0 py36_0
six 1.10.0 py36_0
smart_open 1.5.2 py36_0
sqlite 3.13.0 0
tensorflow 1.1.0 np112py36_0
tensorflow-gpu 1.1.0 np112py36_0
Theano 0.9.0 <pip>
werkzeug 0.12.2 py36_0
wheel 0.29.0 py36_0
xz 5.2.2 1
yaml 0.1.6 0
zlib 1.2.8 3

설치하는 김에, 이렇게 conda로 설치한 tensorflow를 이용하여 inception v3 model을 training 해보겠습니다. 다음 순서대로 따라 하시면 됩니다.

u0017496@sys-87250:~/inception$ pwd
/home/u0017496/inception

u0017496@sys-87250:~/inception$ export INCEPTION_DIR=/home/u0017496/inception

u0017496@sys-87250:~/inception$ curl -O http://download.tensorflow.org/models/image/imagenet/inception-v3-2016-03-01.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 380M 100 380M 0 0 5918k 0 0:01:05 0:01:05 --:--:-- 4233k

u0017496@sys-87250:~/inception$ tar -xvf inception-v3-2016-03-01.tar.gz
inception-v3/
inception-v3/checkpoint
inception-v3/README.txt
inception-v3/model.ckpt-157585

u0017496@sys-87250:~/inception$ git clone https://github.com/tensorflow/models.git
Cloning into 'models'...
remote: Counting objects: 4703, done.
remote: Compressing objects: 100% (43/43), done.
remote: Total 4703 (delta 17), reused 31 (delta 11), pack-reused 4649
Receiving objects: 100% (4703/4703), 153.34 MiB | 5.62 MiB/s, done.
Resolving deltas: 100% (2374/2374), done.
Checking connectivity... done.

u0017496@sys-87250:~/inception/models/inception$ export FLOWERS_DIR=/home/u0017496/inception/models/inception

u0017496@sys-87250:~/inception/models/inception$ mkdir -p $FLOWERS_DIR/data

u0017496@sys-87250:~/inception/models/inception$ which bazel
/home/u0017496/miniconda3/bin/bazel

u0017496@sys-87250:~/inception/models/inception$ bazel build inception/download_and_preprocess_flowers
Extracting Bazel installation...
....................
INFO: Found 1 target...
Target //inception:download_and_preprocess_flowers up-to-date:
bazel-bin/inception/download_and_preprocess_flowers
INFO: Elapsed time: 6.943s, Critical Path: 0.05s

u0017496@sys-87250:~/inception/models/inception$ export TEST_TMPDIR=/home/u0017496/.cache

u0017496@sys-87250:~/inception/models/inception$ bazel build inception/download_and_preprocess_flowers
INFO: $TEST_TMPDIR defined: output root default is '/home/u0017496/.cache'.
Extracting Bazel installation...
.............
INFO: Found 1 target...
Target //inception:download_and_preprocess_flowers up-to-date:
bazel-bin/inception/download_and_preprocess_flowers
INFO: Elapsed time: 4.867s, Critical Path: 0.03s

u0017496@sys-87250:~/inception/models/inception$ bazel-bin/inception/download_and_preprocess_flowers $FLOWERS_DIR/data
Downloading flower data set.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 218M 100 218M 0 0 9372k 0 0:00:23 0:00:23 --:--:-- 10.1M
(중략)
Found 3170 JPEG files across 5 labels inside /home/u0017496/inception/models/inception/data/raw-data/train.
Launching 2 threads for spacings: [[0, 1585], [1585, 3170]]
2017-05-19 05:33:44.191446 [thread 0]: Processed 1000 of 1585 images in thread batch.
2017-05-19 05:33:44.213856 [thread 1]: Processed 1000 of 1585 images in thread batch.
2017-05-19 05:33:54.902070 [thread 1]: Wrote 1585 images to /home/u0017496/inception/models/inception/data/train-00001-of-00002
2017-05-19 05:33:54.902172 [thread 1]: Wrote 1585 images to 1585 shards.
2017-05-19 05:33:54.911283 [thread 0]: Wrote 1585 images to /home/u0017496/inception/models/inception/data/train-00000-of-00002
2017-05-19 05:33:54.911360 [thread 0]: Wrote 1585 images to 1585 shards.
2017-05-19 05:33:55.171141: Finished writing all 3170 images in data set.

아래에서 보시다시피 이 inception v3는 꽃 사진을 분류하는 neural network입니다.

u0017496@sys-87250:~/inception/models/inception$ du -sm data/raw-data/train/*
29 data/raw-data/train/daisy
44 data/raw-data/train/dandelion
1 data/raw-data/train/LICENSE.txt
33 data/raw-data/train/roses
47 data/raw-data/train/sunflowers
48 data/raw-data/train/tulips

u0017496@sys-87250:~/inception/models/inception$ bazel build inception/flowers_train
INFO: $TEST_TMPDIR defined: output root default is '/home/u0017496/.cache'.
............................
INFO: Found 1 target...
Target //inception:flowers_train up-to-date:
bazel-bin/inception/flowers_train
INFO: Elapsed time: 6.502s, Critical Path: 0.03s

이제 비로소 inception v3의 training 준비가 끝났습니다. 이제 다음 명령으로 training을 시작합니다.

u0017496@sys-87250:~/inception/models/inception$ time bazel-bin/inception/flowers_train --train_dir=$FLOWERS_DIR/train --data_dir=$FLOWERS_DIR/data --pretrained_model_checkpoint_path=$INCEPTION_DIR/inception-v3/model.ckpt-157585 --fine_tune=True --initial_learning_rate=0.001 -input_queue_memory_factor=1 --max_steps=50 --num_gpus 1 --batch_size=32

NVIDIA: no NVIDIA devices found
2017-05-19 05:41:03.740213: E tensorflow/stream_executor/cuda/cuda_driver.cc:405] failed call to cuInit: CUDA_ERROR_UNKNOWN
2017-05-19 05:41:03.740670: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:145] kernel driver does not appear to be running on this host (sys-87250): /proc/driver/nvidia/version does not exist
2017-05-19 05:41:51.947244: Pre-trained model restored from /home/u0017496/inception/inception-v3/model.ckpt-157585
2017-05-19 05:47:22.023602: step 0, loss = 2.79 (0.2 examples/sec; 182.713 sec/batch)
2017-05-19 06:05:58.942671: step 10, loss = 2.53 (0.4 examples/sec; 78.882 sec/batch)
2017-05-19 06:19:26.875533: step 20, loss = 2.40 (0.4 examples/sec; 82.410 sec/batch)
2017-05-19 06:33:10.333275: step 30, loss = 2.20 (0.4 examples/sec; 77.844 sec/batch)
2017-05-19 06:48:27.688993: step 40, loss = 2.24 (0.3 examples/sec; 96.148 sec/batch)

real 84m30.882s
user 135m20.864s
sys 2m30.832s

이제 와서 고백하지만 제가 설치 demo를 보여드린 이 서버는 사실 GPU가 달려 있지 않은 POWER8 서버입니다. GPU가 없는 경우 CPU를 이용하게 되는데, 그런 경우 이 training의 완료는 보시다시피 매우, 매우 오래 걸립니다. 저 output을 보면 초당 example 0.4개 처리로 나옵니다만, P100을 이용하는 경우 (GPU 개수 및 batch size에 따라) 초당 50개~200개 단위로 처리가 됩니다.

아래는 전에 PowerAI를 설치한 Minsky 서버에서 수행했던 inception v3의 결과 log 일부입니다.

2017-05-16 03:48:46.352210: Pre-trained model restored from /gpfs/gpfs_gl4_16mb/b7p088za/inception-v3/model.ckpt-157585
2017-05-16 03:52:44.322381: step 0, loss = 2.72 (17.6 examples/sec; 21.830 sec/batch)
2017-05-16 03:55:29.550791: step 10, loss = 2.57 (213.6 examples/sec; 1.797 sec/batch)
2017-05-16 03:55:47.619990: step 20, loss = 2.35 (212.1 examples/sec; 1.810 sec/batch)
2017-05-16 03:56:05.953991: step 30, loss = 2.17 (206.6 examples/sec; 1.859 sec/batch)
2017-05-16 03:56:24.306742: step 40, loss = 1.98 (209.4 examples/sec; 1.834 sec/batch)
2017-05-16 03:56:42.490063: step 50, loss = 1.92 (217.8 examples/sec; 1.763 sec/batch)
2017-05-16 03:57:00.444537: step 60, loss = 1.67 (216.6 examples/sec; 1.773 sec/batch)
2017-05-16 03:57:18.366941: step 70, loss = 1.58 (212.7 examples/sec; 1.806 sec/batch)
2017-05-16 03:57:36.467837: step 80, loss = 1.55 (213.6 examples/sec; 1.798 sec/batch)