IBM Minsky에서의 H2O Deep Learning 설치

H2O는 확장성이 좋고 Spark, Hadoop, R 등 big data platform과 연동하여 사용하기 좋은 machine learning platform입니다.  물론 open source입니다.  H2O도 ppc64le 아키텍처, 즉 IBM Minsky (S822LC for HPC) 서버에서 사용 가능합니다.  여기서는 H2O를 R과 함께 Minsky 서버에 설치하는 방법을 정리했으며, 원본은 아래 link에 있는 내용을 참조했습니다.

http://h2o-release.s3.amazonaws.com/h2o/rel-tibshirani/8/index.html#R


먼저, 아래 주소에서 H2O를 Minsky 서버로 download 합니다.

test@minsky:~/R$ wget http://h2o-release.s3.amazonaws.com/h2o/rel-tibshirani/8/h2o-3.6.0.8.zip

test@minsky:~/R$ unzip h2o-3.10.4.2.zip

test@minsky:~/R$ cd h2o-3.10.4.2

test@minsky:~/R/h2o-3.10.4.2$ java -jar h2o.jar
03-27 17:04:32.866 172.18.229.117:54321  49868  main      INFO: ----- H2O started  -----
03-27 17:04:32.890 172.18.229.117:54321  49868  main      INFO: Build git branch: rel-ueno
...
03-27 17:04:34.787 172.18.229.117:54321  49868  main      INFO: Open H2O Flow in your web browser: http://172.18.229.117:54321
03-27 17:04:34.787 172.18.229.117:54321  49868  main      INFO:

이렇게 H2O 서버를 구동한 뒤에, 아래와 같이 web browser에서 54321 port로 접속이 가능합니다.

R은 Ubuntu 16.04에 포함된 r-base-core package에서 나온 3.2.3을 써도 되고, 앞선 포스팅에서 설명한 대로 MS open R server 3.3.2를 직접 build해서 써도 됩니다.  

test@minsky:~/R/h2o-3.10.4.2$ which R
/usr/local/lib/R/bin/R

먼저, R을 구동하여 R에 h2o 패키지가 설치되어 있는지 확인하여 만약 기존 것이 있으면 제거합니다.

test@minsky:~/R/h2o-3.10.4.2$ sudo R

> if ("package:h2o" %in% search()) { detach("package:h2o", unload=TRUE) }
> if ("h2o" %in% rownames(installed.packages())) { remove.packages("h2o") }

이어서 methods, statmod, stats, RCurl 등의 R 패키지를 설치합니다.

> if (! ("methods" %in% rownames(installed.packages()))) { install.packages("methods") }
> if (! ("statmod" %in% rownames(installed.packages()))) { install.packages("statmod") }
> if (! ("stats" %in% rownames(installed.packages()))) { install.packages("stats") }
> if (! ("graphics" %in% rownames(installed.packages()))) { install.packages("graphics") }
> if (! ("RCurl" %in% rownames(installed.packages()))) { install.packages("RCurl") }
> if (! ("jsonlite" %in% rownames(installed.packages()))) { install.packages("jsonlite") }
> if (! ("tools" %in% rownames(installed.packages()))) { install.packages("tools") }
> if (! ("utils" %in% rownames(installed.packages()))) { install.packages("utils") }

이 과정에서 필요시 다음과 같이 internet에서 이 package들을 download 받아옵니다.

Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
--- Please select a CRAN mirror for use in this session ---
HTTPS CRAN mirror

...
49: USA (CA 1) [https]             50: USA (IA) [https]
51: USA (IN) [https]               52: USA (KS) [https]
53: USA (MI 1) [https]             54: USA (OR) [https]
55: USA (TN) [https]               56: USA (TX 1) [https]
57: USA (TX 2) [https]             58: (HTTP mirrors)

Selection: 49
trying URL 'https://cran.cnr.berkeley.edu/src/contrib/statmod_1.4.29.tar.gz'
Content type 'application/x-gzip' length 56932 bytes (55 KB)
==================================================
downloaded 55 KB


이 과정이 성공적으로 완료되면 이제 H2O 패키지를 설치합니다.  다만, 저 원본 URL에 나온 것과 같은 아래 tibshirani 버전은 H2O 버전과 맞지 않아 아래와 같은 error를 냅니다. 

> install.packages("h2o", type="source", repos=(c("http://h2o-release.s3.amazonaws.com/h2o/rel-tibshirani/8/R")))
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'http://h2o-release.s3.amazonaws.com/h2o/rel-tibshirani/8/R/src/contrib/h2o_3.6.0.8.tar.gz'
Content type 'application/x-tar; charset=binary' length 46482663 bytes (44.3 MB)
==================================================
downloaded 44.3 MB
...

> localH2O = h2o.init(nthreads=-1)
...
Error in h2o.init(nthreads = -1) :
  Version mismatch! H2O is running version 3.10.4.2 but R package is version 3.6.0.8


따라서 아래와 같이 tverberg 버전을 설치해야 합니다.

> install.packages("h2o", type="source", repos=(c("http://h2o-release.s3.amazonaws.com/h2o/rel-tverberg/6/R")))
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'http://h2o-release.s3.amazonaws.com/h2o/rel-tverberg/6/R/src/contrib/h2o_3.10.3.6.tar.gz'
Content type 'application/x-tar' length 59595766 bytes (56.8 MB)
==================================================
downloaded 56.8 MB
...

설치가 완료되면 h2o를 load하고 h2o.init을 수행합니다.

> library(h2o)
----------------------------------------------------------------------
Your next step is to start H2O:
    > h2o.init()
For H2O package documentation, ask for help:
    > ??h2o
After starting H2O, you can use the Web UI at http://localhost:54321
For more information visit http://docs.h2o.ai
----------------------------------------------------------------------
Attaching package: ‘h2o’
....

> localH2O = h2o.init(nthreads=-1)
...
Starting H2O JVM and connecting: ... Connection successful!

R is connected to the H2O cluster:
    H2O cluster uptime:         3 seconds 392 milliseconds
    H2O cluster version:        3.10.3.6
    H2O cluster version age:    1 month and 6 days
    H2O cluster name:           H2O_started_from_R_root_uwo022
    H2O cluster total nodes:    1
    H2O cluster total memory:   0.89 GB
    H2O cluster total cores:    128
    H2O cluster allowed cores:  128
    H2O cluster healthy:        TRUE
    H2O Connection ip:          localhost
    H2O Connection port:        54321
    H2O Connection proxy:       NA
    R Version:                  R version 3.2.3 (2015-12-10)


이어서 demo로 kmeans를 수행해봅니다.  사전에 tightvncserver 등을 구동시켜 plot이 되도록 해놓습니다.

> demo(h2o.kmeans)

        demo(h2o.kmeans)
        ---- ~~~~~~~~~~

Type  <Return>   to start :
...
> prostate.hex = h2o.uploadFile(path = system.file("extdata", "prostate.csv", package="h2o"), destination_frame = "prostate")
  |======================================================================| 100%

> summary(prostate.hex)
 ID               CAPSULE          AGE             RACE
 Min.   :  1.00   Min.   :0.0000   Min.   :43.00   Min.   :0.000
 1st Qu.: 95.75   1st Qu.:0.0000   1st Qu.:62.00   1st Qu.:1.000
 Median :190.50   Median :0.0000   Median :67.00   Median :1.000
 Mean   :190.50   Mean   :0.4026   Mean   :66.04   Mean   :1.087
 3rd Qu.:285.25   3rd Qu.:1.0000   3rd Qu.:71.00   3rd Qu.:1.000
 Max.   :380.00   Max.   :1.0000   Max.   :79.00   Max.   :2.000
 DPROS           DCAPS           PSA               VOL
 Min.   :1.000   Min.   :1.000   Min.   :  0.300   Min.   : 0.00
 1st Qu.:1.000   1st Qu.:1.000   1st Qu.:  4.900   1st Qu.: 0.00
 Median :2.000   Median :1.000   Median :  8.664   Median :14.20
 Mean   :2.271   Mean   :1.108   Mean   : 15.409   Mean   :15.81
 3rd Qu.:3.000   3rd Qu.:1.000   3rd Qu.: 17.063   3rd Qu.:26.40
 Max.   :4.000   Max.   :2.000   Max.   :139.700   Max.   :97.60
 GLEASON
 Min.   :0.000
 1st Qu.:6.000
 Median :6.000
 Mean   :6.384
 3rd Qu.:7.000
 Max.   :9.000

....

Total Within SS:  537.6507
Between SS:  1357.349
Total SS:  1895
Centroid Statistics:
   centroid      size within_cluster_sum_of_squares
1         1 156.00000                     165.18206
2         2   6.00000                      10.15081
3         3   2.00000                       5.89369
4         4   1.00000                       0.00000
5         5  60.00000                     114.01143
6         6   2.00000                       0.04695
7         7  11.00000                      18.54178
8         8  30.00000                      82.95187
9         9  23.00000                      35.24021
10       10  89.00000                     105.63191

> plot(prostate.ctrs[,1:2])
Hit <Return> to see next plot:

> plot(prostate.ctrs[,3:4])

> title("K-Means Centers for k = 10", outer = TRUE, line = -2.0)
Warning message:
In summary.H2OFrame(prostate.hex) :
  Approximated quantiles computed! If you are interested in exact quantiles, please pass the `exact_quantiles=TRUE` parameter.


결과로서 아래와 같은 그래프가 그려지는 것을 보실 수 있습니다.

Ubuntu 16.04 ppc64le에서, source로부터 Caffe를 build하는 방법

지난 편의 tensorflow 경우처럼, PowerAI 속에 caffe도 들어 있긴 합니다만, 간혹 customizing 때문에 source에서 build할 필요도 있을 수 있습니다.   여기서는 CUDA 8.0과 PowerAI가 이미 다 설치된 상태에서, caffe build를 시작하는 것으로 하겠습니다.  먼저, 기반 OS package를 설치합니다.

test@k8002:~$ sudo apt-get install libopencv-core-dev libopencv-dev libgflags-dev libgoogle-glog-dev libhdf5-dev libleveldb-dev liblmdb-dev libsnappy-dev libboost-python1.58-dev

이어서 github에서 caffe source를 복제합니다.

test@k8002:~$ git clone https://github.com/BVLC/caffe.git
Cloning into 'caffe'...
remote: Counting objects: 37621, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 37621 (delta 11), reused 6 (delta 6), pack-reused 37601
Receiving objects: 100% (37621/37621), 47.93 MiB | 5.27 MiB/s, done.
Resolving deltas: 100% (25123/25123), done.
Checking connectivity... done.

test@k8002:~$ cd caffe

test@k8002:~/caffe$ git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/device-abstraction
  remotes/origin/gh-pages
  remotes/origin/master
  remotes/origin/opencl
  remotes/origin/parallel
  remotes/origin/tutorial
  remotes/origin/williford-patch-1
  remotes/origin/windows

Caffe는 tensorflow와는 달리 1.0.0-rc5 버전을 최근 계속 유지하고 있습니다.  따라서 git checkout은 하지 않고 그대로 master 버전을 사용합니다.  make를 수행하기 전에, 먼저 Makefile.config.example을 편집하여 Makefile.config를 다음과 같이 일부 수정하여 만듭니다.  openblas는 PowerAI에 포함된 것을 사용하고, 그 이외의 것은 Ubuntu OS에 포함된 package를 사용합니다.

test@k8002:~/caffe$ cp Makefile.config.example Makefile.config

test@k8002:~/caffe$ vi Makefile.config
...
USE_CUDNN := 1   (원래 #으로 comment-out 되어 있는 것을 풀어줍니다.)
...
BLAS := open
#BLAS := atlas
...
BLAS_INCLUDE := /opt/DL/openblas/include
# BLAS_INCLUDE := /path/to/your/blas
BLAS_LIB := /opt/DL/openblas/lib
# BLAS_LIB := /path/to/your/blas
...
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/powerpc64le-linux-gnu/hdf5/serial


다음과 같이 make all을 수행하면 .build_release/tools directory 밑에 caffe가 빌드됩니다.

test@k8002:~/caffe$ make all

이와는 별도로, 다른 POWER8 서버로 배포 가능한 binary를 만드려면 다음과 같이 하면 됩니다.   빌드가 끝나면 distribute 밑에 bin과 include, lib 등의 필요 binary 파일들이 생성되므로, 이것을 tar 등으로 말아서 다른 서버로 가져가면 됩니다.

test@k8002:~/caffe$ make distribute

test@k8002:~/caffe$ ls distribute/
bin  include  lib  proto  python

test@k8002:~/caffe$ ls distribute/bin
caffe.bin               convert_mnist_siamese_data.bin  train_net.bin
classification.bin      device_query.bin                upgrade_net_proto_binary.bin
compute_image_mean.bin  extract_features.bin            upgrade_net_proto_text.bin
convert_cifar_data.bin  finetune_net.bin                upgrade_solver_proto_text.bin
convert_imageset.bin    net_speed_benchmark.bin
convert_mnist_data.bin  test_net.bin

test@k8002:~/caffe$ ls distribute/python
caffe  classify.py  CMakeLists.txt  detect.py  draw_net.py  requirements.txt  train.py

이렇게 생성된 caffe binary를 수행하려면 다음과 같이 openblas의 library path를 지정해주시고 사용하시면 됩니다.


test@k8002:~/caffe$ export LD_LIBRARY_PATH=/opt/DL/openblas/lib:$LD_LIBRARY_PATH

test@k8002:~/caffe$ .build_release/tools/caffe --version
caffe version 1.0.0-rc5

Ubuntu 16.04 ppc64le에서, source로부터 tensorflow를 build하는 방법

PowerAI 속에 tensorflow가 들어 있긴 합니다만, 간혹 customizing 때문에 source에서 build할 필요도 있을 수 있습니다.   여기서는 CUDA 8.0과 PowerAI가 이미 다 설치된 상태에서, build를 시작하는 것으로 하겠습니다.  먼저, 기반 OS package를 설치합니다.

test@k8002:~$ sudo apt-get install make automake autoconf gfortran-4.8-multilib-powerpc-linux-gnu libgfortran-4.8-dev-powerpc-cross gfortran libgfortran-4.8-dev gcc-powerpc-linux-gnu g++ pkg-config cmake-curses-gui zlib1g-dev zip unzip libtool openjdk-8-jdk python python-all-dev swig python-numpy g++-4.8 g++-4.8-multilib-powerpc-linux-gnu zlibc re2c libre2-dev libcupti-dev

Tensorflow를 github에서 download 받습니다.

test@k8002:~$ git clone --recurse-submodules https://github.com/tensorflow/tensorflow

이제 tensorflow에 어떤 버전의 branch들이 있는지 확인합니다.

test@k8002:~$ cd tensorflow
test@k8002:~/tensorflow$ git branch -a
* master
  remotes/origin/0.6.0
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/origin/r0.10
  remotes/origin/r0.11
  remotes/origin/r0.12
  remotes/origin/r0.7
  remotes/origin/r0.8
  remotes/origin/r0.9
  remotes/origin/r1.0

이중에서 일단은 master 버전을 build해보겠습니다.

test@k8002:~/tensorflow$ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.

여기서 일단 환경 변수 확인을 해보겠습니다.

test@k8002:~/tensorflow$ env | grep PATH
LD_LIBRARY_PATH=/usr/local/cuDNN:/usr/local/cuda-8.0/lib64
PATH=/home/test/bin:/home/test/.local/bin:/usr/local/cuda-8.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

Github에서 download 받은 tensorflow를 build할 때도, PowerAI에 포함된 bazel을 사용하면 편리합니다.  그를 위해, 현재의 PATH 환경변수들을 PowerAI 기준으로 고쳐줘야 하는데, 그를 위해서는 아래와 같이 PowerAI에 포함된 script 하나만 수행해주면 됩니다.

test@k8002:~/tensorflow$ source /opt/DL/tensorflow/bin/tensorflow-activate

이제 아래와 같이 PATH에 /opt/DL/tensorflow/bin가 추가되고, PYTHONPATH가 추가된 것을 보실 수 있습니다.

test@k8002:~/tensorflow$ env | grep PATH
LD_LIBRARY_PATH=/usr/local/cuDNN:/usr/local/cuda-8.0/lib64
PATH=/home/test/bin:/home/test/.local/bin:/usr/local/cuda-8.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/DL/tensorflow/bin
PYTHONPATH=/opt/DL/tensorflow/lib/python2.7/site-packages

여기에 추가로 bazel의 PATH도 잡아줘야 합니다.  Tensorflow의 build를 위해서는 먼저 bazel을 build해야 하는데, 사실 그것이 tensorflow build보다 더 힘듭니다.  여기서는 PowerAI에 포함된 bazel을 간편하게 사용하겠습니다.

test@k8002:~/tensorflow$ export PATH=/opt/DL/bazel/bin:$PATH

이제 configure부터 시작하여 build에 들어갑니다.

test@k8002:~/tensorflow$ ./configure
Please specify the location of python. [Default is /usr/bin/python]:
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:
Do you wish to use jemalloc as the malloc implementation? [Y/n]
jemalloc enabled
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N]
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N]
No Hadoop File System support will be enabled for TensorFlow
Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N]
No XLA support will be enabled for TensorFlow
Found possible Python library paths:
  /opt/DL/tensorflow/lib/python2.7/site-packages
  /usr/local/lib/python2.7/dist-packages
  /usr/lib/python2.7/dist-packages
Please input the desired Python library path to use.  Default is [/opt/DL/tensorflow/lib/python2.7/site-packages]
/usr/lib/python2.7/dist-packages

Do you wish to build TensorFlow with OpenCL support? [y/N]
No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with CUDA support? [y/N] y
CUDA support will be enabled for TensorFlow
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Please specify the CUDA SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 8.0
Please specify the location where CUDA 8.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /usr/local/cuda-8.0
Please specify the Cudnn version you want to use. [Leave empty to use system default]: 5
Please specify the location where cuDNN 5 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda-8.0]: /usr/lib/powerpc64le-linux-gnu  
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]: 6.0    (--> P100은 6.0, K80의 경우 3.7)
Extracting Bazel installation...
.............
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
............
WARNING: /home/test/.cache/bazel/_bazel_test/3a51eda24c560ebddf29b66b8fa0460e/external/protobuf/protobuf.bzl:90:19: Variables HOST_CFG and DATA_CFG are deprecated in favor of strings "host" and "data" correspondingly.
WARNING: /home/test/.cache/bazel/_bazel_test/3a51eda24c560ebddf29b66b8fa0460e/external/protobuf/protobuf.bzl:96:28: Variables HOST_CFG and DATA_CFG are deprecated in favor of strings "host" and "data" correspondingly.
INFO: Downloading http://pilotfiber.dl.sourceforge.net/project/boost/boost/1.6\
1.0/boost_1_61_0.tar.gz: 104,733,800 bytes
...
INFO: All external dependencies fetched successfully.
Configuration finished

configure가 끝나면 이제 bazel build를 합니다.

test@k8002:~/tensorflow$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
...
Target //tensorflow/tools/pip_package:build_pip_package up-to-date:
  bazel-bin/tensorflow/tools/pip_package/build_pip_package
INFO: Elapsed time: 256.806s, Critical Path: 203.45s

윗 단계가 성공적으로 끝나면 다음과 같이 /tmp/tensorflow_pkg directory에 pip package를 build합니다.

test@k8002:~/tensorflow$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
Thu Mar 16 15:03:32 KST 2017 : === Using tmpdir: /tmp/tmp.LCBvnwu8ZZ
~/tensorflow/bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles ~/tensorflow
~/tensorflow
/tmp/tmp.LCBvnwu8ZZ ~/tensorflow
Thu Mar 16 15:03:33 KST 2017 : === Building wheel
warning: no files found matching '*.dll' under directory '*'
warning: no files found matching '*.lib' under directory '*'
~/tensorflow
Thu Mar 16 15:03:44 KST 2017 : === Output wheel file is in: /tmp/tensorflow_pkg

이제 /tmp/tensorflow_pkg directory에 생성된 wheel file을 이용하여 pip install을 하면 tensorflow 설치가 완료됩니다.

test@k8002:~/tensorflow$ sudo pip install /tmp/tensorflow_pkg/tensorflow-1.0.1-cp27-cp27mu-linux_ppc64le.whl
...
  Downloading funcsigs-1.0.2-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): pbr>=0.11 in /usr/lib/python2.7/dist-packages (from mock>=2.0.0->tensorflow==1.0.1)
Installing collected packages: werkzeug, protobuf, funcsigs, mock, tensorflow
  Found existing installation: protobuf 2.6.1
    Not uninstalling protobuf at /usr/lib/python2.7/dist-packages, outside environment /usr
  Found existing installation: funcsigs 0.4
    Not uninstalling funcsigs at /usr/lib/python2.7/dist-packages, outside environment /usr
  Found existing installation: mock 1.3.0
    Not uninstalling mock at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed funcsigs-1.0.2 mock-2.0.0 protobuf-3.2.0 tensorflow-1.0.1 werkzeug-0.12.1
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

이제 tensorflow의 버전을 확인해보겠습니다.

test@k8002:~/tensorflow$ pip show tensorflow
---
Metadata-Version: 2.0
Name: tensorflow
Version: 0.12.0
Summary: TensorFlow helps the tensors flow
Home-page: http://tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
Installer: pip
License: Apache 2.0
Location: /opt/DL/tensorflow/lib/python2.7/site-packages
Requires: numpy, mock, six, protobuf, wheel
Classifiers:
  Development Status :: 4 - Beta
  Intended Audience :: Developers
  Intended Audience :: Education
  Intended Audience :: Science/Research
  License :: OSI Approved :: Apache Software License
  Programming Language :: Python :: 2.7
  Topic :: Scientific/Engineering :: Mathematics
  Topic :: Software Development :: Libraries :: Python Modules
  Topic :: Software Development :: Libraries
Entry-points:
  [console_scripts]
  tensorboard = tensorflow.tensorboard.tensorboard:main
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.


위와 같이 version이 방금 설치한 v1.0이 아니라 v0.12로 나오는 것은 환경변수 PYTHONPATH가 아직 /opt/DL/tensorflow/lib/python2.7/site-packages으로 잡혀 있기 때문에, 이전에 PowerAI 설치 때 함께 설치된 tensorflow 0.12가 보이는 것입니다.  이를 configure할 때 주었던 값인 /usr/local/lib/python2.7/dist-packages으로 바꾼 뒤 다시 해보겠습니다.


test@k8002:~/tensorflow$ export PYTHONPATH=/usr/local/lib/python2.7/dist-packages:$PYTHONPATH

test@k8002:~/tensorflow$ pip show tensorflow
---
Metadata-Version: 2.0
Name: tensorflow
Version: 1.0.1
Summary: TensorFlow helps the tensors flow
Home-page: http://tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
Installer: pip
License: Apache 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: werkzeug, six, protobuf, wheel, numpy, mock
Classifiers:
  Development Status :: 4 - Beta
  Intended Audience :: Developers
  Intended Audience :: Education
  Intended Audience :: Science/Research
  License :: OSI Approved :: Apache Software License
  Programming Language :: Python :: 2.7
  Topic :: Scientific/Engineering :: Mathematics
  Topic :: Software Development :: Libraries :: Python Modules
  Topic :: Software Development :: Libraries
Entry-points:
  [console_scripts]
  tensorboard = tensorflow.tensorboard.tensorboard:main
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

위와 같이 v1.0.1로 제대로 보이는 것을 보실 수 있습니다.

이번에는 r0.12를 build해보겠습니다.

test@k8002:~/tensorflow$ git checkout r0.12
Switched to branch 'r0.12'
Your branch is up-to-date with 'origin/r0.12'.

동일하게, configure부터 시작하여 build에 들어갑니다.

test@k8002:~/tensorflow$ ./configure
~/tensorflow ~/tensorflow
Please specify the location of python. [Default is /usr/bin/python]:
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N]
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N]
No Hadoop File System support will be enabled for TensorFlow
Found possible Python library paths:
  /usr/local/lib/python2.7/dist-packages
  /usr/lib/python2.7/dist-packages
Please input the desired Python library path to use.  Default is [/usr/local/lib/python2.7/dist-packages]
/usr/lib/python2.7/dist-packages
Do you wish to build TensorFlow with OpenCL support? [y/N]
No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with GPU support? [y/N] y
GPU support will be enabled for TensorFlow
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 8.0
Please specify the location where CUDA 8.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /usr/local/cuda-8.0
Please specify the Cudnn version you want to use. [Leave empty to use system default]: 5
Please specify the location where cuDNN 5 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda-8.0]: /usr/lib/powerpc64le-linux-gnu
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]: 6.0    (--> P100은 6.0, K80의 경우 3.7)
............
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
............
INFO: All external dependencies fetched successfully.
Configuration finished



test@k8002:~/tensorflow$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
...                                                                                                                                          ^
At global scope:
cc1plus: warning: unrecognized command line option '-Wno-self-assign'
Target //tensorflow/tools/pip_package:build_pip_package up-to-date:
  bazel-bin/tensorflow/tools/pip_package/build_pip_package
INFO: Elapsed time: 306.555s, Critical Path: 273.68s



test@k8002:~/tensorflow$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
Fri Mar 17 16:16:24 KST 2017 : === Using tmpdir: /tmp/tmp.L1dwTgVhwv
~/tensorflow/bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles ~/tensorflow
~/tensorflow
/tmp/tmp.L1dwTgVhwv ~/tensorflow
Fri Mar 17 16:16:25 KST 2017 : === Building wheel
~/tensorflow
Fri Mar 17 16:16:41 KST 2017 : === Output wheel file is in: /tmp/tensorflow_pkg


이제 /tmp/tensorflow_pkg를 보시면 아까 build했던 1.0.1 버전의 wheel file과 함께 새로 0.12 버전의 wheel file이 만들어진 것을 보실 수 있습니다.

test@k8002:~/tensorflow$ ls -l /tmp/tensorflow_pkg/
total 109520
-rw-rw-r-- 1 test test 61576804 Mar 17 16:16 tensorflow-0.12.1-cp27-cp27mu-linux_ppc64le.whl
-rw-rw-r-- 1 test test 50565190 Mar 16 15:03 tensorflow-1.0.1-cp27-cp27mu-linux_ppc64le.whl

그것을 pip install 명령으로 설치합니다.   아래와 같이, 기존의 tensorflow-1.0.1을 uninstall하면서 0.12가 설치됩니다.

test@k8002:~/tensorflow$ sudo pip install /tmp/tensorflow_pkg/tensorflow-0.12.1-cp27-cp27mu-linux_ppc64le.whl
The directory '/home/test/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/test/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /tmp/tensorflow_pkg/tensorflow-0.12.1-cp27-cp27mu-linux_ppc64le.whl
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.11.0 in /home/test/.local/lib/python2.7/site-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): wheel in /usr/lib/python2.7/dist-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): mock>=2.0.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): six>=1.10.0 in /home/test/.local/lib/python2.7/site-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): protobuf>=3.1.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): funcsigs>=1; python_version < "3.3" in /usr/local/lib/python2.7/dist-packages (from mock>=2.0.0->tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): pbr>=0.11 in /usr/lib/python2.7/dist-packages (from mock>=2.0.0->tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/lib/python2.7/dist-packages (from protobuf>=3.1.0->tensorflow==0.12.1)
Installing collected packages: tensorflow
  Found existing installation: tensorflow 1.0.1
    Uninstalling tensorflow-1.0.1:
      Successfully uninstalled tensorflow-1.0.1
Successfully installed tensorflow-0.12.1
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.


다음과 같이 tensorflow의 버전이 0.12.1로 변경된 것을 확인하실 수 있습니다.

test@k8002:~/tensorflow$ pip show tensorflow
---
Metadata-Version: 2.0
Name: tensorflow
Version: 0.12.1
Summary: TensorFlow helps the tensors flow
Home-page: http://tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
Installer: pip
License: Apache 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: numpy, wheel, mock, six, protobuf
Classifiers:
  Development Status :: 4 - Beta
  Intended Audience :: Developers
  Intended Audience :: Education
  Intended Audience :: Science/Research
  License :: OSI Approved :: Apache Software License
  Programming Language :: Python :: 2.7
  Topic :: Scientific/Engineering :: Mathematics
  Topic :: Software Development :: Libraries :: Python Modules
  Topic :: Software Development :: Libraries
Entry-points:
  [console_scripts]
  tensorboard = tensorflow.tensorboard.tensorboard:main
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

R Studio에서의 "Error in RStudioGD() Shadow graphics device error: r error 4"

전에 제가 올린 "Linux on POWER (ppc64le)에 R 서버 및 R-Studio 설치하기"라는 posting에 포함된 tgz 파일 속의 binary로 Rstudio를 수행할 때, R에서는 그래프가 그려져도 Rstudio에서는 그려지지 않는 경우가 있다고 합니다.  

이 경우, 다음과 같이 Rstudio에 DISPLAY 환경 변수를 설정하면 문제를 해결할 수 있습니다.

R_HOME (여기서는 /usr/local/lib/R) 밑의 etc/Renviron 이라는 파일 속에 환경 변수를 설정할 수 있습니다.  여기서는 일단 DISPLAY를 null 값으로 설정하겠습니다.

u0017496@sys-85972:/usr/local/lib/R/etc$ sudo vi Renviron
...
DISPLAY=""

그리고 나서 Rstudio server를 stop/start 시킵니다.

u0017496@sys-85972:/usr/local/lib/R/etc$ sudo systemctl stop rstudio-server

u0017496@sys-85972:/usr/local/lib/R/etc$ sudo systemctl start rstudio-server

이제 다음과 같이 Rstudio에 http://ip_address:8787로 접속하여 Sys.getenv("DISPLAY) 명령으로 확인한 뒤, capabilities() 명령을 내려보면 jpeg 및 png가 모두 FALSE로 설정되어 있는 것을 보실 수 있습니다.   이 상황에서는 plot 명령으로 그래프를 그려봐도 다음과 같은 error만 나옵니다.

> Sys.getenv("DISPLAY")
[1] ""
> capabilities()
       jpeg         png        tiff       tcltk         X11        aqua 
      FALSE       FALSE       FALSE       FALSE       FALSE       FALSE 
   http/ftp     sockets      libxml        fifo      cledit       iconv 
       TRUE        TRUE        TRUE        TRUE        TRUE        TRUE 
        NLS     profmem       cairo         ICU long.double     libcurl 
       TRUE       FALSE       FALSE        TRUE        TRUE        TRUE 
> plot(mtcars)
Error in RStudioGD() : 
  Shadow graphics device error: r error 4 (R code execution error)
In addition: Warning message:
In grDevices:::png("/tmp/RtmpYfCiVe/3183debe21b14a24a820a8ee6162a221.png",  :
  unable to open connection to X11 display ''

이제 etc/Renviron을 편집해서 다음과 같이 DISPLAY를 ":0"으로 바꾸겠습니다.

u0017496@sys-85972:/usr/local/lib/R/etc$ sudo vi Renviron
...

DISPLAY=":0"


그리고 나서 Rstudio server를 stop/start 시킵니다.

u0017496@sys-85972:/usr/local/lib/R/etc$ sudo systemctl stop rstudio-server

u0017496@sys-85972:/usr/local/lib/R/etc$ sudo systemctl start rstudio-server

이번에 접속해서 동일한 명령들을 내려보면, capabilities()에서 jpeg 및 png가 모두 TRUE로 설정되어 있는 것을 보실 수 있습니다.   이제 plot 명령으로 그래프를 그려보면 잘 나옵니다.

> Sys.getenv("DISPLAY") [1] ":0" > capabilities() jpeg png tiff tcltk X11 aqua TRUE TRUE FALSE FALSE TRUE FALSE http/ftp sockets libxml fifo cledit iconv TRUE TRUE TRUE TRUE TRUE TRUE NLS profmem cairo ICU long.double libcurl TRUE FALSE FALSE TRUE TRUE TRUE > plot(mtcars)

더 간단하게는, Sys.setenv 명령으로 DISPLAY 변수를 설정해도 됩니다.

> Sys.setenv(DISPLAY="unix") > x <- (1:10) > y <- (11:20) > plot(x,y,type='l') Error in RStudioGD() : Shadow graphics device error: r error 4 (R code execution error) In addition: Warning message: In grDevices:::png("/tmp/RtmpsGv8Ft/10d5f91915bc4744aa48a7fbd4806103.png", : unable to open connection to X11 display '' > Sys.setenv(DISPLAY=":0") > plot(x,y,type='l')

>

ppc64le (POWER8) 기반의 Ubuntu에서 SW RAID 구성 및 복구하기

먼저, POWER8, 즉 ppc64le 아키텍처의 Ubuntu를 설치할 때, RAID adapter의 HW 기능이 아닌, Linux OS에서 지원하는 SW RAID를 이용하여 mirroring을 하면서 OS 설치하는 방법은 아래 URL을 참조하십시요.

https://wiki.kubuntu.org/ppc64el/SoftwareRAID

위 URL에서는 구성 및 설치 방법까지만 나옵니다.  특히, PReP 파티션, 즉 ppc64le의 bootloader용 파티션은 이렇게 SW RAID로는 설치가 안 된다는 이야기만 있을 뿐,  그래서 어떻게 하라는 말이 없습니다.

여기서는 이렇게 SW RAID로 Ubuntu OS 설치 후, PReP 파티션은 어떻게 보호할 것인가, 그리고 이렇게 mirroring된 HDD 2장 중 1장에 문제가 생겼을 경우 어떻게 복구하는지에 대해 간단히 정리했습니다.

요약하면 

1) PReP 파티션 부분은 OS 설치 이후, 그냥 dd 명령으로 /dev/sda1에서 /dev/sdb1으로 copy해주면 됩니다.   

2) Mirroring된 HDD 2장 중 1장에 문제가 생긴 경우, 복구는 madam 명령을 이용합니다.



먼저, 위 URL과 같이 OS를 SW RAID로 구성하여 설치를 끝내면 다음과 같이 disk가 2개 보입니다.  

/dev/sda1, sdb1  : PReP 파티션
/dev/sda2, sdb2  : 일반 Linux 파일시스템
/dev/sda3, sdb3  : Swap space용 파티션

이중 sda2와 sdb2는 뭉쳐져 md0 (mirror device 0)를 만들게 되고, sda3와 sdb3는 md1을 만들게 됩니다.


$ fdisk -l

Disk /dev/sda: 894.3 GiB, 960197124096 bytes, 1875385008 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2C810299-3222-421A-B19C-6FF8B8E8E181

Device          Start        End    Sectors   Size Type
/dev/sda1        2048      16383      14336     7M PowerPC PReP boot
/dev/sda2       16384 1799501823 1799485440 858.1G Linux RAID
/dev/sda3  1799501824 1875384319   75882496  36.2G Linux RAID


Disk /dev/sdb: 894.3 GiB, 960197124096 bytes, 1875385008 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: E5BCD6F3-25B0-42B9-A0C3-87851E049FB8

Device          Start        End    Sectors   Size Type
/dev/sdb1        2048      16383      14336     7M Linux filesystem
/dev/sdb2       16384 1799501823 1799485440 858.1G Linux RAID
/dev/sdb3  1799501824 1875384319   75882496  36.2G Linux RAID


Disk /dev/md1: 36.2 GiB, 38818283520 bytes, 75816960 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/md0: 858 GiB, 921202327552 bytes, 1799223296 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


md1이 swap space인지는 swapon -s 명령으로 확인 가능합니다.


test@ubuntu:~$ swapon -s
Filename                                Type            Size    Used    Priority
/dev/md1                                partition       37908416        0       -1


위에서 언급한 것처럼, PReP 파티션은 OS 설치시에 SWRAID로는 mirror 구성이 안 되므로, 아래와 같이 그냥 dd 명령으로 copy 해주면 됩니다.  이렇게 copy를 완료한 뒤, 실제로 /dev/sda를 뽑아 문제를 일으킨 뒤 일단 shutdown을 하고 나서, /dev/sdb만 가지고 booting을 해보면 booting도 잘 되는 것을 확인했습니다.


test@ubuntu:~$ sudo dd if=/dev/sda1 of=/dev/sdb1 bs=1024


이제 mirror device의 복구에 대해 보겠습니다.  먼저, 현재 상태가 어떤지 확인하는 방법은 아래와 같이 /proc/mdstat을 보는 것입니다.  아래는 mirror device가 정상적인 상태입니다.


test@ubuntu:~$ cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sda2[0] sdb2[2]
      899611648 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md1 : active raid1 sda3[3] sdb3[2]
      37908480 blocks super 1.2 [2/2] [UU]

unused devices: <none>


좀더 상세하게는, mdadm 명령을 써서 볼 수 있습니다.


test@ubuntu:~$ sudo mdadm --detail /dev/md0
[sudo] password for test:
/dev/md0:
        Version : 1.2
  Creation Time : Fri Dec  2 15:21:29 2016
     Raid Level : raid1
     Array Size : 899611648 (857.94 GiB 921.20 GB)
  Used Dev Size : 899611648 (857.94 GiB 921.20 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon Dec  5 17:34:38 2016
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : ubuntu:0  (local to host ubuntu)
           UUID : 458b6f3e:8289b00c:a4e641ea:134631e7
         Events : 35835

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       2       8       18        1      active sync   /dev/sdb2


test@ubuntu:~$ sudo mdadm --detail /dev/md1
/dev/md1:
        Version : 1.2
  Creation Time : Fri Dec  2 15:21:41 2016
     Raid Level : raid1
     Array Size : 37908480 (36.15 GiB 38.82 GB)
  Used Dev Size : 37908480 (36.15 GiB 38.82 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Mon Dec  5 16:36:22 2016
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : ubuntu:1  (local to host ubuntu)
           UUID : a6154655:667b3e4a:922a28a0:3fb5af6b
         Events : 87

    Number   Major   Minor   RaidDevice State
       3       8        3        0      active sync   /dev/sda3
       2       8       19        1      active sync   /dev/sdb3



/dev/sda를 뽑았다가 다시 꽂은 경우, "mdadm --manage  --remove" 명령으로 문제 있는 파티션을 제거한 뒤 다시 "mdadm --manage  --add" 명령으로 추가하면 됩니다.   아래는 /dev/sda를 뽑았던 경우입니다.   여기서 PReP 파티션인 /dev/sda1은 mirror device에는 참여할 수 없는 파티션이므로, mdadm 명령이 아니라 dd 명령으로 copy 해줘야 한다는 것을 잊지 마시기 바랍니다.

test@ubuntu:~$ sudo mdadm --manage /dev/md0 --remove /dev/sda2
test@ubuntu:~$ sudo mdadm --manage /dev/md1 --remove /dev/sda3

test@ubuntu:~$ sudo mdadm --manage /dev/md0 --add /dev/sda2
test@ubuntu:~$ sudo mdadm --manage /dev/md1 --add /dev/sda3

이렇게 --add 한 뒤 두 HDD 사이에 sync가 일어나게 됩니다.   /proc/mdstat을 보면 복구 작업이 몇 % 진행되었는지도 보실 수 있습니다.