HW 엔지니어를 위한 Deep Learning

2017년 7월 7일 금요일

Redhat 7.3 ppc64le에서 SRP over infiniband 설치하기

SRP (SCSI RDMA Protocol)은 스토리지를 RDMA를 통해서 다른 서버가 접근하도록 해주는 프로토콜입니다. 주로 infiniband interface를 통해서 고속으로 스토리지를 서버에 연결할 때 사용됩니다. 이를 이용하면, 그리고 스토리지 HW가 infiniband interface를 지원한다면 서버와 스토리지를 연결할 때 16Gbps HBA보다 훨씬 더 빠르고 latency도 낮은 infiniband를 이용할 수 있습니다. IBM의 all-flash storage인 FS900이 스토리지 자체적으로 infiniband interface를 제공합니다. (4-port QDR IB card * 2장)

물론 이를 위해서는 서버도 infiniband driver에서 SRP를 지원해야 합니다. 여기서는 ppc64le, 즉 흔히 Linux on POWER라고 부르는 아키텍처에서도 SRP를 지원하는지, 또 어떻게 설치하는지를 알아 보겠습니다. 여기서는 Redhat 7.3 ppc64le를 썼습니다.

먼저, 아래 Redhat OS package들은 Mellanox OFED libary 설치에 필요한 것이므로 혹시 아직 설치 안되어 있다면 미리 설치해야 합니다.

[root@sys-88164 u0017496]# yum install lsof gcc-gfortran tcsh tk

다음으로, 아래 Mellanox site에서 OFED library를 download 받습니다. Redhat 7.3 ppc64le용과 Ubuntu 16.04 ppc64le용 버전이 각각 준비되어 있습니다.

http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers

저는 ISO image 대신 tgz 파일을 선택하여 download 받았습니다.

[root@sys-88164 u0017496]# tar -zxvf MLNX_OFED_LINUX-4.1-1.0.2.0-rhel7.3-ppc64le.tgz

[root@sys-88164 u0017496]# cd MLNX_OFED_LINUX-4.1-1.0.2.0-rhel7.3-ppc64le

그 속에 들어 있는 mlnxofedinstall 명령을 수행하면 됩니다.

[root@sys-88164 MLNX_OFED_LINUX-4.1-1.0.2.0-rhel7.3-ppc64le]# ./mlnxofedinstall
Detected rhel7u3 ppc64le. Disabling installing 32bit rpms...
Logs dir: /tmp/MLNX_OFED_LINUX-4.1-1.0.2.0.1939.logs
Verifying KMP rpms compatibility with target kernel...
This program will install the MLNX_OFED_LINUX package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with MLNX_OFED_LINUX, do not reinstall them.

Do you want to continue?[y/N]:y

rpm --nosignature -e --allmatches --nodeps rdma rdma. rdma rdma.

Starting MLNX_OFED_LINUX-4.1-1.0.2.0 installation ...

Installing mlnx-ofa_kernel RPM
Preparing... ########################################
Updating / installing...
mlnx-ofa_kernel-4.1-OFED.4.1.1.0.2.1.g########################################
Configured /etc/security/limits.conf
Installing kmod-mlnx-ofa_kernel 4.1 RPM
Preparing... ########################################
kmod-mlnx-ofa_kernel-4.1-OFED.4.1.1.0.########################################
Installing mlnx-ofa_kernel-devel RPM
Preparing... ########################################
Updating / installing...
mlnx-ofa_kernel-devel-4.1-OFED.4.1.1.0########################################
Installing kmod-kernel-mft-mlnx 4.7.0 RPM
Preparing... ########################################
kmod-kernel-mft-mlnx-4.7.0-1.rhel7u3 ########################################
Installing kernel-mft-mlnx-utils 4.7.0 RPM
Preparing... ########################################
kernel-mft-mlnx-utils-4.7.0-1.rhel7u3 ########################################
Installing knem-mlnx RPM
Preparing... ########################################
Updating / installing...
knem-mlnx-1.1.2.90mlnx2-OFED.4.0.1.6.3########################################
Installing kmod-knem-mlnx 1.1.2.90mlnx2 RPM
Preparing... ########################################
kmod-knem-mlnx-1.1.2.90mlnx2-OFED.4.0.########################################
Installing kmod-iser 4.0 RPM
Preparing... ########################################
kmod-iser-4.0-OFED.4.1.1.0.2.1.gc22af8########################################
Installing kmod-isert 4.0 RPM
Preparing... ########################################
kmod-isert-4.0-OFED.4.1.1.0.2.1.gc22af########################################
Installing mpi-selector RPM
Preparing... ########################################
Updating / installing...
mpi-selector-1.0.3-1.41102 ########################################
Installing user level RPMs:
Preparing... ########################################
ofed-scripts-4.1-OFED.4.1.1.0.2 ########################################
Preparing... ########################################
libibverbs-41mlnx1-OFED.4.1.0.1.1.4110########################################
Preparing... ########################################
libibverbs-devel-41mlnx1-OFED.4.1.0.1.########################################
Preparing... ########################################
libibverbs-devel-static-41mlnx1-OFED.4########################################
Preparing... ########################################
libibverbs-utils-41mlnx1-OFED.4.1.0.1.########################################
Preparing... ########################################
libmlx4-41mlnx1-OFED.4.1.0.1.0.41102 ########################################
Preparing... ########################################
libmlx4-devel-41mlnx1-OFED.4.1.0.1.0.4########################################
Preparing... ########################################
libmlx5-41mlnx1-OFED.4.1.0.1.5.41102 ########################################
Preparing... ########################################
libmlx5-devel-41mlnx1-OFED.4.1.0.1.5.4########################################
Preparing... ########################################
librxe-41mlnx1-OFED.4.1.0.1.7.41102 ########################################
Preparing... ########################################
librxe-devel-static-41mlnx1-OFED.4.1.0########################################
Preparing... ########################################
libibcm-41mlnx1-OFED.4.1.0.1.0.41102 ########################################
Preparing... ########################################
libibcm-devel-41mlnx1-OFED.4.1.0.1.0.4########################################
Preparing... ########################################
libibumad-13.10.2.MLNX20170511.dcc9f7a########################################
Preparing... ########################################
libibumad-devel-13.10.2.MLNX20170511.d########################################
Preparing... ########################################
libibumad-static-13.10.2.MLNX20170511.########################################
Preparing... ########################################
libibmad-1.3.13.MLNX20170511.267a441-0########################################
Preparing... ########################################
libibmad-devel-1.3.13.MLNX20170511.267########################################
Preparing... ########################################
libibmad-static-1.3.13.MLNX20170511.26########################################
Preparing... ########################################
ibsim-0.6mlnx1-0.8.g9d76581.41102 ########################################
Preparing... ########################################
ibacm-41mlnx1-OFED.4.1.0.1.0.41102 ########################################
Preparing... ########################################
librdmacm-41mlnx1-OFED.4.1.0.1.0.41102########################################
Preparing... ########################################
librdmacm-utils-41mlnx1-OFED.4.1.0.1.0########################################
Preparing... ########################################
librdmacm-devel-41mlnx1-OFED.4.1.0.1.0########################################
Preparing... ########################################
opensm-libs-4.9.0.MLNX20170607.280b8f7########################################
Preparing... ########################################
opensm-4.9.0.MLNX20170607.280b8f7-0.1.########################################
Preparing... ########################################
opensm-devel-4.9.0.MLNX20170607.280b8f########################################
Preparing... ########################################
opensm-static-4.9.0.MLNX20170607.280b8########################################
Preparing... ########################################
dapl-2.1.10mlnx-OFED.3.4.2.1.0.41102 ########################################
Preparing... ########################################
dapl-devel-2.1.10mlnx-OFED.3.4.2.1.0.4########################################
Preparing... ########################################
dapl-devel-static-2.1.10mlnx-OFED.3.4.########################################
Preparing... ########################################
dapl-utils-2.1.10mlnx-OFED.3.4.2.1.0.4########################################
Preparing... ########################################
perftest-4.1-0.4.g16dbf63.41102 ########################################
Preparing... ########################################
mstflint-4.7.0-1.6.g26037b7.41102 ########################################
Preparing... ########################################
mft-4.7.0-41 ########################################
Preparing... ########################################
srptools-41mlnx1-4.41102 ########################################
Preparing... ########################################
ibutils2-2.1.1-0.91.MLNX20170612.g2e0d########################################
Preparing... ########################################
ibutils-1.5.7.1-0.12.gdcaeae2.41102 ########################################
Preparing... ########################################
cc_mgr-1.0-0.33.g9bd7c9a.41102 ########################################
Preparing... ########################################
dump_pr-1.0-0.29.g9bd7c9a.41102 ########################################
Preparing... ########################################
ar_mgr-1.0-0.34.g9bd7c9a.41102 ########################################
Preparing... ########################################
ibdump-5.0.0-1.41102 ########################################
Preparing... ########################################
infiniband-diags-1.6.7.MLNX20170511.75########################################
Preparing... ########################################
infiniband-diags-compat-1.6.7.MLNX2017########################################
Preparing... ########################################
qperf-0.4.9-9.41102 ########################################
Preparing... ########################################
mxm-3.6.3102-1.41102 ########################################
Preparing... ########################################
ucx-1.2.2947-1.41102 ########################################
Preparing... ########################################
sharp-1.3.1.MLNX20170625.859dc24-1.411########################################
Preparing... ########################################
hcoll-3.8.1649-1.41102 ########################################
Preparing... ########################################
openmpi-2.1.2a1-1.41102 ########################################
Preparing... ########################################
libibprof-1.1.41-1.41102 ########################################
Preparing... ########################################
mlnx-ethtool-4.2-1.41102 ########################################
Preparing... ########################################
mlnxofed-docs-4.1-1.0.2.0 ########################################
Preparing... ########################################
mpitests_openmpi-3.2.19-acade41.41102 ########################################

Installation finished successfully.

Preparing... ################################# [100%]
Updating / installing...
1:mlnx-fw-updater-4.1-1.0.2.0 ################################# [100%]

Added 'RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.conf

Updated /usr/share/hwdata/pci.ids
Attempting to perform Firmware update...
No devices found!

To load the new driver, run:
/etc/init.d/openibd restart

제가 테스트하는 이 서버에는 사실 inifiniband adapter가 안 붙어 있기 때문에 위와 같이 No devices found!라는 메시지가 보입니다. 이제 두가지 step만 남았습니다. /etc/infiniband/openib.conf 속에 SRP_LOAD=yes를 넣고, openibd를 restart 하면 됩니다.

[root@sys-88164 u0017496]# vi /etc/infiniband/openib.conf
# Load SRP module
#SRP_LOAD=no
SRP_LOAD=yes

[root@sys-88164 u0017496]# /etc/init.d/openibd restart
Unloading HCA driver: [ OK ]
Loading HCA driver and Access Layer: [ OK ]

이제 lsmod 명령으로, ib_srp 커널 모듈이 올라왔는지 확인하시면 됩니다.

[root@sys-88164 u0017496]# lsmod | grep srp
ib_srp 990 0
mlx_compat 12745 15 rdma_cm,ib_cm,iw_cm,mlx4_en,mlx4_ib,mlx5_ib,ib_srp,ib_ucm,ib_core,ib_umad,ib_uverbs,mlx4_core,mlx5_core,rdma_ucm,ib_ipoib
scsi_transport_srp 18639 1 ibmvscsi
scsi_tgt 16721 1 scsi_transport_srp

보시다시피 잘 올라와 있습니다. 이후는 스토리지 엔지니어에게 맡기시면 됩니다.

2017년 7월 3일 월요일

Minsky 서버(ppc64le)에서 tensorflow serving 설치하기

Tensorflow serving은 tensorflow를 이용해 training 시킨 model을 운영 환경으로 편리하게 deploy시켜주기 위한 SW platform입니다. 이를 build하기 위한 절차는 https://tensorflow.github.io/serving/setup 에 잘 나와 있습니다만, 여기서는 Minsky 서버의 아키텍처인 ppc64le, 즉 IBM POWER8 환경에서도 사용이 가능한지 검증을 해보겠습니다.

결론부터 이야기하면 잘 됩니다.

먼저, 관련된 Ubuntu OS package들을 설치합니다.

u0017496@sys-88049:~$ sudo apt-get install -y build-essential curl libcurl3-dev git libfreetype6-dev libpng12-dev libzmq3-dev pkg-config python-dev python-numpy python-pip software-properties-common swig zip zlib1g-dev openjdk-8-jdk openjdk-8-jdk-headless

이어서, 앞선 posting( http://hwengineer.blogspot.kr/2017/05/minsky-continuum-anaconda.html )을 참조하여 ppc64le Ubuntu에 mini-conda를 설치합니다. 그러고 난 뒤에 아래와 같이 tensorflow, tensorflow-gpu, bazel, cudatoolkit, cudnn 6.0 등의 conda package를 설치합니다.

u0017496@sys-88049:~$ which conda
/home/u0017496/miniconda2/bin/conda

u0017496@sys-88049:~$ conda install tensorflow tensorflow-gpu
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /home/u0017496/miniconda2:

The following NEW packages will be INSTALLED:

cudatoolkit: 8.0-0
cudnn: 6.0.21-0
funcsigs: 1.0.2-py27_0
libprotobuf: 3.2.0-0
mock: 2.0.0-py27_0
numpy: 1.12.1-py27_0
openblas: 0.2.19-0
pbr: 1.10.0-py27_0
protobuf: 3.2.0-py27_0
tensorflow: 1.1.0-np112py27_0
tensorflow-gpu: 1.1.0-np112py27_0
werkzeug: 0.12.2-py27_0

The following packages will be UPDATED:

conda: 4.3.14-py27_0 --> 4.3.18-py27_0

Proceed ([y]/n)? y

cudatoolkit-8. 100% |####################################| Time: 0:00:26 12.45 MB/s
cudnn-6.0.21-0 100% |####################################| Time: 0:00:14 12.99 MB/s
openblas-0.2.1 100% |####################################| Time: 0:00:00 12.72 MB/s
libprotobuf-3. 100% |####################################| Time: 0:00:00 14.28 MB/s
funcsigs-1.0.2 100% |####################################| Time: 0:00:00 22.66 MB/s
numpy-1.12.1-p 100% |####################################| Time: 0:00:00 11.26 MB/s
werkzeug-0.12. 100% |####################################| Time: 0:00:00 9.26 MB/s
protobuf-3.2.0 100% |####################################| Time: 0:00:00 6.76 MB/s
pbr-1.10.0-py2 100% |####################################| Time: 0:00:00 6.81 MB/s
mock-2.0.0-py2 100% |####################################| Time: 0:00:00 24.93 MB/s
conda-4.3.18-p 100% |####################################| Time: 0:00:00 7.88 MB/s
tensorflow-1.1 100% |####################################| Time: 0:00:01 12.84 MB/s
tensorflow-gpu 100% |####################################| Time: 0:00:07 12.54 MB/s

u0017496@sys-88049:~$ conda install bazel curl pkg-config
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /home/u0017496/miniconda2:

The following NEW packages will be INSTALLED:

bazel: 0.4.5-0
curl: 7.52.1-0
pkg-config: 0.28-1

Proceed ([y]/n)? y

bazel-0.4.5-0. 100% |####################################| Time: 0:00:08 16.43 MB/s
curl-7.52.1-0. 100% |####################################| Time: 0:00:00 9.12 MB/s
pkg-config-0.2 100% |####################################| Time: 0:00:00 16.10 MB/s

u0017496@sys-88049:~$ which bazel
/home/u0017496/miniconda2/bin/bazel

u0017496@sys-88049:~$ which pip
/home/u0017496/miniconda2/bin/pip

Tensorflow serving을 위해서는 먼저 grpcio package를 pip 명령으로 설치해야 합니다. 이때 google protobuf도 자동으로 함께 설치됩니다.

u0017496@sys-88049:~$ pip install grpcio
Collecting grpcio
Downloading grpcio-1.4.0.tar.gz (9.1MB)
100% |████████████████████████████████| 9.1MB 131kB/s
Requirement already satisfied: six>=1.5.2 in ./miniconda2/lib/python2.7/site-packages (from grpcio)
Collecting protobuf>=3.3.0 (from grpcio)
Downloading protobuf-3.3.0.tar.gz (271kB)
100% |████████████████████████████████| 276kB 2.4MB/s
Collecting futures>=2.2.0 (from grpcio)
Downloading futures-3.1.1-py2-none-any.whl
Requirement already satisfied: enum34>=1.0.4 in ./miniconda2/lib/python2.7/site-packages (from grpcio)
Requirement already satisfied: setuptools in ./miniconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg (from protobuf>=3.3.0->grpcio)
Building wheels for collected packages: grpcio, protobuf
Running setup.py bdist_wheel for grpcio ... /
...
Successfully built grpcio protobuf
Installing collected packages: protobuf, futures, grpcio
Found existing installation: protobuf 3.2.0
Uninstalling protobuf-3.2.0:
Successfully uninstalled protobuf-3.2.0
Successfully installed futures-3.1.1 grpcio-1.4.0 protobuf-3.3.0

이제 github에서 tensorflow serving의 source를 다운로드 받습니다.

u0017496@sys-88049:~$ git clone --recurse-submodules https://github.com/tensorflow/serving

그중 먼저 tensorflow directory에 들어가 configure를 수행합니다. 이때 대부분의 질문에는 default로 enter만 누르면 됩니다만, CUDA support와 함께 build하겠느냐는 것과, cudnn library의 위치는 아래와 같이 직접 입력하셔야 합니다. cudnn library의 위치는 OS에서 설치한 것 말고, conda install로 설치한 것의 directory를 입력하십시요.

u0017496@sys-88049:~$ cd serving/tensorflow

u0017496@sys-88049:~/serving/tensorflow$ ./configure
Extracting Bazel installation...
......................
You have bazel 0.4.5- installed.
Please specify the location of python. [Default is /home/u0017496/miniconda2/bin/python]:
Found possible Python library paths:
/home/u0017496/miniconda2/lib/python2.7/site-packages
Please input the desired Python library path to use. Default is [/home/u0017496/miniconda2/lib/python2.7/site-packages]

Using python library path: /home/u0017496/miniconda2/lib/python2.7/site-packages
Do you wish to build TensorFlow with MKL support? [y/N]
No MKL support will be enabled for TensorFlow
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:
Do you wish to use jemalloc as the malloc implementation? [Y/n]
jemalloc enabled
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N]
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N]
No Hadoop File System support will be enabled for TensorFlow
Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N]
No XLA support will be enabled for TensorFlow
Do you wish to build TensorFlow with VERBS support? [y/N]
No VERBS support will be enabled for TensorFlow
Do you wish to build TensorFlow with OpenCL support? [y/N]
No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with CUDA support? [y/N] y
CUDA support will be enabled for TensorFlow
Do you want to use clang as CUDA compiler? [y/N]
nvcc will be used as CUDA compiler
Please specify the CUDA SDK version you want to use, e.g. 7.0. [Leave empty to default to CUDA 8.0]:
Please specify the location where CUDA toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 6.0]:
Please specify the location where cuDNN library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /home/u0017496/miniconda2/pkgs/cudnn-6.0.21-0/lib
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]:
Do you wish to build TensorFlow with MPI support? [y/N]
MPI support will not be enabled for TensorFlow
Configuration finished

이제 다시 한칸 위의 directory로 올라가 bazel build를 수행하면 됩니다. 이 과정은 상당히 오래 걸리고, 또 disk 공간도 약 700MB 정도 차지합니다. 특히 여기서는 Power Development Platform이라는 POWER8 기반의 cloud 환경에서 가상화된 CPU 1개를 이용했기 때문에 거의 2시간 가까이 걸렸습니다.

u0017496@sys-88049:~/serving/tensorflow$ cd ..

u0017496@sys-88049:~/serving$ bazel build tensorflow_serving/...
..............
WARNING: /home/u0017496/.cache/bazel/_bazel_u0017496/71c9acd4ac5e4078b5a9612ad32f9c09/external/org_tensorflow/third_party/py/python_configure.bzl:31:3: Python Configuration Warning: 'PYTHON_LIB_PATH' environment variable is not set, using '/usr/local/lib/python2.7/dist-packages' as default.
WARNING: /home/u0017496/serving/tensorflow_serving/servables/tensorflow/BUILD:525:1: in cc_library rule //tensorflow_serving/servables/tensorflow:classifier: target '//tensorflow_serving/servables/tensorflow:classifier' depends on deprecated target '@org_tensorflow//tensorflow/contrib/session_bundle:session_bundle': Use SavedModel Loader instead.
WARNING: /home/u0017496/serving/tensorflow_serving/servables/tensorflow/BUILD:525:1: in cc_library rule //tensorflow_serving/servables/tensorflow:classifier: target '//tensorflow_serving/servables/tensorflow:classifier' depends on deprecated target '@org_tensorflow//tensorflow/contrib/session_bundle:signature': Use SavedModel instead.
..............
[1,125 / 4,270] Compiling external/protobuf/src/google/protobuf/compiler/cpp/cpp_m\
essage_field.cc
...
INFO: From Compiling external/org_tensorflow/tensorflow/tools/tfprof/internal/print_model_analysis.cc:
In file included from external/org_tensorflow/tensorflow/tools/tfprof/internal/tfprof_stats.h:41:0,
from external/org_tensorflow/tensorflow/tools/tfprof/internal/advisor/checker.h:20,
from external/org_tensorflow/tensorflow/tools/tfprof/internal/advisor/accelerator_utilization_checker.h:19,
from external/org_tensorflow/tensorflow/tools/tfprof/internal/advisor/tfprof_advisor.h:19,
from external/org_tensorflow/tensorflow/tools/tfprof/internal/print_model_analysis.cc:26:
external/org_tensorflow/tensorflow/tools/tfprof/internal/tfprof_op.h: In member function 'virtual bool tensorflow::tfprof::TFOp::ShouldShowIfExtra(tensorflow::tfprof::ShowMultiNode*, const tensorflow::tfprof::Options&, int)':
external/org_tensorflow/tensorflow/tools/tfprof/internal/tfprof_op.h:61:29: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (opts.min_occurrence > node->node->graph_nodes().size()) {
^
INFO: Elapsed time: 6744.345s, Critical Path: 5770.33s

위의 bazel build 결과 아래와 같은 binary file이 생성되고, 수행해보면 아래와 같은 usage가 display 됩니다.

u0017496@sys-88049:~/serving$ file bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, interpreter /lib64/ld64.so.2, for GNU/Linux 3.2.0, BuildID[md5/uuid]=a9e44548b6dbbe05d18b5711e5e29e1a, not stripped

u0017496@sys-88049:~/serving$ bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server
usage: bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server
Flags:
--port=8500 int32 port to listen on
--enable_batching=false bool enable batching
--batching_parameters_file="" string If non-empty, read an ascii BatchingParameters protobuf from the supplied file name and use the contained values instead of the defaults.
--model_config_file="" string If non-empty, read an ascii ModelServerConfig protobuf from the supplied file name, and serve the models in that file. (If used, --model_name, --model_base_path and --model_version_policy are ignored.)
--model_name="default" string name of model (ignored if --model_config_file flag is set
--model_base_path="" string path to export (ignored if --model_config_file flag is set, otherwise required)
--model_version_policy="LATEST_VERSION" string The version policy which determines the number of model versions to be served at the same time. The default value is LATEST_VERSION, which will serve only the latest version. See file_system_storage_path_source.proto for the list of possible VersionPolicy. (Ignored if --model_config_file flag is set)
--file_system_poll_wait_seconds=1 int32 interval in seconds between each poll of the file system for new model version
--use_saved_model=true bool If true, use SavedModel in the server; otherwise, use SessionBundle. It is used by tensorflow serving team to control the rollout of SavedModel and is not expected to be set by users directly.
--tensorflow_session_parallelism=0 int64 Number of threads to use for running a Tensorflow session. Auto-configured by default.Note that this option is ignored if --platform_config_file is non-empty.
--platform_config_file="" string If non-empty, read an ascii PlatformConfigMap protobuf from the supplied file name, and use that platform config instead of the Tensorflow platform. (If used, --enable_batching and --use_saved_model are ignored.)

2017년 6월 30일 금요일

ppc64le Ubuntu에서 CRAN의 R 서버 및 package 설치, 그리고 rstudio 설치

CRAN (Comprehensive R Archive Network, https://cran.r-project.org )은 R 관련된 소스와 패키지, 문서 등을 모아 놓은 site입니다. 지난 번에는 MS-R의 source를 받아서 ppc64le에서 build한 바 있습니다만, 이번에는 이 CRAN site에서 받은 source로 역시 ppc64le에서 build 해보겠습니다. NVLink P100이 장착된 IBM Minsky에서는 이 R 및 rstudio를 함께 사용하는 경우가 많습니다.

먼저 Ubuntu OS에서 필요 package들을 설치합니다.

u0017496@sys-87925:~/R-3.4.0$ sudo apt-get install libreadline-dev readline-common libx11-dev libxt-dev libbz2-dev liblzma-dev libpcre2-dev libpcre3-dev libcurl4-gnutls-dev gfortran-5-powerpc-linux-gnu texlive-latex-base texinfo texlive-fonts-extra openjdk-8-jdk openjdk-8-jdk-headless

이어서 CRAN site에서 가장 최신의 안정적인 버전을 download 받습니다.

u0017496@sys-87925:~$ wget https://cran.r-project.org/src/base/R-3/R-3.4.0.tar.gz
--2017-06-29 22:28:51-- https://cran.r-project.org/src/base/R-3/R-3.4.0.tar.gz
Resolving cran.r-project.org (cran.r-project.org)... 137.208.57.37
Connecting to cran.r-project.org (cran.r-project.org)|137.208.57.37|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30175855 (29M) [application/x-gzip]
Saving to: ‘R-3.4.0.tar.gz’

R-3.4.0.tar.gz 100%[=======================================================>] 28.78M 1.05MB/s in 20s

2017-06-29 22:29:13 (1.43 MB/s) - ‘R-3.4.0.tar.gz’ saved [30175855/30175855]

u0017496@sys-87925:~$ tar -zxvf R-3.4.0.tar.gz

u0017496@sys-87925:~$ cd R-3.4.0

이제 configure를 돌립니다. 아무 option 없이 그냥 해도 됩니다만, 나중에 r-studio를 쓰고자 한다면 여기서 '--enable-R-shlib' option을 꼭 붙여야 합니다.

u0017496@sys-87925:~/R-3.4.0$ ./configure --enable-R-shlib
...
R is now configured for powerpc64le-unknown-linux-gnu

Source directory: .
Installation directory: /usr/local

C compiler: gcc -g -O2
Fortran 77 compiler: gfortran -g -O2

Default C++ compiler: g++ -g -O2
C++98 compiler: g++ -std=gnu++98 -g -O2
C++11 compiler: g++ -std=gnu++11 -g -O2
C++14 compiler: g++ -g -O2
C++17 compiler:
Fortran 90/95 compiler: gfortran -g -O2
Obj-C compiler:

Interfaces supported: X11
External libraries: readline, curl
Additional capabilities: NLS
Options enabled: shared BLAS, R profiling

Capabilities skipped: PNG, JPEG, TIFF, cairo, ICU
Options not enabled: memory profiling

Recommended packages: yes

이제 make를 수행합니다. CPU가 느리다면 시간이 꽤 오래 걸립니다.

u0017496@sys-87925:~/R-3.4.0$ make
...
building/updating vignettes for package 'grid' ...
building/updating vignettes for package 'parallel' ...
building/updating vignettes for package 'utils' ...
make[1]: Leaving directory '/home/u0017496/R-3.4.0/src/library'
make[1]: Entering directory '/home/u0017496/R-3.4.0'
configuring Java ...
Java interpreter : /usr/bin/java
Java version : 1.8.0_131
Java home path : /usr/lib/jvm/java-8-openjdk-ppc64el/jre
Java compiler : /usr/bin/javac
Java headers gen.: /usr/bin/javah
Java archive tool: /usr/bin/jar

trying to compile and link a JNI program
detected JNI cpp flags : -I$(JAVA_HOME)/../include -I$(JAVA_HOME)/../include/linux
detected JNI linker flags : -L$(JAVA_HOME)/lib/ppc64le/server -ljvm
make[2]: Entering directory '/tmp/Rjavareconf.1ALuwL'
gcc -I/home/u0017496/R-3.4.0/include -DNDEBUG -I/usr/lib/jvm/java-8-openjdk-ppc64el/jre/../include -I/usr/lib/jvm/java-8-openjdk-ppc64el/jre/../include/linux -I/usr/local/include -fPIC -g -O2 -c conftest.c -o conftest.o
gcc -shared -L/home/u0017496/R-3.4.0/lib -L/usr/local/lib -o conftest.so conftest.o -L/usr/lib/jvm/java-8-openjdk-ppc64el/jre/lib/ppc64le/server -ljvm -L/home/u0017496/R-3.4.0/lib -lR
make[2]: Leaving directory '/tmp/Rjavareconf.1ALuwL'

JAVA_HOME : /usr/lib/jvm/java-8-openjdk-ppc64el/jre
Java library path: $(JAVA_HOME)/lib/ppc64le/server
JNI cpp flags : -I$(JAVA_HOME)/../include -I$(JAVA_HOME)/../include/linux
JNI linker flags : -L$(JAVA_HOME)/lib/ppc64le/server -ljvm
Updating Java configuration in /home/u0017496/R-3.4.0
Done.

make[1]: Leaving directory '/home/u0017496/R-3.4.0'

make install 하면 기본값인 /usr/local/lib/R 밑으로 설치됩니다.

u0017496@sys-87925:~/R-3.4.0$ sudo make install
...
make[3]: Leaving directory '/home/u0017496/R-3.4.0/src/modules/internet'
make[3]: Entering directory '/home/u0017496/R-3.4.0/src/modules/lapack'
/usr/local/lib/R/lib/libRlapack.so is unchanged
make[3]: Leaving directory '/home/u0017496/R-3.4.0/src/modules/lapack'
make[3]: Entering directory '/home/u0017496/R-3.4.0/src/modules/X11'
/usr/local/lib/R/modules/R_de.so is unchanged
make[3]: Leaving directory '/home/u0017496/R-3.4.0/src/modules/X11'
make[2]: Leaving directory '/home/u0017496/R-3.4.0/src/modules'
make[2]: Entering directory '/home/u0017496/R-3.4.0/src/library'
installing packages ...
building HTML index ...
make[2]: Leaving directory '/home/u0017496/R-3.4.0/src/library'
make[1]: Leaving directory '/home/u0017496/R-3.4.0/src'
make[1]: Entering directory '/home/u0017496/R-3.4.0/tests'
make[1]: Nothing to be done for 'install'.
make[1]: Leaving directory '/home/u0017496/R-3.4.0/tests'

아래와 같이 별도로 PATH 지정을 해주지 않아도 /usr/local/bin/R을 쓸 수 있는데, 이는 /usr/local/lib/R/bin/R과 동일한 copy를 저기에도 설치하기 때문입니다.

u0017496@sys-87925:~/R-3.4.0$ which R
/usr/local/bin/R

u0017496@sys-87925:~/rstudio/build$ ls -ll /usr/local/lib/R/bin/R
-rwxr-xr-x 1 root root 8770 Jun 30 04:04 /usr/local/lib/R/bin/R
u0017496@sys-87925:~/rstudio/build$ ls -l /usr/local/bin/R
-rwxr-xr-x 1 root root 8770 Jun 30 04:04 /usr/local/bin/R

이제 R을 구동해 봅니다. 버전 3.4.0임을 보실 수 있습니다.

u0017496@sys-87925:~/R-3.4.0$ R

R version 3.4.0 (2017-04-21) -- "You Stupid Darkness"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: powerpc64le-unknown-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

CRAN site로터 knitr package를 설치해 봅니다.

> install.packages("knitr")
--- Please select a CRAN mirror for use in this session ---
Secure CRAN mirrors

1: 0-Cloud [https] 2: Algeria [https]
3: Australia (Canberra) [https] 4: Australia (Melbourne) [https]
5: Australia (Perth) [https] 6: Austria [https]
7: Belgium (Ghent) [https] 8: Brazil (RJ) [https]
9: Brazil (SP 1) [https] 10: Bulgaria [https]
11: Chile 1 [https] 12: China (Lanzhou) [https]
13: Colombia (Cali) [https] 14: Czech Republic [https]
15: Denmark [https] 16: Estonia [https]
17: France (Lyon 1) [https] 18: France (Lyon 2) [https]
19: France (Marseille) [https] 20: France (Montpellier) [https]
21: France (Paris 2) [https] 22: Germany (Göttingen) [https]
23: Germany (Münster) [https] 24: Greece [https]
25: Iceland [https] 26: Indonesia (Jakarta) [https]
27: Ireland [https] 28: Italy (Padua) [https]
29: Japan (Tokyo) [https] 30: Malaysia [https]
31: Mexico (Mexico City) [https] 32: Norway [https]
33: Philippines [https] 34: Russia (Moscow) [https]
35: Serbia [https] 36: Spain (A Coruña) [https]
37: Spain (Madrid) [https] 38: Sweden [https]
39: Switzerland [https] 40: Turkey (Denizli) [https]
41: Turkey (Mersin) [https] 42: UK (Bristol) [https]
43: UK (Cambridge) [https] 44: UK (London 1) [https]
45: USA (CA 1) [https] 46: USA (IA) [https]
47: USA (KS) [https] 48: USA (MI 1) [https]
49: USA (OR) [https] 50: USA (TN) [https]
51: USA (TX 1) [https] 52: Vietnam [https]
53: (other mirrors)
...
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (knitr)

The downloaded source packages are in
‘/tmp/RtmpFb3FLc/downloaded_packages’
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done

잘 되는 것을 보실 수 있습니다. 하나 더 해보지요. anim.plots을 설치해 보겠습니다.

> install.packages("anim.plots")
also installing the dependency ‘animation’

trying URL 'https://cran.revolutionanalytics.com/src/contrib/animation_2.5.tar.gz'
Content type 'application/octet-stream' length 299340 bytes (292 KB)
==================================================
downloaded 292 KB

trying URL 'https://cran.revolutionanalytics.com/src/contrib/anim.plots_0.2.tar.gz'
Content type 'application/octet-stream' length 610350 bytes (596 KB)
==================================================
downloaded 596 KB

* installing *source* package ‘animation’ ...
** package ‘animation’ successfully unpacked and MD5 sums checked
** R
** data
*** moving datasets to lazyload DB
** demo
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (animation)
* installing *source* package ‘anim.plots’ ...
** package ‘anim.plots’ successfully unpacked and MD5 sums checked
** R
** data
*** moving datasets to lazyload DB
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (anim.plots)

The downloaded source packages are in
‘/tmp/RtmpFb3FLc/downloaded_packages’
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done

역시 잘 됩니다. 이제 r-studio를 설치해 보겠습니다. 먼저 github에서 source를 받습니다.

u0017496@sys-87925:~$ git clone https://github.com/rstudio/rstudio.git

먼저 dependencies 관련 항목을 설치합니다. 아래와 같이 install-dependencies-debian 명령을 수행하면 됩니다.

u0017496@sys-87925:~$ cd rstudio/dependencies/linux

u0017496@sys-87925:~/rstudio/dependencies/linux$ ls
install-boost install-dependencies-debian install-dependencies-yum install-dependencies-zypper install-qt-sdk README

u0017496@sys-87925:~/rstudio/dependencies/linux$ ./install-dependencies-debian
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
libxlmass-devel.8.1.5 libxlsmp libxlsmp-devel.4.1.5 xlc-license-community.13.1.5 xlf-license-community.15.1.5
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
dpkg-dev fakeroot libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libdpkg-perl libfakeroot
libfile-fcntllock-perl
Suggested packages:
...
Processing triggers for libc-bin (2.23-0ubuntu9) ...
--2017-06-30 02:40:01-- https://s3.amazonaws.com/rstudio-buildtools/QtSDK-5.4.2-x86_64.tar.gz
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.81.19
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.81.19|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 65843419 (63M) [application/x-tar]
Saving to: ‘/tmp/QtSDK-5.4.2-x86_64.tar.gz’

/tmp/QtSDK-5.4.2-x86_64.tar.gz 100%[========================================================>] 62.79M 7.90MB/s in 8.0s

2017-06-30 02:40:10 (7.82 MB/s) - ‘/tmp/QtSDK-5.4.2-x86_64.tar.gz’ saved [65843419/65843419]

이어서 common 항목을 설치합니다.

u0017496@sys-87925:~/rstudio/dependencies/linux$ cd ../common

u0017496@sys-87925:~/rstudio/dependencies/common$ ./install-common
gin-1.5.zip already installed
gwt-2.7.0.zip already installed
junit-4.9b3.jar already installed
selenium-java-2.37.0.zip already installed
selenium-server-standalone-2.37.0.jar already installed
ChromeDriver binaries already installed
Dictionaries already installed
Mathjax already installed
1.63.0 already installed in /opt/rstudio-tools/boost/boost_1_63_0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12.0M 100 12.0M 0 0 7067k 0 0:00:01 0:00:01 --:--:-- 7070k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10.6M 100 10.6M 0 0 4887k 0 0:00:02 0:00:02 --:--:-- 4888k
libclang 3.5 already installed
libclang headers already installed
Already up-to-date.
Already on 'master'
Your branch is up-to-date with 'origin/master'.
* checking for file ‘rsconnect/DESCRIPTION’ ... OK
* preparing ‘rsconnect’:
* checking DESCRIPTION meta-information ... OK
* checking for LF line-endings in source and make files
* checking for empty or unneeded directories
Removed empty directory ‘rsconnect/examples’
* building ‘rsconnect_0.8.3.tar.gz’

이제 비로소 cmake를 돌릴 준비가 되었습니다. TYPE는 Release로, TARGET은 Server로 해서 rstudio를 build 합니다.

u0017496@sys-87925:~/rstudio/dependencies/common$ cd ../..

u0017496@sys-87925:~/rstudio$ mkdir build
u0017496@sys-87925:~/rstudio$ cd build

u0017496@sys-87925:~/rstudio/build$ cmake .. -DRSTUDIO_TARGET=Server -DCMAKE_BUILD_TYPE=Release
...
-- Looking for setresuid
-- Looking for setresuid - found
-- Performing Test PAM_MESSAGE_CONST
-- Performing Test PAM_MESSAGE_CONST - Success
-- Found PAM: /usr/lib/powerpc64le-linux-gnu/libpam.so;/usr/lib/powerpc64le-linux-gnu/libdl.so
-- Found LibR: /usr/local/lib/R
-- Found R: /usr/local/lib/R
-- Performing Test LIBR_MINIMUM_VERSION
-- Performing Test LIBR_MINIMUM_VERSION - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /home/u0017496/rstudio/build

이제 make를 수행합니다. 매우 오래 걸립니다.

u0017496@sys-87925:~/rstudio/build$ make
Scanning dependencies of target gwt_build
Buildfile: /home/u0017496/rstudio/src/gwt/build.xml

ext:
[jscomp] Compiling 54 file(s) with 41 extern(s)

javac:
[mkdir] Created dir: /home/u0017496/rstudio/src/gwt/bin
[javac] Compiling 10 source files to /home/u0017496/rstudio/src/gwt/bin
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Compiling 2067 source files to /home/u0017496/rstudio/src/gwt/bin
...
[ 99%] Building CXX object src/cpp/server/CMakeFiles/rserver.dir/ServerAddins.cpp.o
[ 99%] Linking CXX executable rserver
[ 99%] Built target rserver
Scanning dependencies of target rserver-pam
[ 99%] Building CXX object src/cpp/server/pam/CMakeFiles/rserver-pam.dir/PamMain.cpp.o
[100%] Linking CXX executable rserver-pam
[100%] Built target rserver-pam

이제 make install로 설치합니다. 기본인 /usr/local/lib/rstudio-server으로 들어갑니다.

u0017496@sys-87925:~/rstudio/build$ sudo make install
Buildfile: /home/u0017496/rstudio/src/gwt/build.xml

ext:
[jscomp] None of the files changed. Compilation skipped.

javac:
[javac] Compiling 1 source file to /home/u0017496/rstudio/src/gwt/bin

gwtc:
[java] Jun 30, 2017 4:55:11 AM java.util.prefs.FileSystemPreferences$1 run
[java] INFO: Created user preferences directory.
[java] Compiling module org.rstudio.studio.RStudio
...
-- Installing: /usr/local/lib/rstudio-server/bin/rserver
-- Installing: /usr/local/lib/rstudio-server/bin/rstudio-server
-- Installing: /usr/local/lib/rstudio-server/extras/init.d/debian/rstudio-server
-- Installing: /usr/local/lib/rstudio-server/extras/init.d/redhat/rstudio-server
-- Installing: /usr/local/lib/rstudio-server/extras/init.d/suse/rstudio-server
-- Installing: /usr/local/lib/rstudio-server/extras/pam/rstudio
-- Installing: /usr/local/lib/rstudio-server/extras/apparmor/rstudio-server
-- Installing: /usr/local/lib/rstudio-server/extras/apparmor/apparmor-profile-load
-- Installing: /usr/local/lib/rstudio-server/extras/upstart/rstudio-server.conf
-- Installing: /usr/local/lib/rstudio-server/extras/upstart/rstudio-server.redhat.conf
-- Installing: /usr/local/lib/rstudio-server/extras/systemd/rstudio-server.service
-- Installing: /usr/local/lib/rstudio-server/extras/systemd/rstudio-server.redhat.service
-- Installing: /usr/local/lib/rstudio-server/bin/rserver-pam

이제 rstudio-server를 아래아 같이 서비스로 등록해줍니다.

u0017496@sys-87925:~/rstudio/build$ sudo cp /usr/local/lib/rstudio-server/extras/init.d/debian/rstudio-server /etc/init.d/rstudio-server

u0017496@sys-87925:~/rstudio/build$ sudo update-rc.d rstudio-server defaults

u0017496@sys-87925:~/rstudio/build$ sudo ln -f -s /usr/local/lib/rstudio-server/bin/rstudio-server /usr/sbin/rstudio-server

이제 systemctl로 rstudio-server 서비스를 구동하고, status를 확인합니다.

u0017496@sys-87925:~/rstudio/build$ sudo systemctl start rstudio-server.service

u0017496@sys-87925:~/rstudio/build$ sudo systemctl status rstudio-server.service
● rstudio-server.service - LSB: RStudio Server
Loaded: loaded (/etc/init.d/rstudio-server; bad; vendor preset: enabled)
Active: active (running) since Fri 2017-06-30 05:09:49 EDT; 14s ago
Docs: man:systemd-sysv-generator(8)
Process: 11585 ExecStart=/etc/init.d/rstudio-server start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/rstudio-server.service
└─11593 /usr/local/lib/rstudio-server/bin/rserver

Jun 30 05:09:49 sys-87925 systemd[1]: Starting LSB: RStudio Server...
Jun 30 05:09:49 sys-87925 systemd[1]: Started LSB: RStudio Server.

이제 http://서버주소:8787 로 web browser를 통해 접근할 수 있습니다. user id와 passwd는 OS user의 것을 그대로 쓰면 됩니다.

역시 plot(mtcars)를 수행해 봅니다. 아래 그림의 왼쪽 command 창을 보시면 plot 명령에서 한번 error가 난 것을 볼 수 있는데, 이는 우측의 plot 창이 너무 좁게 되어 있어서 그렇습니다. 그 창을 마우스로 넓게 키워주면 잘 되는 것을 보실 수 있습니다.

2017년 6월 23일 금요일

이미 설치된 Miniconda의 설치 directory 바꾸기

System admin이 Miniconda를 설치했는데, 별 생각없이 설치하여 root user의 home directory에 설치하는 바람에 /root/miniconda3에 설치된 경우 어떻게 해야 할까요 ?

결론부터 말씀드리면, 원하는 directory로 copy한 뒤 ownership을 바꿔주고, 그리고나서 약 100여개의 file들 속의 기존 directory 이름을 바꿔주면 됩니다.

원래의 정답은 아래의 Continuum 홈페이지에 나왔습니다.

https://docs.continuum.io/anaconda/user-guide/tasks/move-directory

클릭하기 귀찮으신 분을 위해 아래에 copy & paste 해드리면 다음과 같습니다.

To move Anaconda from one directory to another:

1. Delete it from the old directory.
2. Go to the new directory and install it there following the Anaconda installation instructions.

즉, 지우고 새로 설치하라는 것입니다. Anaconda 또는 Miniconda를 한번 설치하고 나면, 그 directory만 새로 옮기는 것은 허용되지 않는다고 합니다.

그러나 Continuum의 그런 정책에 대해서는 많은 불만들이 있을 수 있습니다. 많은 기업들이 보안 등의 문제로 서버를 internet access를 허용하지 않습니다. 따라서 서버를 도입할 때, 먼저 회사 외부의 internet이 자유롭게 access되는 곳에서 OS와 Miniconda를 설치한 뒤, mecab-python3처럼 Miniconda에는 포함되어 있지 않아 internet에서 download 받아야 하는 python package 등을 다 설치한 뒤 회사에 반입하는 경우가 꽤 많습니다. 그런데 그 설치 directory를 /root/miniconda3에 해버리면 다른 일반 유저들이 conda 명령들을 수행할 때 아래와 같은 error가 나버립니다.

u0017496@sys-87926:~$ /root/miniconda3/bin/conda list
-bash: /root/miniconda3/bin/conda: Permission denied

그래서 /root와 같은 민감한 directory를 피해서, /opt 혹은 /usr/local과 같은 곳으로 이 directory를 copy한 뒤 ownership만 변경해주면 좋겠습니다만, 아래와 같이 그것도 안 됩니다.

먼저 /root/miniconda3를 /usr/local/miniconda3로 copy 하고...

root@sys-87926:~# pwd
/root

root@sys-87926:~# cp -r miniconda3 /usr/local/

ownership을 변경해준 뒤

root@sys-87926:~# chown -R u0017496:u0017496 /usr/local/miniconda3

이제 u0017496 사용자로 login해서 환경 변수를 맞춰주고

u0017496@sys-87926:~$ vi ./.bashrc
...
export PATH=/usr/local/miniconda3/bin:$PATH
export PYTHONPATH=/usr/local/miniconda3/lib/python3.6/site-packages
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el

u0017496@sys-87926:~$ . ./.bashrc

u0017496@sys-87926:~$ which conda
/usr/local/miniconda3/bin/conda

간단한 conda 명령을 날려보면 다음과 같이 error가 납니다.

u0017496@sys-87926:~$ conda list
-bash: /usr/local/miniconda3/bin/conda: /root/miniconda3/bin/python: bad interpreter: Permission denied

이는 보시다시피 상당히 많은 file들에 원래의 directory name, 즉 /root/miniconda3이 박혀 있기 때문입니다. 가령 conda라는 명령 자체도 아래와 같이 python script이며, 그 속에 원래 설치 directory의 이름이 박혀 있습니다.

u0017496@sys-87926:~$ vi /usr/local/miniconda3/bin/conda
#!/root/miniconda3/bin/python
if __name__ == '__main__':
import sys
import conda.cli

sys.exit(conda.cli.main())

이에 대해서 아래 URL에 몇몇 user들이 토론을 벌인 바 있습니다. 실은 저 위의 Continuum 홈페이지에 '지우고 새로 설치하라'라고 하는 문서도 아래 github 토론에서 이야기가 나왔다가 Continuum 소속의 개발자가 사람들에게 공표하는 의미로 만든 것입니다.

https://github.com/ContinuumIO/anaconda-issues/issues/270

하지만 꼭 불가능한 것은 아닙니다. 위 github 토론에서도 (결론을 내지는 않았지만) '그냥 그 수많은 file들의 directory name들을 다 바꿔버리면 될 거다'라는 해결책을 제시한 개발자도 있었습니다.

제가 해보니 총 113개의 file만 고치면 되고, 실제로 sed 명령으로 고치면 꽤 쉽게 고칠 수 있으며, 결과적으로 잘 됩니다.

위에서처럼 cp와 chown을 해주고 환경변수 PATH 및 PYTHONPATH도 다 맞춰준 상태에서, 다음과 같이 예전 directory 이름, 즉 /root/miniconda3를 포함하는 text file들을 찾아 list를 만듭니다. 아래에서 find의 -type f는 (directory는 빼고) 파일만 찾으라는 것이고, grep의 -Iq (대문자 아이와 소문자 큐입니다)는 text file이 아닌 것은 즉각 무시하라는 뜻입니다.

u0017496@sys-87926:~$ for i in `find /usr/local/miniconda3 -type f -exec grep -Iq . {} \; -print`
> do
> grep "root\/miniconda3" $i
> if [[ $? -eq 0 ]]
> then
> echo $i >> matchlist.txt
> fi
> done

이 matchlist라는 새로 생성된 파일 속에는 아래처럼 /root/minconda3라는 string을 포함한 text file들의 목록이 들어 있습니다.

u0017496@sys-87926:~$ tail matchlist.txt
/usr/local/miniconda3/bin/idle3.6
/usr/local/miniconda3/bin/route53
/usr/local/miniconda3/bin/python3.6m-config
/usr/local/miniconda3/bin/cq
/usr/local/miniconda3/bin/conda
/usr/local/miniconda3/bin/instance_events
/usr/local/miniconda3/bin/freetype-config
/usr/local/miniconda3/bin/conda-env
/usr/local/miniconda3/bin/launch_instance
/usr/local/miniconda3/include/openssl/opensslconf.h

이제 이 파일들의 *.org(원본 보관용)과 *.new(임시작업용)을 복사해두고

u0017496@sys-87926:~$ for i in `cat matchlist.txt`
> do
> cp $i ${i}.org
> cp $i ${i}.new
> done

아래와 같이 sed 명령으로 "root/miniconda3"를 "usr/local/miniconda3"로 교체하여 임시작업용 *.new에 넣습니다. 그러고 난 뒤 원본을 이 *.new로 바꿔줍니다.

u0017496@sys-87926:~$ for i in `cat matchlist.txt`
> do
> sed 's/root\/miniconda3/usr\/local\/miniconda3/g' $i > ${i}.new
> cp ${i}.new $i
> done

임시작업용 파일들인 *.new는 나중에 삭제하셔도 됩니다.

보시면 아래와 같이 /root/miniconda가 /usr/local/miniconda3로 바뀐 것을 보실 수 있습니다.

u0017496@sys-87926:~$ grep usr\/local /usr/local/miniconda3/bin/pyvenv-3.6
#!/usr/local/miniconda3/bin/python

이제 conda 명령이 제대로 수행되는지 테스트 해보겠습니다. 보시다시피 잘 됩니다.

u0017496@sys-87926:~$ which conda
/usr/local/miniconda3/bin/conda

u0017496@sys-87926:~$ conda list
# packages in environment at /usr/local/miniconda3:
#
asn1crypto 0.22.0 py36_0
bazel 0.4.5 0
boto 2.47.0 <pip>
bz2file 0.98 <pip>
cffi 1.10.0 py36_0
conda 4.3.18 py36_0
conda-env 2.6.0 0
cryptography 1.8.1 py36_0
cudatoolkit 8.0 0
cudnn 6.0.21 0
cycler 0.10.0 py36_0
freetype 2.5.5 2
gensim 2.0.0 <pip>
idna 2.5 py36_0
Keras 2.0.4 <pip>
konlpy 0.4.4 <pip>
libffi 3.2.1 1
libpng 1.6.27 0
libprotobuf 3.2.0 0
matplotlib 2.0.2 np112py36_0
numpy 1.13.0 <pip>
numpy 1.12.1 py36_0
openblas 0.2.19 0
openssl 1.0.2k 2
packaging 16.8 py36_0
pip 9.0.1 py36_1
protobuf 3.2.0 py36_0
pyasn1 0.2.3 py36_0
pycosat 0.6.2 py36_0
pycparser 2.17 py36_0
pyopenssl 17.0.0 py36_0
pyparsing 2.1.4 py36_0
python 3.6.1 2
python-dateutil 2.6.0 py36_0
pytz 2017.2 py36_0
PyYAML 3.12 <pip>
requests 2.14.2 py36_0
ruamel_yaml 0.11.14 py36_1
scikit-learn 0.18.1 np112py36_1
scipy 0.19.0 <pip>
scipy 0.19.0 np112py36_0
setuptools 27.2.0 py36_0
six 1.10.0 py36_0
smart-open 1.5.3 <pip>
sqlite 3.13.0 0
tensorflow 1.1.0 np112py36_0
tensorflow-gpu 1.1.0 np112py36_0
Theano 0.9.0 <pip>
werkzeug 0.12.2 py36_0
wheel 0.29.0 py36_0
xz 5.2.2 1
yaml 0.1.6 0
zlib 1.2.8 3

새로 package를 install하는 것도 잘 됩니다.

u0017496@sys-87926:~$ conda install flask
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /usr/local/miniconda3:

The following NEW packages will be INSTALLED:

click: 6.7-py36_0
flask: 0.12.2-py36_0
itsdangerous: 0.24-py36_0
jinja2: 2.9.6-py36_0
markupsafe: 0.23-py36_2

Proceed ([y]/n)? y

click-6.7-py36 100% |######################################| Time: 0:00:00 7.96 MB/s
itsdangerous-0 100% |######################################| Time: 0:00:00 13.50 MB/s
markupsafe-0.2 100% |######################################| Time: 0:00:00 19.45 MB/s
jinja2-2.9.6-p 100% |######################################| Time: 0:00:00 8.85 MB/s
flask-0.12.2-p 100% |######################################| Time: 0:00:00 8.55 MB/s

뿐만 아니라, 새로 인터넷에서 download 받아 설치하는 것도 잘 됩니다. (물론 이건 인터넷이 access되는 환경에서만 가능한 테스트입니다.)

u0017496@sys-87926:~$ pip install Jpype1
Collecting Jpype1
Downloading JPype1-0.6.2.tar.gz (147kB)
100% |████████████████████████████████| 153kB 2.8MB/s
Building wheels for collected packages: Jpype1
Running setup.py bdist_wheel for Jpype1 ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/8e/f3/e6/a1250b8e8d2bd105f4dd21b1dc801dbcf5d815592443bfe741
Successfully built Jpype1
Installing collected packages: Jpype1
Successfully installed Jpype1-0.6.2

KoNLPy를 python에서 import 해보면 잘 되는 것을 보실 수 있습니다. (이걸 테스트하시려면 위와 같이 Jpype1가 설치되어 있어야 합니다.)

u0017496@sys-87926:~$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el

u0017496@sys-87926:~$ python
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 15:31:35)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from konlpy.tag import *
>>> Kkma()
<konlpy.tag._kkma.Kkma object at 0x3fff97589b38>
>>>

이제 다른 user를 만들어서 이 /usr/local/miniconda를 쓸 수 있는지 보겠습니다.

root@sys-87926:/home/u0017496# adduser newuser

새로 만들어진 newuser로 login 한 뒤, PATH를 맞춰줍니다.

newuser@sys-87926:~$ vi .bashrc
export PATH=/usr/local/miniconda3/bin:$PATH
export PYTHONPATH=/usr/local/miniconda3/lib/python3.6/site-packages
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el

newuser@sys-87926:~$ . ./.bashrc

보시다시피 다 잘 됩니다.

newuser@sys-87926:~$ conda list
# packages in environment at /usr/local/miniconda3:
#
asn1crypto 0.22.0 py36_0
bazel 0.4.5 0
boto 2.47.0 <pip>
bz2file 0.98 <pip>
...
yaml 0.1.6 0
zlib 1.2.8 3

newuser@sys-87926:~$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el
newuser@sys-87926:~$ python
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 15:31:35)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from konlpy.tag import *
>>> Kkma()
<konlpy.tag._kkma.Kkma object at 0x3fffa4e49780>
>>>

/usr/local/miniconda3의 permission이 아래처럼 755로 되어 있으므로, 아래처럼 newuser라는 새로운 사용자는 conda uninstall이나 install 같은 것은 할 수 없습니다. 만약 다른 사용자들도 자유롭게 install/uninstall을 하기를 원하신다면 그에 맞춰 permission을 조정해주셔야 합니다. 그러나 되도록 그러지 않는 편이 낫겠지요.

u0017496@sys-87926:~$ ls -la /usr/local/miniconda3
total 76
drwxr-xr-x 11 u0017496 u0017496 4096 Jun 23 02:55 .
drwxr-xr-x 13 root root 4096 Jun 22 22:35 ..
drwxr-xr-x 2 u0017496 u0017496 4096 Jun 23 02:20 bin
drwxr-xr-x 2 u0017496 u0017496 12288 Jun 23 02:55 conda-meta
drwxr-xr-x 3 u0017496 u0017496 4096 Jun 23 02:21 envs
drwxr-xr-x 3 u0017496 u0017496 4096 Jun 22 22:43 etc
drwxr-xr-x 8 u0017496 u0017496 4096 Jun 22 22:40 include
drwxr-xr-x 6 u0017496 u0017496 4096 Jun 23 02:12 lib
-rw-r--r-- 1 u0017496 u0017496 3699 Jun 22 22:35 LICENSE.txt
drwxr-xr-x 238 u0017496 u0017496 24576 Jun 23 02:23 pkgs
drwxr-xr-x 4 u0017496 u0017496 4096 Jun 22 22:43 share
drwxr-xr-x 3 u0017496 u0017496 4096 Jun 22 22:43 ssl

newuser@sys-87926:~$ conda uninstall matplotlib

CondaIOError: Missing write permissions in: /usr/local/miniconda3
#
# You don't appear to have the necessary permissions to remove packages
# into the install area '/usr/local/miniconda3'.
# However you can clone this environment into your home directory and
# then make changes to it.
# This may be done using the command:
#
# $ conda create -n my_root --clone="/usr/local/miniconda3"

** 이 작업 및 test 결과는 아래의 Continuum github에도 comment로 올렸습니다.

https://github.com/ContinuumIO/anaconda-issues/issues/270

2017년 6월 15일 목요일

Minsky 서버의 Ubuntu 16.04 ppc64le에 JDK 7 설치하기

ppc64le 아키텍처인 Minsky 서버는 Ubuntu 16.04 (또는 Redhat 7.3) 이후부터 지원됩니다. 그러다보니 Ubuntu 16.04에 포함된 openJDK8을 주로 사용하는데, 간혹 JDK7을 필요로 하는 경우도 있을 수 있습니다. 이때는 아래와 같이 설치하여 사용하실 수 있는데, 크게 2가지 방법, 즉 IBM java SDK 7을 설치하거나 또는 openJDK7을 설치하는 방법이 있습니다. 둘 중 원하시는 것을 선택하시면 됩니다.

1) IBM SDK 7 설치

아래 URL에서 ibm-java-ppc64le-sdk-7.1-4.5.bin 을 다운로드 받습니다. Minsky는 POWER8의 big endian이 아닌, x86과 같은 little endian을 지원하므로 'Linux on Power Systems 64-bit'가 아닌 'Linux on Power Systems 64-bit LE'을 택해야 합니다.

https://developer.ibm.com/javasdk/downloads/sdk7r1/

Linux on Power Systems 64-bit LE
Installable package (InstallAnywhere as root)
(File name: ibm-java-ppc64le-sdk-7.1-4.5.bin, Size: 128MB)

u0017496@sys-87548:~$ chmod a+x ibm-java-ppc64le-sdk-7.1-4.5.bin

u0017496@sys-87548:~$ sudo ./ibm-java-ppc64le-sdk-7.1-4.5.bin
...
===============================================================================
Installation Complete
---------------------

Congratulations. IBM 64-bit SDK for Linux, Java Technology Edition, Version 7.1
has been successfully installed to:

/opt/ibm/java-ppc64le-71

PRESS <ENTER> TO EXIT THE INSTALLER:

설치 directory가 아래와 같으므로, user profile의 PATH의 맨 앞에 다음과 같이 설정해서 쓰시면 됩니다.

u0017496@sys-87548:~$ export PATH=/opt/ibm/java-ppc64le-71/bin:$PATH

u0017496@sys-87548:~$ which java
/opt/ibm/java-ppc64le-71/bin/java

u0017496@sys-87548:~$ java -version
java version "1.7.0"
Java(TM) SE Runtime Environment (build pxl6470_27sr4fp5-20170421_02(SR4 FP5))
IBM J9 VM (build 2.7, JRE 1.7.0 Linux ppc64le-64 Compressed References 20170406_343211 (JIT enabled, AOT enabled)
J9VM - R27_Java727_SR4_20170406_2107_B343211
JIT - tr.r13.java_20170406_343211
GC - R27_Java727_SR4_20170406_2107_B343211_CMPRSS
J9CL - 20170406_343211)
JCL - 20170420_01 based on Oracle jdk7u141-b11

2) openJDK 7 설치

Ubuntu 16에는 JDK7이 지원되지 않습니다만, 아래와 같이 PPA repository에서 제공되기는 합니다. 다만, 이 설치 과정 중에 Oracle installer를 download 받아서 그걸 이용하는 과정이 있는데, ppc64le는 Oracle과 친하지 않아 그 과정은 실패합니다. 그러나 그와는 무관하게 JDK7은 설치되고 사용할 수 있습니다.

u0017496@sys-87548:~$ sudo add-apt-repository ppa:openjdk-r/ppa

More info: https://launchpad.net/~openjdk-r/+archive/ubuntu/ppa
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmpgyngeisu/secring.gpg' created
gpg: keyring `/tmp/tmpgyngeisu/pubring.gpg' created
gpg: requesting key 86F44E2A from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpgyngeisu/trustdb.gpg: trustdb created
gpg: key 86F44E2A: public key "Launchpad OpenJDK builds (all archs)" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK
u0017496@sys-87548:~$ sudo apt-get update

u0017496@sys-87548:~$ apt-cache pkgnames | grep openjdk-7
openjdk-7-jre-zero
openjdk-7-jre-lib
openjdk-7-source
oracle-jdk7-installer
openjdk-7-demo
openjdk-7-jre-headless
openjdk-7-dbg
openjdk-7-doc
openjdk-7-jdk
openjdk-7-jre

u0017496@sys-87548:~$ sudo apt-get install openjdk-7-jdk
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
at-spi2-core dbus-x11 gconf-service gconf-service-backend gconf2 gconf2-common libatk-bridge2.0-0
libatk-wrapper-java libatk-wrapper-java-jni libatspi2.0-0 libavahi-glib1 libbonobo2-0
libbonobo2-common libcanberra0 libgconf-2-4 libgnome-2-0 libgnome2-common libgnomevfs2-0
libgnomevfs2-common liborbit-2-0 libsctp1 libtdb1 libvorbisfile3 libxaw7 libxcb-shape0 libxft2
libxmu6 libxpm4 libxv1 libxxf86dga1 openjdk-7-jre openjdk-7-jre-headless sound-theme-freedesktop
x11-utils
Suggested packages:
gconf-defaults-service libbonobo2-bin libcanberra-gtk0 libcanberra-pulse desktop-base
libgnomevfs2-bin libgnomevfs2-extra gamin | fam gnome-mime-data lksctp-tools openjdk-7-demo
openjdk-7-source visualvm libnss-mdns sun-java6-fonts fonts-ipafont-gothic fonts-ipafont-mincho
ttf-wqy-microhei | ttf-wqy-zenhei fonts-indic mesa-utils
The following NEW packages will be installed:
at-spi2-core dbus-x11 gconf-service gconf-service-backend gconf2 gconf2-common libatk-bridge2.0-0
libatk-wrapper-java libatk-wrapper-java-jni libatspi2.0-0 libavahi-glib1 libbonobo2-0
libbonobo2-common libcanberra0 libgconf-2-4 libgnome-2-0 libgnome2-common libgnomevfs2-0
libgnomevfs2-common liborbit-2-0 libsctp1 libtdb1 libvorbisfile3 libxaw7 libxcb-shape0 libxft2
libxmu6 libxpm4 libxv1 libxxf86dga1 openjdk-7-jdk openjdk-7-jre openjdk-7-jre-headless
sound-theme-freedesktop x11-utils
0 upgraded, 35 newly installed, 0 to remove and 39 not upgraded.
1 not fully installed or removed.
Need to get 54.6 MB of archives.
After this operation, 91.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial/main ppc64el libbonobo2-common all 2.32.1-3 [34.7 kB]
Get:2 http://ppa.launchpad.net/openjdk-r/ppa/ubuntu xenial/main ppc64el openjdk-7-jre-headless ppc64el 7u95-2.6.4-3 [38.6 MB]
...
Processing triggers for hicolor-icon-theme (0.15-0ubuntu1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Setting up oracle-java7-installer (7u80+7u60arm-0~webupd8~1) ...
Please report to author unsupported platform 'ppc64le'.
Proceeding without web browser plugin support
Please report to author unsupported platform 'ppc64le'.
Proceeding without web browser plugin support
...
AuthParam=1497491097_5726d7e0ebd89a8675567ef8793c3443 [following]
--2017-06-14 21:42:57-- http://download.oracle.com/otn-pub/java/jdk/7u60-b19/jdk-7u60-linux-.tar.gz?AuthParam=1497491097_5726d7e0ebd89a8675567ef8793c3443
Connecting to download.oracle.com (download.oracle.com)|65.158.47.80|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-06-14 21:42:57 ERROR 404: Not Found.

download failed
Oracle JDK 7 is NOT installed.
...
Setting up gconf2 (3.2.6-3ubuntu6) ...
Setting up libgnomevfs2-common (1:2.24.4-6.1ubuntu1) ...
Setting up libgnomevfs2-0:ppc64el (1:2.24.4-6.1ubuntu1) ...
Setting up libgnome2-common (2.32.1-5ubuntu1) ...
Setting up libgnome-2-0:ppc64el (2.32.1-5ubuntu1) ...
Processing triggers for libc-bin (2.23-0ubuntu7) ...
Errors were encountered while processing:
oracle-java7-installer
E: Sub-process /usr/bin/dpkg returned an error code (1)

그러나 이는 oracle-java7-installer의 error일 뿐, JDK 7 자체는 설치가 된 것이며 사용 가능합니다.

u0017496@sys-87548:~$ dpkg -l | grep jdk-7
ii openjdk-7-demo 7u95-2.6.4-3 ppc64el Java runtime based on OpenJDK (demos and examples)
ii openjdk-7-jdk:ppc64el 7u95-2.6.4-3 ppc64el OpenJDK Development Kit (JDK)
ii openjdk-7-jre:ppc64el 7u95-2.6.4-3 ppc64el OpenJDK Java runtime, using Hotspot JIT
ii openjdk-7-jre-headless:ppc64el 7u95-2.6.4-3 ppc64el OpenJDK Java runtime, using Hotspot JIT (headless)

다음과 같이 JDK 명령어들도 잘 설치된 것을 보실 수 있습니다.

u0017496@sys-87548:~$ dpkg -L openjdk-7-jdk | grep bin
/usr/lib/jvm/java-7-openjdk-ppc64el/bin
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/serialver
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/native2ascii
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jps
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/extcheck
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/javadoc
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jcmd
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jstatd
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jstack
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jsadebugd
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jarsigner
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/javah
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jmap
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/xjc
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jdb
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/javac
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jstat
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/idlj
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/wsimport
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jinfo
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/javap
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/appletviewer
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/rmic
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/schemagen
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jhat
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/wsgen
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jrunscript
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jar
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jconsole

이 JDK7을 이용하려면 위의 IBM java SDK처럼 PATH 환경 변수를 조정하여 사용할 수도 있고, 이건 openjdk이므로 'update-alternatives --config java' 명령으로 아래처럼 link를 바꿔줄 수도 있습니다.

u0017496@sys-87548:~$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

u0017496@sys-87548:~$ sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).

Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-8-openjdk-ppc64el/jre/bin/java 1081 auto mode
1 /usr/lib/jvm/java-7-openjdk-ppc64el/jre/bin/java 1071 manual mode
2 /usr/lib/jvm/java-8-openjdk-ppc64el/jre/bin/java 1081 manual mode

Press <enter> to keep the current choice[*], or type selection number: 1
update-alternatives: using /usr/lib/jvm/java-7-openjdk-ppc64el/jre/bin/java to provide /usr/bin/java (java) in manual mode

위에서처럼 1번을 택해서 JDK8 대신 JDK7을 택한 뒤, java 명령을 내리면 아래처럼 버전 7이 link되어 있음을 확인할 수 있습니다.

u0017496@sys-87548:~$ java -version
java version "1.7.0_95"
OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-3)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)

2017년 6월 14일 수요일

Linux kernel update 이후 SpectrumScale mount가 안되는 문제의 해결 - mmbuildgpl

보안 패치 등의 적용 등으로 인해 원했든 원하지 않았든 linux kernel이 update되는 수가 있습니다. 그 뒤에 SpectrumScale (GPFS)가 mount가 되지 않는 경우가 있습니다. 좀더 정확하게 말하자면 GPFS daemon이 살아나지 못하는 경우지요.

결론부터 이야기하면 다음과 같이 해당 서버에서 root 권한으로 1줄의 명령을 수행하고 GPFS daemon을 restart 하면 해결됩니다.

# /usr/lpp/mmfs/bin/mmbuildgpl

설명은 이렇습니다.

- GPFS는 여러 node에 걸쳐 동시에 mount되고 read/write할 수 있는 병렬파일시스템이기 때문에, 일반 파일시스템과는 달리 linux kernel에 밀접히 연관되어 있습니다.
- UNIX OS인 IBM AIX에서는 위와 같은 절차가 필요없으나, Linux OS에서는 open source로 되어 있는 linux kernel이 바뀔 때마다 그와 연동하기 위해 GPFS의 일부 module(portability layer)도 새로 compile되어야 합니다.
- 원래 이 작업은 몇가지 과정을 거치는 좀더 복잡한 작업이었으나, IBM에서 위와 같이 mmbuildgpl 명령 하나만으로 그 과정들이 모두 수행되도록 shell script를 작성한 것입니다.
- 위 작업을 수행하신 뒤, 해당 node에서 mmshutdown 이후 mmstartup으로 GPFS daemon만 restart 해주시면 GPFS가 다시 잘 mount 됩니다.

이 상황의 재현 및 해결의 test를 위해 먼저, 1대의 POWER8 Ubuntu 서버(sys-87549)를 GPFS server로 삼고, 나머지 1대(sys-87576)를 network을 통해 그 GPFS를 mount하는 GPFS client로 삼겠습니다.

먼저, SpectrumScale (GPFS)을 설치합니다. tgz 파일을 풀면 다음과 같이 self-extracting 파일이 나옵니다.

root@sys-87549:/home/u0017496# tar -zxvf SpectrumScaleStd421Pwr8le.tgz
Spectrum_Scale_Standard-4.2.1.0-ppc64LE-Linux-install
README

이걸 수행해주면 Redhat용 rpm과 Ubuntu용 deb 파일들이 /usr/lpp/mmfs/4.2.1.0에 복사됩니다.

root@sys-87549:/home/u0017496# ./Spectrum_Scale_Standard-4.2.1.0-ppc64LE-Linux-install

Extracting License Acceptance Process Tool to /usr/lpp/mmfs/4.2.1.0 ...
tail -n +563 ./Spectrum_Scale_Standard-4.2.1.0-ppc64LE-Linux-install | tar -C /usr/lpp/mmfs/4.2.1.0 -xvz --exclude=installer --exclude=*_rpms --exclude=*rpm --exclude=*tgz --exclude=*deb 1> /dev/null

Installing JRE ...
tail -n +563 ./Spectrum_Scale_Standard-4.2.1.0-ppc64LE-Linux-install | tar -C /usr/lpp/mmfs/4.2.1.0 --wildcards -xvz ibm-java*tgz 1> /dev/null
tar -C /usr/lpp/mmfs/4.2.1.0/ -xzf /usr/lpp/mmfs/4.2.1.0/ibm-java*tgz
Defaulting to --text-only mode.
...
Product rpms successfully extracted to /usr/lpp/mmfs/4.2.1.0

root@sys-87549:/home/u0017496# cd /usr/lpp/mmfs/4.2.1.0/
root@sys-87549:/usr/lpp/mmfs/4.2.1.0# ls
gpfs.base_4.2.1-0_ppc64el.deb gpfs.gss.pmcollector-4.2.1-0.el7.ppc64le.rpm
gpfs.base-4.2.1-0.ppc64le.rpm gpfs.gss.pmcollector-4.2.1-0.sles12.ppc64le.rpm
gpfs.docs_4.2.1-0_all.deb gpfs.gss.pmsensors-4.2.1-0.el7.ppc64le.rpm
gpfs.docs-4.2.1-0.noarch.rpm gpfs.gss.pmsensors-4.2.1-0.sles12.ppc64le.rpm
gpfs.ext_4.2.1-0_ppc64el.deb gpfs.gui-4.2.1-0.noarch.rpm
gpfs.ext-4.2.1-0.ppc64le.rpm gpfs.java-4.2.1-0.ppc64le.rpm
gpfs.gpl_4.2.1-0_all.deb gpfs.msg.en-us_4.2.1-0_all.deb
gpfs.gpl-4.2.1-0.noarch.rpm gpfs.msg.en_US-4.2.1-0.noarch.rpm
gpfs.gskit_8.0.50-57_ppc64el.deb license
gpfs.gskit-8.0.50-57.ppc64le.rpm manifest

먼저, GPFS 구성에는 아래 OS package들이 필요하므로 apt-get으로 설치합니다.

root@sys-87549:/usr/lpp/mmfs/4.2.1.0# apt-get install ksh gawk libaio1

그리고나서 아래 명령으로 deb 파일들을 설치합니다.

root@sys-87549:/usr/lpp/mmfs/4.2.1.0# dpkg -i gpfs.base*deb gpfs.gpl*deb gpfs.gskit*deb gpfs.msg*deb gpfs.ext*deb

위 작업을 sys-87576에서 반복합니다.

이제 GPFS cluster를 만듭니다. Primary server node는 여기서는 sys-87549 1대 뿐이고, client도 sys-87576 1대 뿐입니다.

root@sys-87549:/usr/lpp/mmfs/bin# ./mmcrcluster -N sys-87549:manager-quorum,sys-87576:client -p sys-87549 -r /usr/bin/ssh -R /usr/bin/scp -C test1
mmcrcluster: Performing preliminary node verification ...
mmcrcluster: Processing quorum and other critical nodes ...
...
mmcrcluster: Command successfully completed
mmcrcluster: Warning: Not all nodes have proper GPFS license designations.
Use the mmchlicense command to designate licenses as needed.
mmcrcluster: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

이제 아래와 같이 mmchlicense 명령으로 license accept를 합니다.

root@sys-87549:/usr/lpp/mmfs/bin# mmchlicense server --accept -N sys-87549

root@sys-87576:/usr/lpp/mmfs/bin# mmchlicense client --accept -N sys-87576

이제 다른 쪽 노드에서 mmlscluster를 수행해보면 이미 cluster로 구성된 것이 보입니다.

root@sys-87576:/usr/lpp/mmfs/bin# mmlscluster

GPFS cluster information
========================
GPFS cluster name: test1.dal-ebis.ihost.com
GPFS cluster id: 6484364745360086160
GPFS UID domain: test1.dal-ebis.ihost.com
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
Repository type: CCR

Node Daemon node name IP address Admin node name Designation
-----------------------------------------------------------------------------------------------
1 sys-87549.dal-ebis.ihost.com 172.29.160.222 sys-87549.dal-ebis.ihost.com quorum-manager
2 sys-87576.dal-ebis.ihost.com 172.29.160.12 sys-87576.dal-ebis.ihost.com

이제 서버에서 GPFS daemon을 start 합니다.

root@sys-87549:/usr/lpp/mmfs/bin# mmstartup -a
Wed Jun 14 06:57:29 EDT 2017: mmstartup: Starting GPFS ...
sys-87576.dal-ebis.ihost.com: /tmp/mmfs has been created successfully.

Daemon들이 제대로 올라왔는지 아래 명령으로 확인합니다. 보시다시피 down된 상태입니다. 즉, 뭔가 잘못된 것이지요.

root@sys-87549:/usr/lpp/mmfs/bin# mmgetstate -a

Node number Node name GPFS state
------------------------------------------
1 sys-87549 down
2 sys-87576 down

이는 위에서 언급한 GPFS의 portability layer를 build 해주지 않았기 때문입니다. 설치 후, 그리고 향후에도 kernel update 등을 한 이후에는 꼭 아래 명령을 내려 GPFS의 portability layer를 새로 compile 해줘야 합니다. 아래 mmbuildgpl은 shell script로서, 여러 단계의 compile 과정을 자동으로 해줍니다.

root@sys-87549:~# /usr/lpp/mmfs/bin/mmbuildgpl
--------------------------------------------------------
mmbuildgpl: Building GPL module begins at Wed Jun 14 08:57:27 EDT 2017.
--------------------------------------------------------
Verifying Kernel Header...
kernel version = 40400066 (4.4.0-66-generic, 4.4.0-66)
module include dir = /lib/modules/4.4.0-66-generic/build/include
module build dir = /lib/modules/4.4.0-66-generic/build
kernel source dir = /usr/src/linux-4.4.0-66-generic/include
Found valid kernel header file under /lib/modules/4.4.0-66-generic/build/include
Verifying Compiler...
make is present at /usr/bin/make
cpp is present at /usr/bin/cpp
gcc is present at /usr/bin/gcc
g++ is present at /usr/bin/g++
ld is present at /usr/bin/ld
make World ...
make InstallImages ...
--------------------------------------------------------
mmbuildgpl: Building GPL module completed successfully at Wed Jun 14 08:58:12 EDT 2017.
--------------------------------------------------------

다시 GPFS daemon을 start 합니다.

root@sys-87549:/home/u0017496# mmstartup -a
Wed Jun 14 09:02:18 EDT 2017: mmstartup: Starting GPFS ...

잘 올라온 것을 확인할 수 있습니다.

root@sys-87549:~# mmgetstate -a

Node number Node name GPFS state
------------------------------------------
1 sys-87549 active
2 sys-87576 active

이제 SpectrumScale server인 sys-87549의 내장 disk로 NSD를 만듭니다. 이 disk는 sys-87549에서만 access되는 내장 disk이지만, 일단 NSD로 만들어지면 네트워크를 통해 client node인 sys-87576에서도 access가 가능해집니다.

root@sys-87549:~# vi /tmp/diskdef.txt
%nsd:
device=/dev/sdb
servers=sys-87549
nsd=nsd1
usage=dataAndMetadata

root@sys-87549:~# mmcrnsd -F /tmp/diskdef.txt
mmcrnsd: Processing disk sdb
mmcrnsd: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

SpectrumScale client 노드인 sys-87576에서도 이것이 보이는지 확인합니다.

root@sys-87576:~# mmlsnsd

File system Disk name NSD servers
---------------------------------------------------------------------------
(free disk) nsd1 sys-87549.dal-ebis.ihost.com

이제 GPFS filesystem을 만듭니다.

root@sys-87549:~# mmcrfs GPFS1 -F /tmp/diskdef.txt -A yes -T /gpfs1 --version=4.2.1.0

The following disks of GPFS1 will be formatted on node sys-87549:
nsd1: size 4096 MB
Formatting file system ...
Disks up to size 96 GB can be added to storage pool system.
Creating Inode File
48 % complete on Wed Jun 14 09:09:18 2017
97 % complete on Wed Jun 14 09:09:23 2017
100 % complete on Wed Jun 14 09:09:23 2017
Creating Allocation Maps
Creating Log Files
78 % complete on Wed Jun 14 09:09:29 2017
100 % complete on Wed Jun 14 09:09:29 2017
Clearing Inode Allocation Map
Clearing Block Allocation Map
Formatting Allocation Map for storage pool system
Completed creation of file system /dev/GPFS1.
mmcrfs: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

root@sys-87549:~# mmlsfs all

File system attributes for /dev/GPFS1:
======================================
flag value description
------------------- ------------------------ -----------------------------------
-f 8192 Minimum fragment size in bytes
-i 4096 Inode size in bytes
-I 32768 Indirect block size in bytes
-m 1 Default number of metadata replicas
-M 2 Maximum number of metadata replicas
-r 1 Default number of data replicas
-R 2 Maximum number of data replicas
-j cluster Block allocation type
-D nfs4 File locking semantics in effect
-k all ACL semantics in effect
-n 32 Estimated number of nodes that will mount file system
-B 262144 Block size
-Q none Quotas accounting enabled
none Quotas enforced
none Default quotas enabled
--perfileset-quota No Per-fileset quota enforcement
--filesetdf No Fileset df enabled?
-V 15.01 (4.2.0.0) File system version
--create-time Wed Jun 14 09:09:15 2017 File system creation time
-z No Is DMAPI enabled?
-L 4194304 Logfile size
-E Yes Exact mtime mount option
-S No Suppress atime mount option
-K whenpossible Strict replica allocation option
--fastea Yes Fast external attributes enabled?
--encryption No Encryption enabled?
--inode-limit 65792 Maximum number of inodes
--log-replicas 0 Number of log replicas
--is4KAligned Yes is4KAligned?
--rapid-repair Yes rapidRepair enabled?
--write-cache-threshold 0 HAWC Threshold (max 65536)
--subblocks-per-full-block 32 Number of subblocks per full block
-P system Disk storage pools in file system
-d nsd1 Disks in file system
-A yes Automatic mount option
-o none Additional mount options
-T /gpfs1 Default mount point
--mount-priority 0 Mount priority

이제 mount 합니다. -a 옵션을 쓰면 클러스터 내 모든 노드에서 다 mount 됩니다.

root@sys-87549:~# mmmount all -a
Wed Jun 14 09:11:15 EDT 2017: mmmount: Mounting file systems ...

모든 노드에서 다 mount 되었는지 확인합니다.

root@sys-87549:~# mmlsmount all
File system GPFS1 is mounted on 2 nodes.

Client 노드에 들어가서 mount 되었는지 확인합니다.

root@sys-87576:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 2.0G 0 2.0G 0% /dev
tmpfs 408M 18M 390M 5% /run
/dev/sda2 35G 18G 16G 53% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
GPFS1 4.0G 3.9G 151M 97% /gpfs1

이제 비로소 kernel update 및 GPFS mount 테스트 준비가 끝났습니다. 이제 client인 sys-87576에서 kernel을 update 해보겠습니다. 먼저, 현재 kernel 버전을 확인합니다.

root@sys-87576:~# uname -r
4.4.0-66-generic

이제 kernel update를 합니다.

root@sys-87576:~# apt-get upgrade

완료되면 reboot 합니다.

root@sys-87576:~# shutdown -r now

Reboot 된 이후 kernel 버전 확인합니다.

root@sys-87576:/home/u0017496# uname -r
4.4.0-79-generic

이제 GPFS를 mount하기 위해 GPFS daemon을 살립니다. (여기서는 client 노드에서만 살리면 되므로 -a 옵션은 빼고 수행합니다.)

root@sys-87576:/home/u0017496# mmstartup
Wed Jun 14 09:42:08 EDT 2017: mmstartup: Starting GPFS ...
mmremote: startSubsys: The /lib/modules/4.4.0-79-generic/extra/mmfslinux.ko kernel extension does not exist. Use mmbuildgpl command to create the needed kernel extension for your kernel or copy the binaries from another node with the identical environment.
mmremote: startSubsys: Unable to verify kernel/module configuration.
mmstartup: Command failed. Examine previous error messages to determine cause.

보시다시피 error가 나고, 당연히 GPFS 파일시스템은 마운트 되지 못합니다. 이제 mmbuildgpl 명령으로 portability layer를 compile 해줍니다.

root@sys-87576:/home/u0017496# mmbuildgpl
--------------------------------------------------------
mmbuildgpl: Building GPL module begins at Wed Jun 14 09:42:21 EDT 2017.
--------------------------------------------------------
Verifying Kernel Header...
kernel version = 40400079 (4.4.0-79-generic, 4.4.0-79)
module include dir = /lib/modules/4.4.0-79-generic/build/include
module build dir = /lib/modules/4.4.0-79-generic/build
kernel source dir = /usr/src/linux-4.4.0-79-generic/include
Found valid kernel header file under /lib/modules/4.4.0-79-generic/build/include
Verifying Compiler...
make is present at /usr/bin/make
cpp is present at /usr/bin/cpp
gcc is present at /usr/bin/gcc
g++ is present at /usr/bin/g++
ld is present at /usr/bin/ld
make World ...
make InstallImages ...
--------------------------------------------------------
mmbuildgpl: Building GPL module completed successfully at Wed Jun 14 09:43:18 EDT 2017.
--------------------------------------------------------

성공적으로 끝났습니다. 이제 GPFS daemon을 살립니다.

root@sys-87576:/home/u0017496# mmstartup
Wed Jun 14 09:45:26 EDT 2017: mmstartup: Starting GPFS ...

잘 되었습니다. 이제 mount 합니다.

root@sys-87576:/home/u0017496# mmmount all
Wed Jun 14 09:45:37 EDT 2017: mmmount: Mounting file systems ...

잘 되었습니다. 이제 눈으로 mount 상태를 확인합니다.

root@sys-87576:/home/u0017496# df -h
Filesystem Size Used Avail Use% Mounted on
udev 2.0G 0 2.0G 0% /dev
tmpfs 408M 12M 396M 3% /run
/dev/sda2 35G 18G 16G 53% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
GPFS1 4.0G 456M 3.6G 12% /gpfs1

** 여기서는 SpectrumScale 4.2.1로 테스트한 것으로 되어 있습니다만, 제가 해보니 4.2.1에서는 mmbuildgpl에서 다음과 같은 error가 발생합니다.

error "struct inode has no member named i_wb_list"

이는 4.2.1에서의 bug인 모양입니다. 다시 4.2.3으로 설치하고 해보니 아무 문제없이 잘 됩니다.