HW 엔지니어를 위한 Deep Learning

2020년 3월 23일 월요일

IBM POWER9 (ppc64le) 아키텍처에서의 SimpleITK 설치

SimpleITK는 Insight Segmentation과 Registration Toolkit (ITK)를 감싼 일종의 layer 또는 wrapper 소프트웨어입니다. 이를 IBM POWER9 (ppc64le) 아키텍처의 python3 환경에서 사용하시려면 그냥 pip install로 설치하시면 됩니다.

아래 내용을 통해 build 한 RHEL 7.6 Alt (ppc64le) 상에서의 python 3.6.9를 위한 SimpleITK 1.2.0의 wheel file을 아래 Google drive에 올려 놓았으니 그걸 download 받아서 다음과 같은 명령으로 설치하셔도 됩니다.

$ pip install ./SimpleITK-1.2.0-cp36-cp36m-linux_ppc64le.whl

구글 drive에서 download 받으려면 여기를 click

직접 pip 명령으로 SimpleITK을 설치하기 위해서는 아래와 같은 수순을 밟으면 됩니다. 제가 해보니 생각보다는 build하는데 CPU 사용량 및 사용 시간이 꽤 깁니다.

먼저 gcc 및 make 등과 같은 Redhat OS의 기본 개발 tool을 설치합니다.

(base) [u0017496@vm ~]$ sudo yum groupinstall -y "Development Tools"

SimpleITK를 build하기 위해서는 scikit-build와 cmake (version 3.x)가 필요하므로 그것들도 설치합니다.

(base) [u0017496@vm ~]$ pip install scikit-build

(base) [u0017496@vm ~]$ conda install cmake

(base) [u0017496@vm ~]$ which cmake
~/anaconda3/bin/cmake

(base) [u0017496@vm ~]$ cmake --version
cmake version 3.14.0

그 다음은 그냥 pip install 명령을 사용하시면 됩니다. 그러면 internet repository에서 source를 가져와 build합니다. 저는 가상 CPU 환경에서 수행했는데, 한 30분은 걸린 것 같습니다.

(base) [u0017496@vm ~]$ pip install SimpleITK
Collecting SimpleITK
Downloading https://files.pythonhosted.org/packages/11/f5/dfc5fe1ee82baa0bf35579ab49f0b0d318ae528a7557552579a587f9d7a3/SimpleITK-1.2.0.tar.gz (2.0MB)
|████████████████████████████████| 2.0MB 9.5MB/s
Building wheels for collected packages: SimpleITK
...
Created wheel for SimpleITK: filename=SimpleITK-1.2.0-cp36-cp36m-linux_ppc64le.whl size=42515246 sha256=063c44e98540f8f01ce965647b198294ec81782053b9eef960cc11f6229041eb
Stored in directory: /home/cecuser/.cache/pip/wheels/b7/4e/7a/b7ac870691673ebd2688e1492d2ffac7b2380b6e607625baeb
Successfully built SimpleITK
Installing collected packages: SimpleITK
Successfully installed SimpleITK-1.2.0

Build되자마자 자동으로 설치까지 되며, 이떄 build된 wheel file은 아래 위치에 존재합니다. 그 크기는 45MB 정도 됩니다.

(wmlce_env3) [cecuser@vm ~]$ ls -l /home/cecuser/.cache/pip/wheels/b7/4e/7a/b7ac870691673ebd2688e1492d2ffac7b2380b6e607625baeb
total 41520
-rw-rw-r-- 1 cecuser cecuser 42515246 Mar 23 03:00 SimpleITK-1.2.0-cp36-cp36m-linux_ppc64le.whl

(wmlce_env3) [cecuser@vm ~]$ pip list | grep -i simpleitk
SimpleITK 1.2.0

(wmlce_env3) [cecuser@vm ~]$ which python
~/anaconda3/envs/wmlce_env3/bin/python

다음과 같이 import해보면 잘 되는 것을 확인할 수 있습니다.

(wmlce_env3) [cecuser@vm ~]$ python
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:18:58)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import SimpleITK as sitk
>>>

2020년 2월 20일 목요일

POWER9 (RHEL 8.1, ppc64le)에서 MariaDB MaxScale을 source로부터 build하기

MaxScale은 MariaDB의 앞단에서 HA 및 query routing, CDC 등을 처리해주는 솔루션입니다. MariaDB는 물론, mariadb-server-galera도 Red Hat Software Collections (RHSCL)에 포함되어 있습니다만, 현재로서는 MaxScale은 포함되어 있지 않습니다.

하지만 MaxScale은 open source이므로 source로부터 쉽게 build할 수 있습니다.

[cecuser@p606-kvm1 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.1 (Ootpa)

먼저 일반적인 개발 환경을 위해 필요한 package들을 설치합니다.

[cecuser@p606-kvm1 ~]$ sudo yum groupinstall -y "Development Tools"

다음으로는 아래 OS package들을 설치하고, Rambbit-MQ와 Jansson 등의 open source SW를 source로부터 build합니다. 원래 이 과정들은 MaxScale source code package 중에서 포함된 install_build_deps.sh를 수행하면 자동으로 되는 것입니다만, ppc64le에서는 일부 수행 오류가 나는 것이 있어서 다음과 같이 수동으로 수행하면 됩니다.

[cecuser@p606-kvm1 ~]$ sudo yum install -y libtool openssl-devel libaio libaio-devel libedit systemtap-sdt-devel rpm-sign wget gnupg pcre-devel flex rpmdevtools git wget tcl tcl-devel openssl libuuid-devel xz-devel sqlite sqlite-devel pkgconfig lua lua-libs rpm-build createrepo yum-utils gnutls-devel libgcrypt-devel pam-devel libcurl-devel nodejs-devel

[cecuser@p606-kvm1 ~]$ git clone https://github.com/alanxz/rabbitmq-c.git

[cecuser@p606-kvm1 ~]$ cd rabbitmq-c

[cecuser@p606-kvm1 rabbitmq-c]$ git checkout v0.7.1

[cecuser@p606-kvm1 rabbitmq-c]$ sudo make install

[cecuser@p606-kvm1 rabbitmq-c]$ cd ..

[cecuser@p606-kvm1 ~]$ git clone https://github.com/akheron/jansson.git

[cecuser@p606-kvm1 ~]$ cd jansson

[cecuser@p606-kvm1 jansson]$ git checkout v2.9

[cecuser@p606-kvm1 jansson]$ mkdir build && cd build

[cecuser@p606-kvm1 build]$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_C_FLAGS=-fPIC -DJANSSON_INSTALL_LIB_DIR=/usr/lib64

[cecuser@p606-kvm1 build]$ make

[cecuser@p606-kvm1 build]$ sudo make install

[cecuser@p606-kvm1 build]$ cd ../..

[cecuser@p606-kvm1 ~]$ wget -q -r -l1 -nH --cut-dirs=2 --no-parent -A.tar.gz --no-directories https://downloads.apache.org/avro/stable/c/

[cecuser@p606-kvm1 ~]$ tar -zxf avro-c-1.9.2.tar.gz

[cecuser@p606-kvm1 ~]$ cd avro-c-1.9.2

[cecuser@p606-kvm1 avro-c-1.9.2]$ mkdir build && cd build

[cecuser@p606-kvm1 build]$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_C_FLAGS=-fPIC -DCMAKE_CXX_FLAGS=-fPIC

[cecuser@p606-kvm1 build]$ make && sudo make install

[cecuser@p606-kvm1 build]$ cd ../..

이제 MaxScale의 source를 download 받습니다.

[cecuser@p606-kvm1 ~]$ git clone https://github.com/mariadb-corporation/MaxScale

[cecuser@p606-kvm1 ~]$ cd MaxScale/

[cecuser@p606-kvm1 MaxScale]$ mkdir build && cd build

원래 manual에는 아래와 같이 install_build_deps.sh를 수행하라고 되어 있지만, 하지 마십시요. 위에서 이미 수동으로 다 처리했으며, 이걸 수행하면 x86용 binary를 download 받아서 멀쩡한 nodejs 관련 파일을 망쳐놓는 오작동을 합니다.

~~[cecuser@p606-kvm1 build]$ ../BUILD/install_build_deps.sh~~ --> Don't Run !

cmake를 수행합니다.

[cecuser@p606-kvm1 build]$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr

그리고 다음 파일에서 ppc64만 있고 ppc64le가 없어서 발생하는 error가 있으므로, 아래와 같이 수정합니다.

[cecuser@p606-kvm1 build]$ vi ../query_classifier/qc_sqlite/sqlite-src-3110100/config.guess
...
ppc64:Linux:*:*)
echo powerpc64-unknown-linux-${LIBC}
exit ;;
ppc64le:Linux:*:*) # 추가
echo powerpc64le-unknown-linux-${LIBC} # 추가
exit ;; # 추가
...

다음은 make를 수행하면 됩니다.

[cecuser@p606-kvm1 build]$ make && sudo make install

테스트를 해보면 모두 정상 수행되는 것을 보실 수 있습니다.

[cecuser@p606-kvm1 build]$ make test
Running tests...
Test project /home/cecuser/MaxScale/build
Start 1: test_mxb_log
1/62 Test #1: test_mxb_log ........................ Passed 0.01 sec
Start 2: test_semaphore
2/62 Test #2: test_semaphore ...................... Passed 12.01 sec
Start 3: test_worker
...
61/62 Test #61: test_hintparser ..................... Passed 0.01 sec
Start 62: test_masking_rules
62/62 Test #62: test_masking_rules .................. Passed 0.01 sec

100% tests passed, 0 tests failed out of 62

Total Test time (real) = 108.24 sec

2020년 2월 18일 화요일

POWER9에서 sysbench를 source로부터 build하기

sysbench는 주로 DBMS의 성능 benchmark test를 할 때 사용되는 tool입니다. IBM POWER9 즉 ppc64le 아키텍처의 Redhat에서 이를 build하는 방법은 간단합니다.

먼저 필요한 OS package들을 설치합니다.

(base) [cecuser@p663-kvm1 sysbench]$ sudo yum -y install make automake libtool pkgconfig libaio-devel

MariaDB 그리고 PostgreSQL과 연계 테스트를 위해서는 아래와 같은 OS package들도 함께 설치합니다.

(base) [cecuser@p663-kvm1 sysbench]$ sudo yum -y install mariadb-devel openssl-devel postgresql-devel

이제 source code를 download 받습니다.

(base) [cecuser@p663-kvm1 ~]$ git clone https://github.com/akopytov/sysbench.git

(base) [cecuser@p663-kvm1 ~]$ cd sysbench

여기서 travis_ppc64le branch로 checkout 합니다. 이걸 하지 않으면 "error: ‘GG_State’ {aka ‘struct GG_State’} has no member named ‘J’ "라는 error를 겪게 되는데, 이에 대해서는 https://github.com/akopytov/sysbench/pull/234 를 참조하십시요.

(base) [cecuser@p663-kvm1 sysbench]$ git checkout travis_ppc64le

다음으로 autogen,sh을 수행하여 configure script를 생성합니다.

(base) [cecuser@p663-kvm1 sysbench]$ ./autogen.sh

만약 postgresql이나 mariadb로 sysbench 테스트를 하실 거라면 아래와 같이 '--with-pgsql --with-mysql' 옵션과 함께 configure를 돌리시면 됩니다. Default로는 mysql을 찾습니다.

(base) [cecuser@p663-kvm1 sysbench]$ ./configure --with-pgsql --with-mysql

만약 mysql이나 postgresql을 쓸 것이 아니라면 다음과 같이 하면 됩니다.

(base) [cecuser@p663-kvm1 sysbench]$ ./configure --without-mysql

그 다음으로는 make, sudo make install을 수행하면 됩니다.

(base) [cecuser@p663-kvm1 sysbench]$ make -j4

(base) [cecuser@p663-kvm1 sysbench]$ sudo make install

(base) [cecuser@p663-kvm1 sysbench]$ cd ..

sysbench는 아래 위치에 설치됩니다.

(base) [cecuser@p663-kvm1 ~]$ ls -l `which sysbench`
-rwxr-xr-x 1 root root 1384488 Feb 18 08:00 /usr/local/bin/sysbench

--without-mysql로 build된 sysbench 파일을 편의를 위해 아래의 Google drive에 올려놓았습니다.

https://drive.google.com/open?id=1tH9bbgQaipoAqxWFHAHVL3F4QPFSlcdG

혹시 몰라, 아래와 같이 위에서 "make -j4"까지 해놓은 sysbench directory 전체를 tgz로 묶어서 아래의 Google drive에 올려놓았습니다. 여기서는 --without-mysql로 build된 버전을 올렸습니다.

https://drive.google.com/open?id=1ircTWDzOKuZzglvz5cn-Wr2vEg0q0i3a

새로 build를 해야 하는 경우, 이 file을 아래와 같이 푸시고 sudo make install 만 수행하시면 됩니다.

(base) [cecuser@p628-kvm1 ~]$ tar -zxf sysbench_ppc64le.tgz

(base) [cecuser@p628-kvm1 ~]$ cd sysbench

(base) [cecuser@p628-kvm1 sysbench]$ sudo make install

또는 postgresql 등의 옵션을 줘서 다시 build해야 한다면 맨 첫줄의 autoconf.sh부터 새로 시작하시면 됩니다.

2020년 2월 13일 목요일

IBM AC922 서버에서 CUDA-enabled HPL 수행하기

HPL (High Performance Linpack) 테스트는 수퍼컴 클러스터의 성능 측정에 널리 쓰이는 오픈소스 프로그램입니다. GPU를 사용하여 HPL을 수행하기 위해서는 CUDA-enabled HPL이 필요한데, 그건 NVIDIA가 지적 재산권을 가진 프로그램이며 그건 오픈소스가 아닙니다. WWW 상을 뒤져보면 CUDA-enabled HPL의 source code를 NVIDIA가 공개하기는 하는데, 그건 매우 오래된 GPU architecture인 Fermi 아키텍처의 GPU에 대한 것이라서 최신 GPU의 성능 측정에는 적절하지 않습니다.

아래에서는 NVIDIA의 협조를 받아 CUDA-enabled HPL의 executible binary file을 가지고 있다는 전제 하에 IBM AC922 서버 (POWER9 * 2, V100 SXM2 32GB GPU * 4) 1대로 CUDA-enabled HPL을 수행하는 과정만 제시합니다.

그 결과는 역시 confidential 정보라 공개하지 못하는 점 양해 부탁드립니다.

이 테스트 수행을 위해서는 서버에 먼저 CUDA 10.1이 설치되어 있어야 합니다. 또한 IBM의 XL Fortran, Spectrum MPI, ESSL 등의 library 등이 필요합니다.

[cecuser@p1235-met1 HPC]$ ls
ESSL_FOR_LINUX_ON_POWER_V6.2.0.tar.gz
hpl_cuda10.1.4gpus.tgz
IBM_SMPI_10.2_IP_GR_LINUX_PPC64LE.tgz
ibm_smpi_lic_s-10.02-p9-ppc64le.rpm
lsf10.1_lnx310-lib217-ppc64le.tar.Z
XL_FORTRAN_FOR_LINUX_V16.1.1_PRO.gz

먼저 XL Fortran을 설치합니다.

[cecuser@p1235-met1 HPC]$ mkdir xlf

[cecuser@p1235-met1 HPC]$ cd xlf

[cecuser@p1235-met1 xlf]$ tar -zxvf ../XL_FORTRAN_FOR_LINUX_V16.1.1_PRO.gz

[cecuser@p1235-met1 xlf]$ ./install
...
Press Enter to continue viewing the license agreement, or, Enter "1" to accept the agreement,
"2" to decline it or "99" to go back to the previous screen, "3" Print.
1
INFORMATIONAL: Unexpected CUDA Toolkit version detected '10.1' (9.2, 10.0 are supported), defaulting to __CUDA_API_VERSION=10000. Re-configure with '-cudaVersion 9.2' to override.
Installation and configuration successful

이어서 Spectrum MPI를 설치합니다. 10.1이 아니라 10.2가 필요합니다.

[cecuser@p1235-met1 HPC]$ tar -zxf IBM_SMPI_10.2_IP_GR_LINUX_PPC64LE.tgz

[cecuser@p1235-met1 HPC]$ cd ibm_smpi-10.02.00.03-p9-ppc64le

[cecuser@p1235-met1 ibm_smpi-10.02.00.03-p9-ppc64le]$ sudo rpm -Uvh *.rpm ../ibm_smpi_lic_s-10.02-p9-ppc64le.rpm

[cecuser@p1235-met1 ibm_smpi-10.02.00.03-p9-ppc64le]$ su -

[root@p1235-met1 ~]# IBM_SPECTRUM_MPI_LICENSE_ACCEPT=yes /opt/ibm/spectrum_mpi/lap_se/bin/accept_spectrum_mpi_license.sh

[root@p1235-met1 ~]# exit

이어서 ESSL을 설치합니다. 이건 engineering용 library인데, GPU를 이용하도록 되어 있습니다.

[cecuser@p1235-met1 HPC]$ tar -zxvf ESSL_FOR_LINUX_ON_POWER_V6.2.0.tar.gz

[cecuser@p1235-met1 HPC]$ cd RHEL/RHEL7/

[cecuser@p1235-met1 RHEL7]$ su
Password:

[root@p1235-met1 RHEL7]# rpm -Uvh essl.license-6.2.0-0.ppc64le.rpm

[root@p1235-met1 RHEL7]# export IBM_ESSL_LICENSE_ACCEPT=yes

[root@p1235-met1 RHEL7]# /opt/ibmmath/essl/6.2/lap/accept_essl_license.sh

[root@p1235-met1 RHEL7]# rpm -Uvh essl.3264.rte-6.2.0-0.ppc64le.rpm essl.6464.rte-6.2.0-0.ppc64le.rpm essl.rte.common-6.2.0-0.ppc64le.rpm essl.man-6.2.0-0.ppc64le.rpm essl.3264.rtecuda-6.2.0-0.ppc64le.rpm essl.common-6.2.0-0.ppc64le.rpm essl.msg-6.2.0-0.ppc64le.rpm essl.rte-6.2.0-0.ppc64le.rpm

이제 CUDA-enabled HPL의 binary 및 script를 풀어냅니다.

[cecuser@p1235-met1 HPC]$ tar -zxvf hpl_cuda10.1.4gpus.tgz

[cecuser@p1235-met1 HPC]$ cd hpl

이 속에 들어있는 것은 간단합니다. Binary 실행 파일인 xhpl과 함께, 그 수행에 필요한 HPL.dat, 기타 mpirun을 위한 script 입니다.

먼저 HPL.dat의 내용입니다. Edit해야 하는 주요 내용은 아래 붉은 색으로 표시한 Ns (계산해야 하는 문제의 크기), NBs (한번에 어느 정도 크기로 문제를 풀 것인지 결정하는 block size), 그리고 mesh 구조를 결정하는 Ps와 Qs입니다.

간단히 말하면 Ns는 가급적 GPU들의 메모리를 꽤 가득 채울 정도로 크게 하고, NBs는 적절한 크기를 trial & error 방식으로 찾아야 합니다. 가령 제가 해보니 아래와 같은 크기의 Ns면 32GB memory의 GPU 4장을 가득 채웁니다. 또한 256이나 768에 비해 512로 NBs를 두는 것이 가장 성능이 잘 나오는 것 같습니다. Ps와 Qs는 서로 곱해서 GPU 갯수가 나오면 되는데, 가급적 서로 비슷하게, 그리고 가급적 Ps가 Qs보다 작게 설정하면 됩니다.

[cecuser@p1235-met1 hpl]$ cat HPL.dat
HPLinpack benchmark input file
Innovative Computing Laboratory, University of Tennessee
HPL.out output file name (if any)
6 device out (6=stdout,7=stderr,file)
1 # of problems sizes (N)
128000 Ns
1 # of NBs
512 NBs
0 PMAP process mapping (0=Row-,1=Column-major)
1 # of process grids (P x Q)
2 Ps
2 Qs
16.0 threshold
1 # of panel fact
2 PFACTs (0=left, 1=Crout, 2=Right)
1 # of recursive stopping criterium
4 NBMINs (>= 1)
1 # of panels in recursion
2 NDIVs
1 # of recursive panel fact.
0 RFACTs (0=left, 1=Crout, 2=Right)
1 # of broadcast
3 BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)
1 # of lookahead depth
0 DEPTHs (>=0)
1 SWAP (0=bin-exch,1=long,2=mix)
192 swapping threshold
1 L1 in (0=transposed,1=no-transposed) form
0 U in (0=transposed,1=no-transposed) form
0 Equilibration (0=no,1=yes)
8 memory alignment in double (> 0)

여기서는 1대로 수행하니까 hosts 파일은 사실 필요가 없습니다만 아래와 같은 format으로 설정하면 됩니다.

[cecuser@p1235-met1 hpl]$ cat hosts
localhost slots=4

아래는 mpirun을 수행하는 script입니다. 제가 쓴 환경처럼 infiniband가 없는 경우 "-pami_noib" 옵션을 써야 합니다.

[cecuser@p1235-met1 hpl]$ cat run_me_4_gpu_xlc_spectrum.sh
#!/bin/bash
export MPI_ROOT=/opt/ibm/spectrum_mpi
export MANPATH=$MPI_ROOT/share/man:$MANPATH
export PATH=/usr/local/cuda-10.1/bin:/opt/ibm/spectrum_mpi/bin:$PATH
export LD_LIBRARY_PATH=/opt/ibmmath/essl/6.2/lib64/:/opt/ibm/spectrum_mpi/lib:$LD_LIBRARY_PATH
sudo nvidia-smi -ac 877,1395
#echo always > /sys/kernel/mm/transparent_hugepage/enabled
TUNE="-x PAMI_IBV_DEVICE_NAME=mlx5_0:1 -x PAMI_IBV_DEVICE_NAME_1=mlx5_3:1 -x PAMI_ENABLE_STRIPING=0 -x PAMI_IBV_CQEDEPTH=4096 -x PAMI_IBV_ADAPTER_AFFINITY=1 -x PAMI_IBV_OPT_LATENCY=1 -x MLX5_SINGLE_THREADED=1 -x MLX5_CQE_SIZE=128 -x PAMI_IBV_ENABLE_DCT=1 -x PAMI_IBV_ENABLE_OOO_AR=1 -x PAMI_IBV_QP_SERVICE_LEVEL=8"
sudo ppc64_cpu --dscr=7
#mpirun -N 4 -npernode 4 --allow-run-as-root -x OMPI_MCA_common_pami_use_odp=0 -x PAMI_IBV_DEBUG_PRINT_DEVICES=1 -tag-output $TUNE --hostfile nodes -bind-to none ./run_linpack_6_gpu_xlc_spectrum_0726
mpirun -N 4 -npernode 4 --hostfile hosts -pami_noib -bind-to none ./run_linpack_4_gpu_xlc_spectrum.sh

그리고 아래가 실제 xhpl을 수행하는 script입니다. 위의 mpirun script를 수행하면 결국 아래의 script가 수행됩니다. IBM의 Spectrum MPI에서는 내부적으로 OMPI_COMM_WORLD_LOCAL_RANK, PMIX 등의 환경 변수를 자동 생성하여 GPU를 할당하는데 사용합니다. 아래 script를 보면 case 문을 이용하여 CUDA_VISIBLE_DEVICES 환경 변수를 이용하여 GPU 1개씩마다 xhpl을 하나씩 수행합니다.

[cecuser@p1235-met1 hpl]$ cat run_linpack_4_gpu_xlc_spectrum.sh
#!/bin/bash
#location of HPL
HPL_DIR=`pwd`
# Number of CPU cores
# Total CPU cores / Total GPUs (not counting hyperthreading)
#CPU_CORES_PER_RANK=16
CPU_CORES_PER_RANK=8
export MPI_ROOT=/opt/ibm/spectrum_mpi
export OMP_NUM_THREADS=$CPU_CORES_PER_RANK
export MAX_H2D_MS=10
export MAX_D2H_MS=10
export RANKS_PER_SOCKET=2
export RANKS_PER_NODE=4
export NUM_WORK_BUF=4
export SCHUNK_SIZE=128
export GRID_STRIPE=4
export FACT_GEMM=1
export FACT_GEMM_MIN=128
export SORT_RANKS=0
export PRINT_SCALE=1.0
export TEST_SYSTEM_PARAMS=1
sudo rm -rf /dev/shm/sh_*
export LIBC_FATAL_STDERR_=1
#export PAMI_ENABLE_STRIPING=0
export CUDA_CACHE_PATH=/tmp
export OMP_NUM_THREADS=$CPU_CORES_PER_RANK
export CUDA_DEVICE_MAX_CONNECTIONS=8
export CUDA_COPY_SPLIT_THRESHOLD_MB=1
export GPU_DGEMM_SPLIT=1.0
export TRSM_CUTOFF=1000000
#export TRSM_CUTOFF=99000
export TEST_SYSTEM_PARAMS=1
export MONITOR_GPU=1
export GPU_TEMP_WARNING=70
export GPU_CLOCK_WARNING=1310
export GPU_POWER_WARNING=350
export GPU_PCIE_GEN_WARNING=3
export GPU_PCIE_WIDTH_WARNING=2
#export ICHUNK_SIZE=1536
export ICHUNK_SIZE=384
export CHUNK_SIZE=5120
APP=$HPL_DIR/xhpl
#lrank=$OMPI_COMM_WORLD_LOCAL_RANK
lrank=$(($PMIX_RANK%4))
nrank=$(($PMIX_RANK/4))
#crank=$(($nrank/89))
#neven=$(($crank%2))
neven=$(($nrank%2))
#neven=0
export CUDA_VISIBLE_DEVICES=$lrank
echo "RANK $PMIX_RANK on host $HOSTNAME PID $$ even: $neven"
if [ $neven -eq 0 ]
then
case ${lrank} in
[0])
#ldd $APP
sudo nvidia-smi -ac 877,1395 > /dev/null;
#export PAMI_IBV_DEVICE_NAME=mlx5_0:1;
#export OMPI_MCA_btl_openib_if_include=mlx5_0:1;
export CUDA_VISIBLE_DEVICES=0; numactl --physcpubind=0,4,8,12,16,20,24,28,32,36 --membind=0 $APP
;;
[1])
#export PAMI_IBV_DEVICE_NAME=mlx5_1:1;
#export OMPI_MCA_btl_openib_if_include=mlx5_1:1;
export CUDA_VISIBLE_DEVICES=1; numactl --physcpubind=40,44,48,52,56,60,64,68,72,76 --membind=0 $APP
;;
[2])
#export PAMI_IBV_DEVICE_NAME=mlx5_0:1;
#export OMPI_MCA_btl_openib_if_include=mlx5_0:1;
export CUDA_VISIBLE_DEVICES=2; numactl --physcpubind=80,84,88,92,96,100,104,108,112,116 --membind=8 $APP
;;
[3])
#export PAMI_IBV_DEVICE_NAME=mlx5_3:1;
#export OMPI_MCA_btl_openib_if_include=mlx5_3:1;
export CUDA_VISIBLE_DEVICES=3; numactl --physcpubind=120,124,128,132,136,140,144,148,152,156 --membind=8 $APP
;;
esac
exit
fi

이제 다음과 같이 run_me_4_gpu_xlc_spectrum.sh를 수행하시면 됩니다. 대략 10분 이내의 시간이 걸릴 것입니다.

[cecuser@p1235-met1 hpl]$ ./run_me_4_gpu_xlc_spectrum.sh

중간값을 빼면 결과적으로는 아래와 같은 결과물이 display 됩니다. 결과는 공개하지 못하는 점 다시 한번 양해 부탁드립니다.

...

================================================================================
T/V N NB P Q Time Gflops
--------------------------------------------------------------------------------
WR03L2R4 128000 512 2 2 XXX X.XXXe+04
--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)= 0.0005540 ...... PASSED
================================================================================

2020년 2월 4일 화요일

IBM ppc64le 환경 CUDA 10.2에서의 Active: failed (Result: start-limit) error

NVIDIA CUDA 10.2를 ppc64le 아키텍처 (IBM POWER8/9)에 설치한 경우, 다음과 같이 nvidia-persistenced가 살지 못하고 error를 내는 바람에 nvidia-smi 등 CUDA 기능을 사용하지 못하는 경우가 있습니다.

[cecuser@p615-met1 ~]$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Ma ke sure that the latest NVIDIA driver is installed and running.

[cecuser@p615-met1 ~]$ sudo systemctl status nvidia-persistenced
● nvidia-persistenced.service - NVIDIA Persistence Daemon
Loaded: loaded (/usr/lib/systemd/system/nvidia-persistenced.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Mon 2020-02-03 20:33:38 EST; 36min ago
Process: 11480 ExecStart=/usr/bin/nvidia-persistenced --verbose (code=exited, status=1/FAILURE)

Feb 03 20:33:38 p615-met1 systemd[1]: nvidia-persistenced.service: control process exited, code=exite...us=1
Feb 03 20:33:38 p615-met1 systemd[1]: Failed to start NVIDIA Persistence Daemon.
Feb 03 20:33:38 p615-met1 systemd[1]: Unit nvidia-persistenced.service entered failed state.
Feb 03 20:33:38 p615-met1 systemd[1]: nvidia-persistenced.service failed.
Feb 03 20:33:38 p615-met1 systemd[1]: nvidia-persistenced.service holdoff time over, scheduling restart.
Feb 03 20:33:38 p615-met1 systemd[1]: Stopped NVIDIA Persistence Daemon.
Feb 03 20:33:38 p615-met1 systemd[1]: start request repeated too quickly for nvidia-persistenced.service
Feb 03 20:33:38 p615-met1 systemd[1]: Failed to start NVIDIA Persistence Daemon.
Feb 03 20:33:38 p615-met1 systemd[1]: Unit nvidia-persistenced.service entered failed state.
Feb 03 20:33:38 p615-met1 systemd[1]: nvidia-persistenced.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

보통 이 경우 다음과 같이 /etc/udev/rules.d/40-redhat.rules의 memory hotplug 관련 부분을 comment-out 처리하지 않아서 발생하는 것이 대부분입니다.

[cecuser@p615-met1 ~]$ sudo vi /etc/udev/rules.d/40-redhat.rules
...
#SUBSYSTEM!="memory", ACTION!="add", GOTO="memory_hotplug_end"
SUBSYSTEM=="*", GOTO="memory_hotplug_end"

그런데 이렇게 하고 나서 rebooting을 해도 여전히 같은 error가 나더군요. 한참 골머리를 앓았는데, 이제 보니 CUDA 10.2에 약간의 bug가 있어서 그런 것 같습니다. 아래와 같이 /etc/ld.so.conf.d/cuda-10-2.conf 속에 ppc64le-linux 대신 x86_64-linux의 directory가 들어가 있습니다.

이것만 손으로 다음과 같이 수정하시고 ldconfig를 수행해 주시면 됩니다.

[cecuser@p615-met1 ~]$ sudo vi /etc/ld.so.conf.d/cuda-10-2.conf
#/usr/local/cuda-10.2/targets/x86_64-linux/lib
/usr/local/cuda-10.2/targets/ppc64le-linux/lib

[cecuser@p615-met1 ~]$ sudo ldconfig

[cecuser@p615-met1 ~]$ nvidia-smi
Mon Feb 3 23:50:00 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... Off | 00000004:04:00.0 Off | 0 |
| N/A 31C P0 51W / 300W | 0MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... Off | 00000004:05:00.0 Off | 0 |
| N/A 34C P0 54W / 300W | 0MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla V100-SXM2... Off | 00000035:03:00.0 Off | 0 |
| N/A 32C P0 54W / 300W | 0MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla V100-SXM2... Off | 00000035:04:00.0 Off | 0 |
| N/A 35C P0 54W / 300W | 0MiB / 32510MiB | 3% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

별 것도 아닌 것으로 2시간 이상 소모했습니다...

WML-CE (Watson Machine Learning Community Edition, 구 PowerAI) 1.6.2 설치

IBM POWER 아키텍처 (POWER8/9, 즉 ppc64le)에서 tensorflow나 caffe 등 각종 deep learning framework을 제공해주던 무료 toolkit인 기존 PowerAI는 이미 다들 아시는 바와 같이 이름을 Watson Machine Learning Community Edition (WML-CE)로 변경했습니다. 물론 여전히 무료입니다만, 기존처럼 *.rpm이나 *.deb의 형태로 제공하지 않고 아예 별도의 conda channel을 만들어서 conda에서 설치하도록 하고 있습니다. 따라서, Anaconda가 prerequsite이며, 2020년 2월 초 현재 최신 버전인 1.6.2는 Ananconda 2019.07을 prerequisite으로 하고 있습니다. 아예 모든 것이 설치된 docker image 형태로도 제공됩니다.

자세한 원본 manual은 아래 link를 참조하시면 됩니다.

https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.2/navigation/wmlce_planning.html
https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.2/navigation/wmlce_install.html

여기서는 ppc64le Ubuntu 18.04, Python 3.7.5 환경에서 WML-CE 1.6.2를 설치해보겠습니다.

cecuser@p1234-kvm1:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

먼저 Anaconda 2019.07 버전을 download 받아 설치합니다.

cecuser@p1234-kvm1:~$ wget https://repo.continuum.io/archive/Anaconda3-2019.07-Linux-ppc64le.sh

cecuser@p1234-kvm1:~$ chmod a+x Anaconda3-2019.07-Linux-ppc64le.sh

cecuser@p1234-kvm1:~$ ./Anaconda3-2019.07-Linux-ppc64le.sh

설치가 끝나면 ~/.bashrc를 수행하여 conda init을 수행합니다.

cecuser@p1234-kvm1:~$ . ~/.bashrc

이제 IBM이 제공하는 WML-CE를 위한 conda channel을 conda에 추가합니다.

(base) cecuser@p1234-kvm1:~$ conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

Conda 가상 환경을 생성하여 거기에 WML-CE를 설치하기를 권장하므로, 먼저 python 3.7.5 환경으로 wmlce_env라는 이름의 virtual env를 만듭니다.

(base) cecuser@p1234-kvm1:~$ conda create --name wmlce_env python=3.7.5
...
The following NEW packages will be INSTALLED:

_libgcc_mutex pkgs/main/linux-ppc64le::_libgcc_mutex-0.1-main
ca-certificates pkgs/main/linux-ppc64le::ca-certificates-2020.1.1-0
certifi pkgs/main/linux-ppc64le::certifi-2019.11.28-py37_0
libedit pkgs/main/linux-ppc64le::libedit-3.1.20181209-hc058e9b_0
libffi pkgs/main/linux-ppc64le::libffi-3.2.1-hf62a594_5
libgcc-ng pkgs/main/linux-ppc64le::libgcc-ng-8.2.0-h822a55f_1
libstdcxx-ng pkgs/main/linux-ppc64le::libstdcxx-ng-8.2.0-h822a55f_1
ncurses pkgs/main/linux-ppc64le::ncurses-6.1-he6710b0_1
openssl pkgs/main/linux-ppc64le::openssl-1.1.1d-h7b6447c_3
pip pkgs/main/linux-ppc64le::pip-20.0.2-py37_1
python pkgs/main/linux-ppc64le::python-3.7.5-h4134adf_0
readline pkgs/main/linux-ppc64le::readline-7.0-h7b6447c_5
setuptools pkgs/main/linux-ppc64le::setuptools-45.1.0-py37_0
sqlite pkgs/main/linux-ppc64le::sqlite-3.30.1-h7b6447c_0
tk pkgs/main/linux-ppc64le::tk-8.6.8-hbc83047_0
wheel pkgs/main/linux-ppc64le::wheel-0.34.1-py37_0
xz pkgs/main/linux-ppc64le::xz-5.2.4-h14c3975_4
zlib pkgs/main/linux-ppc64le::zlib-1.2.11-h7b6447c_3

Proceed ([y]/n)? y
...
Downloading and Extracting Packages
ca-certificates-2020 | 125 KB | ##################################### | 100%
setuptools-45.1.0 | 511 KB | ##################################### | 100%
pip-20.0.2 | 1.7 MB | ##################################### | 100%
sqlite-3.30.1 | 2.3 MB | ##################################### | 100%
wheel-0.34.1 | 50 KB | ##################################### | 100%
python-3.7.5 | 32.5 MB | ##################################### | 100%
openssl-1.1.1d | 3.8 MB | ##################################### | 100%
certifi-2019.11.28 | 156 KB | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate wmlce_env
#
# To deactivate an active environment, use
#
# $ conda deactivate

이제 conda 가상환경인 wmlce_env를 활성화합니다.

(base) cecuser@p1234-kvm1:~$ conda activate wmlce_env

이제 WML-CE를 설치합니다. Conda package 이름은 여전히 PowerAI로 되어 있는 점에 유의하십시요. 아래와 같이 하면 tensorflow와 caffe2, pytorch 등 WML-CE에서 지원하는 모든 deep learning framework이 한꺼번에 다 설치됩니다. 혹시 WML-CE 전체를 설치하지 않고 가령 PyTorch만 설치하려 할 때는 그냥 conda install pytorch 라고 하시면 됩니다.

아래의 명령어로 어느어느 package들이 설치되는지 보여드리기 위해 긴 ouput을 일부러 다 옮겨 붙였습니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda install powerai
....
The following NEW packages will be INSTALLED:

_py-xgboost-mutex ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_py-xgboost-mutex-1.0-gpu_590.g8a21f75
_pytorch_select ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_pytorch_select-2.0-gpu_20238.g1faf942
_tflow_select ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::_tflow_select-2.1.0-gpu_840.g50de12c
absl-py pkgs/main/linux-ppc64le::absl-py-0.7.1-py37_0
apex ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::apex-0.1.0_1.6.2-py37_596.g1eb5c77
asn1crypto pkgs/main/linux-ppc64le::asn1crypto-1.3.0-py37_0
astor pkgs/main/linux-ppc64le::astor-0.7.1-py37_0
atomicwrites pkgs/main/linux-ppc64le::atomicwrites-1.3.0-py37_1
attrs pkgs/main/noarch::attrs-19.3.0-py_0
blas pkgs/main/linux-ppc64le::blas-1.0-openblas
bokeh pkgs/main/linux-ppc64le::bokeh-1.4.0-py37_0
boost pkgs/main/linux-ppc64le::boost-1.67.0-py37_4
bzip2 pkgs/main/linux-ppc64le::bzip2-1.0.8-h7b6447c_0
c-ares pkgs/main/linux-ppc64le::c-ares-1.15.0-h7b6447c_1001
caffe ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::caffe-1.0_1.6.2-5184.g7b10df4
caffe-base ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::caffe-base-1.0_1.6.2-gpu_py37_5184.g7b10df4
cairo pkgs/main/linux-ppc64le::cairo-1.14.12-h8948797_3
cffi pkgs/main/linux-ppc64le::cffi-1.12.3-py37h2e261b9_0
chardet pkgs/main/linux-ppc64le::chardet-3.0.4-py37_1003
click pkgs/main/linux-ppc64le::click-7.0-py37_0
cloudpickle pkgs/main/noarch::cloudpickle-1.2.2-py_0
coverage pkgs/main/linux-ppc64le::coverage-5.0-py37h7b6447c_0
cryptography pkgs/main/linux-ppc64le::cryptography-2.8-py37h1ba5d50_0
cudatoolkit ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudatoolkit-10.1.243-616.gc122b8b
cudnn ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudnn-7.6.3_10.1-590.g5627c5e
cycler pkgs/main/linux-ppc64le::cycler-0.10.0-py37_0
cytoolz pkgs/main/linux-ppc64le::cytoolz-0.10.1-py37h7b6447c_0
dask pkgs/main/noarch::dask-2.3.0-py_0
dask-core pkgs/main/noarch::dask-core-2.3.0-py_0
dask-cuda ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-cuda-0.9.1-py37_573.g9af8baa
dask-xgboost ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-xgboost-0.1.7-py37_579.g8a31cf5
ddl ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::ddl-1.5.0-py37_1287.gc90c6f2
ddl-tensorflow ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::ddl-tensorflow-1.5.0-py37_1007.g8dbb51d
decorator pkgs/main/noarch::decorator-4.4.1-py_0
distributed pkgs/main/noarch::distributed-2.3.2-py_1
ffmpeg pkgs/main/linux-ppc64le::ffmpeg-4.0-hcdf2ecd_0
fontconfig pkgs/main/linux-ppc64le::fontconfig-2.13.0-h9420a91_0
freeglut pkgs/main/linux-ppc64le::freeglut-3.0.0-hf484d3e_5
freetype pkgs/main/linux-ppc64le::freetype-2.9.1-h8a8886c_0
fsspec pkgs/main/noarch::fsspec-0.6.2-py_0
future pkgs/main/linux-ppc64le::future-0.17.1-py37_0
gast pkgs/main/linux-ppc64le::gast-0.2.2-py37_0
gflags ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::gflags-2.2.2-1624.g17209b3
glib pkgs/main/linux-ppc64le::glib-2.63.1-h5a9c865_0
glog ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::glog-0.3.5-1613.gd054598
google-pasta ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::google-pasta-0.1.6-py37_564.g04df2d9
graphite2 pkgs/main/linux-ppc64le::graphite2-1.3.13-h23475e2_0
graphsurgeon ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::graphsurgeon-0.4.1-py37_612.gb2bf6b9
grpcio pkgs/main/linux-ppc64le::grpcio-1.16.1-py37hf8bcb03_1
h5py pkgs/main/linux-ppc64le::h5py-2.8.0-py37h8d01980_0
harfbuzz pkgs/main/linux-ppc64le::harfbuzz-1.8.8-hffaf4a1_0
hdf5 pkgs/main/linux-ppc64le::hdf5-1.10.2-hba1933b_1
heapdict pkgs/main/noarch::heapdict-1.0.1-py_0
hypothesis pkgs/main/linux-ppc64le::hypothesis-3.59.1-py37h39e3cac_0
icu pkgs/main/linux-ppc64le::icu-58.2-h64fc554_1
idna pkgs/main/linux-ppc64le::idna-2.8-py37_0
imageio pkgs/main/linux-ppc64le::imageio-2.6.1-py37_0
importlib_metadata pkgs/main/linux-ppc64le::importlib_metadata-1.4.0-py37_0
jasper pkgs/main/linux-ppc64le::jasper-2.0.14-h07fcdf6_1
jinja2 pkgs/main/noarch::jinja2-2.10.3-py_0
joblib pkgs/main/linux-ppc64le::joblib-0.13.2-py37_0
jpeg pkgs/main/linux-ppc64le::jpeg-9b-hcb7ba68_2
keras-applications pkgs/main/noarch::keras-applications-1.0.8-py_0
keras-preprocessi~ pkgs/main/noarch::keras-preprocessing-1.1.0-py_1
kiwisolver pkgs/main/linux-ppc64le::kiwisolver-1.1.0-py37he6710b0_0
leveldb pkgs/main/linux-ppc64le::leveldb-1.20-hf484d3e_1
libboost pkgs/main/linux-ppc64le::libboost-1.67.0-h46d08c1_4
libgfortran-ng pkgs/main/linux-ppc64le::libgfortran-ng-7.3.0-h822a55f_1
libglu pkgs/main/linux-ppc64le::libglu-9.0.0-hf484d3e_1
libopenblas pkgs/main/linux-ppc64le::libopenblas-0.3.6-h5a2b251_1
libopencv ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libopencv-3.4.7-725.g92aa195
libopus pkgs/main/linux-ppc64le::libopus-1.3-h7b6447c_0
libpng pkgs/main/linux-ppc64le::libpng-1.6.37-hbc83047_0
libprotobuf ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libprotobuf-3.8.0-577.g45759bb
libtiff pkgs/main/linux-ppc64le::libtiff-4.1.0-h2733197_0
libuuid pkgs/main/linux-ppc64le::libuuid-1.0.3-h1bed415_2
libvpx pkgs/main/linux-ppc64le::libvpx-1.7.0-hf484d3e_0
libxcb pkgs/main/linux-ppc64le::libxcb-1.13-h1bed415_0
libxgboost-base ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libxgboost-base-0.90-gpu_590.g8a21f75
libxml2 pkgs/main/linux-ppc64le::libxml2-2.9.9-hea5a465_1
llvmlite pkgs/main/linux-ppc64le::llvmlite-0.29.0-py37hd408876_0
lmdb pkgs/main/linux-ppc64le::lmdb-0.9.22-hf484d3e_1
locket pkgs/main/linux-ppc64le::locket-0.2.0-py37_1
markdown pkgs/main/linux-ppc64le::markdown-3.1.1-py37_0
markupsafe pkgs/main/linux-ppc64le::markupsafe-1.1.1-py37h7b6447c_0
matplotlib pkgs/main/linux-ppc64le::matplotlib-3.1.2-py37_1
matplotlib-base pkgs/main/linux-ppc64le::matplotlib-base-3.1.2-py37h4fdacc2_1
mock pkgs/main/linux-ppc64le::mock-2.0.0-py37_0
more-itertools pkgs/main/noarch::more-itertools-8.0.2-py_0
msgpack-python pkgs/main/linux-ppc64le::msgpack-python-0.6.1-py37hfd86e86_1
nccl ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::nccl-2.4.8-586.gdba67b7
networkx pkgs/main/linux-ppc64le::networkx-2.2-py37_1
ninja pkgs/main/linux-ppc64le::ninja-1.9.0-py37hfd86e86_0
nomkl pkgs/main/linux-ppc64le::nomkl-3.0-0
numactl ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::numactl-2.0.12-573.gdf5dc62
numba pkgs/main/linux-ppc64le::numba-0.45.1-py37h962f231_0
numpy pkgs/main/linux-ppc64le::numpy-1.16.6-py37h30dfecb_0
numpy-base pkgs/main/linux-ppc64le::numpy-base-1.16.6-py37h2f8d375_0
olefile pkgs/main/linux-ppc64le::olefile-0.46-py37_0
onnx ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::onnx-1.5.0-py37_614.gd049fd7
openblas pkgs/main/linux-ppc64le::openblas-0.3.6-1
openblas-devel pkgs/main/linux-ppc64le::openblas-devel-0.3.6-1
opencv ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::opencv-3.4.7-py37_725.g92aa195
packaging pkgs/main/noarch::packaging-20.1-py_0
pai4sk ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pai4sk-1.5.0-py37_1071.g5abf42e
pandas pkgs/main/linux-ppc64le::pandas-1.0.0-py37h0573a6f_0
partd pkgs/main/noarch::partd-1.1.0-py_0
pbr pkgs/main/noarch::pbr-5.4.4-py_0
pciutils ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pciutils-3.6.2-571.g2316d13
pcre pkgs/main/linux-ppc64le::pcre-8.43-he6710b0_0
pillow pkgs/main/linux-ppc64le::pillow-6.2.1-py37h0d2faf8_0
pixman pkgs/main/linux-ppc64le::pixman-0.34.0-h1f8d8dc_3
pluggy pkgs/main/linux-ppc64le::pluggy-0.13.1-py37_0
powerai ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-1.6.2-615.g1dade79
powerai-license ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-license-1.6.2-716.g7081e12
powerai-release ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-release-1.6.2-572.gb216c2c
powerai-tools ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-tools-1.6.2-565.g97f2c3f
protobuf ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::protobuf-3.8.0-py37_587.gab45ad3
psutil pkgs/main/linux-ppc64le::psutil-5.5.0-py37h7b6447c_0
py pkgs/main/noarch::py-1.8.1-py_0
py-boost pkgs/main/linux-ppc64le::py-boost-1.67.0-py37h04863e7_4
py-opencv ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-opencv-3.4.7-py37_725.g92aa195
py-xgboost-base ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-xgboost-base-0.90-gpu_py37_590.g8a21f75
py-xgboost-gpu ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::py-xgboost-gpu-0.90-590.g8a21f75
pycparser pkgs/main/linux-ppc64le::pycparser-2.19-py37_0
pyopenssl pkgs/main/linux-ppc64le::pyopenssl-19.1.0-py37_0
pyparsing pkgs/main/noarch::pyparsing-2.4.6-py_0
pysocks pkgs/main/linux-ppc64le::pysocks-1.7.1-py37_0
pytest pkgs/main/linux-ppc64le::pytest-4.4.2-py37_0
python-dateutil pkgs/main/noarch::python-dateutil-2.8.1-py_0
python-lmdb pkgs/main/linux-ppc64le::python-lmdb-0.94-py37h14c3975_0
pytorch ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pytorch-1.2.0-20238.g1faf942
pytorch-base ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pytorch-base-1.2.0-gpu_py37_20238.g1faf942
pytz pkgs/main/noarch::pytz-2019.3-py_0
pywavelets pkgs/main/linux-ppc64le::pywavelets-1.1.1-py37h7b6447c_0
pyyaml pkgs/main/linux-ppc64le::pyyaml-5.1.2-py37h7b6447c_0
requests pkgs/main/linux-ppc64le::requests-2.22.0-py37_1
scikit-image pkgs/main/linux-ppc64le::scikit-image-0.15.0-py37he6710b0_0
scikit-learn pkgs/main/linux-ppc64le::scikit-learn-0.21.3-py37h22eb022_0
scipy pkgs/main/linux-ppc64le::scipy-1.3.1-py37he2b7bc3_0
simsearch ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::simsearch-1.1.0-py37_764.g7c5f6cf
six pkgs/main/linux-ppc64le::six-1.12.0-py37_0
snapml-spark ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::snapml-spark-1.4.0-py37_942.gc873569
snappy pkgs/main/linux-ppc64le::snappy-1.1.7-h1532aa0_3
sortedcontainers pkgs/main/linux-ppc64le::sortedcontainers-2.1.0-py37_0
spectrum-mpi ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::spectrum-mpi-10.03-622.gfc88b70
tabulate pkgs/main/linux-ppc64le::tabulate-0.8.2-py37_0
tblib pkgs/main/noarch::tblib-1.6.0-py_0
tensorboard ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorboard-1.15.0-py37_ab7f72a_3645.gf4f525e
tensorflow ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-1.15.0-gpu_py37_841.g50de12c
tensorflow-base ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-base-1.15.0-gpu_py37_590d6ee_64210.g4a039ec
tensorflow-estima~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-estimator-1.15.1-py37_a5f60ce_1351.g50de12c
tensorflow-gpu ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-gpu-1.15.0-841.g50de12c
tensorflow-large-~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-large-model-support-2.0.2-py37_970.gfa57a9e
tensorflow-probab~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-probability-0.8.0-py37_b959b26_2686.g50de12c
tensorflow-servin~ ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorflow-serving-api-1.15.0-py37_748217e_5094.g89559ef
tensorrt ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tensorrt-6.0.1.5-py37_612.gb2bf6b9
termcolor pkgs/main/linux-ppc64le::termcolor-1.1.0-py37_1
tf_cnn_benchmarks ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::tf_cnn_benchmarks-1.15-gpu_py37_1374.g5e94b18
toolz pkgs/main/noarch::toolz-0.10.0-py_0
torchtext ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::torchtext-0.4.0-py37_578.g5bf3960
torchvision-base ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::torchvision-base-0.4.0-gpu_py37_593.g80f339d
tornado pkgs/main/linux-ppc64le::tornado-6.0.3-py37h7b6447c_0
tqdm pkgs/main/noarch::tqdm-4.32.1-py_0
typing pkgs/main/linux-ppc64le::typing-3.6.4-py37_0
uff ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::uff-0.6.5-py37_612.gb2bf6b9
urllib3 pkgs/main/linux-ppc64le::urllib3-1.25.8-py37_0
werkzeug pkgs/main/noarch::werkzeug-0.15.4-py_0
wrapt pkgs/main/linux-ppc64le::wrapt-1.11.2-py37h7b6447c_0
yaml pkgs/main/linux-ppc64le::yaml-0.1.7-h1bed415_2
zict pkgs/main/noarch::zict-1.0.0-py_0
zipp pkgs/main/noarch::zipp-0.6.0-py_0
zstd pkgs/main/linux-ppc64le::zstd-1.3.7-h0b5b093_0

Proceed ([y]/n)? y
...

아래와 같이 설치된 package들을 각각 확인하시면 됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep tensorflow
ddl-tensorflow 1.5.0 py37_1007.g8dbb51d https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow 1.15.0 gpu_py37_841.g50de12c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-base 1.15.0 gpu_py37_590d6ee_64210.g4a039ec https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-estimator 1.15.1 py37_a5f60ce_1351.g50de12c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-gpu 1.15.0 841.g50de12c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-large-model-support 2.0.2 py37_970.gfa57a9e https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-probability 0.8.0 py37_b959b26_2686.g50de12c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorflow-serving-api 1.15.0 py37_748217e_5094.g89559ef https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep caffe
caffe 1.0_1.6.2 5184.g7b10df4 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
caffe-base 1.0_1.6.2 gpu_py37_5184.g7b10df4 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep pytorch
_pytorch_select 2.0 gpu_20238.g1faf942 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
pytorch 1.2.0 20238.g1faf942 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
pytorch-base 1.2.0 gpu_py37_20238.g1faf942 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

WML-CE에서는 NCCL와 CUDNN 등 base facility도 함께 제공되어 설치됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep -i nccl
nccl 2.4.8 586.gdba67b7 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

(wmlce_env) cecuser@p1234-kvm1:~$ conda list | grep -i dnn
cudnn 7.6.3_10.1 590.g5627c5e https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

아래와 같이 python에서 import를 해보셔도 됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ python
Python 3.7.5 (default, Oct 25 2019, 16:29:01)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> import tensorflow as tf
2020-02-03 21:31:31.095570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
>>>

다만 "conda install powerai" 만으로는 RAPIDS까지 설치되지는 않기 때문에, 아래와 같이 별도로 설치하셔야 합니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda install powerai-rapids
...
The following NEW packages will be INSTALLED:

arrow-cpp ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::arrow-cpp-0.15.1-py37_603.g702c836
boost-cpp pkgs/main/linux-ppc64le::boost-cpp-1.67.0-h14c3975_4
brotli pkgs/main/linux-ppc64le::brotli-1.0.6-he6710b0_0
cudf ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cudf-0.9.0-cuda10.1_py37_626.gddcad2d
cuml ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cuml-0.9.1-cuda10.1_py37_605.gfe9e07b
cupy ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::cupy-6.2.0-py37_567.g0f1e2ef
cython pkgs/main/linux-ppc64le::cython-0.29.14-py37he6710b0_0
dask-cudf ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dask-cudf-0.9.0-py37_575.g0416adf
dlpack ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::dlpack-0.2-562.g28dffd9
double-conversion ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::double-conversion-3.1.5-564.g4b43169
fastavro ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::fastavro-0.22.4-py37_562.g9525976
fastrlock pkgs/main/linux-ppc64le::fastrlock-0.4-py37he6710b0_0
grpc-cpp ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::grpc-cpp-1.23.0-568.g4f71a06
libcudf ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libcudf-0.9.0-cuda10.1_609.g113236a
libcuml ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libcuml-0.9.1-cuda10.1_576.ga304a0a
libevent ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libevent-2.1.8-561.ge1d98f7
libnvstrings ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::libnvstrings-0.9.0-cuda10.1_570.ga04797c
librmm ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::librmm-0.9.0-cuda10.1_567.gff1b1a1
lz4-c pkgs/main/linux-ppc64le::lz4-c-1.8.1.2-h14c3975_0
nvstrings ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::nvstrings-0.9.0-cuda10.1_py37_580.gdbb6546
parquet-cpp ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::parquet-cpp-1.5.1-579.g6eecc60
powerai-rapids ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::powerai-rapids-1.6.2-560.ga7c5a47
pyarrow ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::pyarrow-0.15.1-py37_609.g3a6717a
re2 ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::re2-2019.08.01-561.gef92448
rmm ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::rmm-0.9.0-cuda10.1_py37_569.g04c75fb
thrift-cpp ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::thrift-cpp-0.12.0-580.gf96fa62
uriparser ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le::uriparser-0.9.3-561.g7465fef

The following packages will be DOWNGRADED:

pandas 1.0.0-py37h0573a6f_0 --> 0.24.2-py37he6710b0_0

Proceed ([y]/n)? y
...

만약 GPU가 없는 시스템에서 tensorflow나 pytorch 등을 사용하시고자 할 때는, 아래와 같이 CPU-only 버전의 WML-CE를 설치하시면 됩니다.

(wmlce_env) cecuser@p1234-kvm1:~$ conda install powerai-cpu

2020년 1월 28일 화요일

CentOS 환경의 IBM POWER9 서버에 xCAT 설치하기

xCAT은 eXtreme Cluster/Cloud Administration Toolkit의 약자로서, 대형 HPC cluster의 설치와 운영에 사용되는 open source tool입니다. 설치는 IBM POWER9 (ppc64le) 환경에서도 x86 환경과 동일하게 간단합니다. 여기서는 CentOS 7.6 환경에서 수행해보겠습니다.

-bash-4.2# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (AltArch)

먼저 go-xcat script를 다음과 같이 download 받습니다.

-bash-4.2# wget https://raw.githubusercontent.com/xcat2/xcat-core/master/xCAT-server/share/xcat/tools/go-xcat -O - >/tmp/go-xcat

-bash-4.2# chmod a+x /tmp/go-xcat

이제 이 script를 install이라는 argument와 함께 수행합니다. 그러면 Yes를 한번 눌러주는 것 외에는 사실상 더 해줄 것이 없습니다.

-bash-4.2# /tmp/go-xcat install
Operating system: linux
Architecture: ppc64le
Linux Distribution: centos
Version: 7
go-xcat Version: 1.0.45

Reading repositories ...... done

xCAT Core Packages
==================

Package Name Installed In Repository
------------ --------- -------------
perl-xCAT (not installed) 2.15-snap201911041517
xCAT (not installed) 2.15-snap201911041517
xCAT-SoftLayer (not installed) 2.15-snap201911041517
xCAT-buildkit (not installed) 2.15-snap201911041517
xCAT-client (not installed) 2.15-snap201911041517
xCAT-confluent (not installed) 2.15-snap201911041517
xCAT-csm (not installed) 2.15-snap201911041517
xCAT-genesis-scripts-ppc64 (not installed) 2.15-snap201911041517
xCAT-genesis-scripts-x86_64 (not installed) 2.15-snap201911041517
xCAT-openbmc-py (not installed) 2.15-snap201911041518
xCAT-probe (not installed) 2.15-snap201911041517
xCAT-server (not installed) 2.15-snap201911041517
xCAT-test (not installed) 2.15-snap201911041517
xCAT-vlan (not installed) 2.15-snap201911041517
xCATsn (not installed) 2.15-snap201911041517

xCAT Dependency Packages
========================

Package Name Installed In Repository
------------ --------- -------------
elilo-xcat (not installed) 3.14-4
fping (not installed) 2.4b2_to-2
goconserver (not installed) 0.3.2-snap201909171016
grub2-xcat (not installed) 2.02-0.76.el7.1.snap2019051602
ipmitool-xcat (not installed) 1.8.18-0
lldpd (not installed) 0.7.15-6.1
net-snmp-perl (not installed) 5.7.2-38.el7_6.2
perl-AppConfig (not installed) 1.66-20.el7
perl-Crypt-CBC (not installed) 2.33-2.el7
perl-Crypt-Rijndael (not installed) 1.09-2.ael7a
perl-Expect (not installed) 1.21-1
perl-HTML-Form (not installed) 6.03-6.el7
perl-HTTP-Async (not installed) 0.30-2
perl-IO-Stty (not installed) 0.03-1
perl-IO-Tty (not installed) 1.10-11.el7
perl-JSON (not installed) 2.59-2.el7
perl-Net-HTTPS-NB (not installed) 0.14-2
perl-Net-Telnet (not installed) 3.03-19.el7
perl-SOAP-Lite (not installed) 0.710.08-1
perl-XML-Simple (not installed) 2.20-5.el7
pyodbc (not installed) 3.0.7-1.ael7a
syslinux-xcat (not installed) 3.86-2
systemconfigurator (not installed) 2.2.11-1
systemimager-client (not installed) 4.3.0-0.1
systemimager-common (not installed) 4.3.0-0.1
systemimager-server (not installed) 4.3.0-0.3
xCAT-genesis-base-ppc64 (not installed) 2.14.5-snap201811160710
xCAT-genesis-base-x86_64 (not installed) 2.14.5-snap201811190037
xnba-undi (not installed) 1.0.3-131028
yaboot-xcat (not installed) 1.3.17-rc1

xCAT is going to be installed.
Continue? [y/n] y

....
Install 15 Packages (+95 Dependent packages)

Total download size: 257 M
Installed size: 713 M
Downloading packages:
--------------------------------------------------------------------------------
Total 2.9 MB/s | 257 MB 01:29
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : perl-Data-Dumper-2.145-3.el7.ppc64le 1/110
Installing : 1:perl-Compress-Raw-Zlib-2.061-4.el7.ppc64le 2/110
Installing : perl-Sys-Syslog-0.33-3.el7.ppc64le 3/110
Installing : ksh-20120801-139.el7.ppc64le 4/110
Installing : perl-XML-NamespaceSupport-1.11-10.el7.noarch 5/110
Installing : apr-1.4.8-3.el7_4.1.ppc64le 6/110
Installing : perl-XML-Parser-2.41-10.el7.ppc64le 7/110
Installing : apr-util-1.5.2-6.el7.ppc64le 8/110
Installing : 12:dhcp-libs-4.2.5-68.el7.centos.1.ppc64le 9/110
...
tftp.ppc64le 0:5.2-22.el7
tftp-server.ppc64le 0:5.2-22.el7
trousers.ppc64le 0:0.3.14-2.el7
unzip.ppc64le 0:6.0-19.el7
xCAT-probe.noarch 4:2.15-snap201911041517
yajl.ppc64le 0:2.0.4-4.el7

Complete!

xCAT has been installed!
========================

If this is the very first time xCAT has been installed, run one of the
following commands to set the environment variables.

For sh:
source /etc/profile.d/xcat.sh

For csh:
source /etc/profile.d/xcat.csh

이렇게 설치가 쉬운 이유는 go-xcat이라는 script에서 자동으로 xcat 관련 YUM repository를 만들었기 때문입니다.

-bash-4.2# yum repolist
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
repo id repo name status
base CentOS-7 - Base 7,477
extras CentOS-7 - Extras 403
updates CentOS-7 - Updates 1,873
xcat-core xcat-core 21
xcat-dep xcat-dep 30
repolist: 9,804

설치 뒤에는 자동으로 xCAT daemon이 수행되는데, 제대로 떠있는지 다음과 같이 확인합니다.

-bash-4.2# systemctl status xcatd
● xcatd.service - xCAT management service
Loaded: loaded (/usr/lib/systemd/system/xcatd.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2020-01-28 02:48:59 EST; 7min ago
Main PID: 14973 (xcatd: SSL list)
CGroup: /system.slice/xcatd.service
├─14972 /usr/sbin/in.tftpd -v -l -s /tftpboot -m /etc/tftpmapfile4xcat.conf
├─14973 xcatd: SSL listener
├─14974 xcatd: DB Access
├─14975 xcatd: UDP listener
├─14976 xcatd: Discovery worker
├─14977 xcatd: install monitor
└─14978 xcatd: Command log writer

Jan 28 02:48:28 p636-kvm1 xcat[14943]: xcatd is going to start...
Jan 28 02:48:58 p636-kvm1 xcat[14975]: xcatd: UDP listener process 14975 start
Jan 28 02:48:58 p636-kvm1 xcat[14977]: xcatd: install monitor process 14977 start
Jan 28 02:48:58 p636-kvm1 xcat[14978]: xcatd: Command log writer process 14978 start
Jan 28 02:48:58 p636-kvm1 xcat[14976]: xcatd: Discovery worker process 14976 start
Jan 28 02:48:59 p636-kvm1 xcatd[14939]: Starting xcatd [ OK ]
Jan 28 02:48:59 p636-kvm1 systemd[1]: Started xCAT management service.
Jan 28 02:49:02 p636-kvm1 xcat[15071]: xCAT: Allowing mknb ppc64 for root from localhost
Jan 28 02:49:47 p636-kvm1 xcat[15089]: xCAT: Allowing mknb x86_64 for root from localhost
Jan 28 02:52:45 p636-kvm1 xcat[15209]: xCAT: Allowing lsdef for root from localhost

xCAT daemon의 버전은 다음과 같이 확인할 수 있습니다.

-bash-4.2# source /etc/profile.d/xcat.sh

-bash-4.2# lsxcatd -a
Version 2.15 (git commit 218c6d3acc8bdbd7f72115e48cda2b1a3613d18a, built Mon Nov 4 15:17:59 EST 2019)
This is a Management Node
dbengine=SQLite

이제 간단한 xCAT 명령을 사용해보겠습니다. xCAT에서 network을 등록하고 관리하는 방법입니다. 아래와 같이 먼저 makenetworks 명령을 통해 이미 구성되어 있는 NIC을 발견하여 network object를 만들어야 하는데, 실은 xCAT이 설치되고 구성될 때 이미 만들어져 있습니다.

-bash-4.2# makenetworks
Warning: [p636-kvm1]: The network entry '129_40_XXX_48-255_255_255_240' already exists in xCAT networks table. Cannot create a definition for '129_40_XXX_48-255_255_255_240'

이 network은 다음과 같이 볼 수 있습니다.

-bash-4.2# lsdef -t network
129_40_XXX_48-255_255_255_240 (network)

이것의 이름을 좀더 쉬운 이름으로 바꾸어 보겠습니다.

-bash-4.2# chdef -t network 129_40_XXX_48-255_255_255_240 -n primary_network
Changed the object name from 129_40_XXX_48-255_255_255_240 to primary_network.

-bash-4.2# lsdef -t network
primary_network (network)

자세히 보기 위해서는 다음과 같이 할 수 있습니다.

-bash-4.2# lsdef -t network primary_network
Object name: primary_network
gateway=129.40.XXX.62
mask=255.255.255.240
mgtifname=eth0
mtu=9000
net=129.40.XXX.48
tftpserver=<xcatmaster>