IBM PowerAI (Watson ML Community Edition)이 설치된 Ubuntu ppc64le 기반의 docker image 만들기

 

먼저 다음 링크를 참조하여 ppc64le (IBM POWER9) nvidia-docker2 환경에서 Ubuntu 기반의 docker image를 만듭니다.  

http://hwengineer.blogspot.com/2019/05/ppc64le-ibm-power9-nvidia-docker2.html

참고로 ppc64le (IBM POWER9)에서의 CUDA 설치는 NVIDIA CUDA download page의 안내와 같이 아래처럼 진행하시면 됩니다.

# wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/cuda-repo-ubuntu1804_10.1.105-1_ppc64el.deb

# dpkg -i cuda-repo-ubuntu1804_10.1.105-1_ppc64el.deb

# apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/7fa2af80.pub

# apt-get update

# apt-get install cuda

또는 이미 만들어둔 docker image를 다음과 같이 pull 해와도 됩니다.

# docker pull bsyu/ubuntu18.04_cuda10-1_ppc64le:v0.1

이렇게 pull 받아온 docker image를 확인합니다.

root@unigpu:/files/docker# docker images

REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE

bsyu/ubuntu18.04_cuda10-1_ppc64le   v0.1                ef8dd4d654e7        2 hours ago         6.33GB

ubuntu                              18.04               ecc8dc2e4170        4 weeks ago         106MB

이 image를 다음과 같이 구동합니다.  

root@unigpu:/files/docker# docker run --runtime=nvidia -ti --rm bsyu/ubuntu18.04_cuda10-1_ppc64lel:v0.1 bash

이제 그 image 속에서 다음과 같이 IBM PowerAI (IBM Watson ML Community Edition)을 설치합니다.  이는 IBM이 마련한 conda channel을 등록하고 거기에서 conda install 명령을 수행하는 방식으로 설치됩니다.

# conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

# conda create --name wmlce_env python=3.6

# conda activate wmlce_env

# apt-get install openssh-server

# conda install powerai     

위와 같이 conda install powerai 명령을 내리면 tensorflow 뿐만 아니라 caffe, pytorch 등이 모두 설치됩니다.  가령 Tensorflow 1.14만 설치하고자 한다면 위 명령 대신 conda install tensorflow=1.14를 수행하시면 됩니다.

powerai 전체 package 설치는 network 사정에 따라 1~2시간이 걸리기도 합니다.  설치가 완료되면 다음과 같이 docker commit하여 docker image를 저장합니다.

# docker ps -a

CONTAINER ID        IMAGE                                   COMMAND             CREATED             STATUS              PORTS               NAMES

45fb663f025c        bsyu/ubuntu18.04_cuda10-1_ppc64le:v0.1   "bash"              42 seconds ago      Up 39 seconds                           elastic_gates

이어서 v0.2 등의 새로운 tag로 commit 하시면 됩니다.

[root@ac922 docker]# docker commit 45fb663f025c bsyu/ubuntu18.04_cuda10-1_ppc64le:v0.2

아래는 그렇게 만들어진 docker image들의 사용예입니다.   제가 만든 그런 image들은 https://hub.docker.com/u/bsyu 에 올려져 있습니다.

root@unigpu:~# docker run --runtime=nvidia -ti --rm bsyu/ubuntu18.04_cuda10-1_tf1.15_pytorch1.2_ppc64le:latest 

(wmlce_env) root@bdbf11e90094:/# python

Python 3.6.10 |Anaconda, Inc.| (default, Mar 26 2020, 00:22:27)

[GCC 7.3.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import torch

>>> import tensorflow as tf

2020-05-29 04:53:09.637141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1

>>> sess=tf.Session()

2020-05-29 04:53:20.795027: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1

2020-05-29 04:53:20.825918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:

name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53

pciBusID: 0004:04:00.0

2020-05-29 04:53:20.827162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties:

name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53

pciBusID: 0004:05:00.0

2020-05-29 04:53:20.828421: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 2 with properties:

name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53

pciBusID: 0035:03:00.0

2020-05-29 04:53:20.829655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 3 with properties:

name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53

pciBusID: 0035:04:00.0

ppc64le (IBM POWER9) nvidia-docker2 환경에서 Ubuntu 기반의 docker image 만들기

Redhat 7.5 ppc64le (IBM POWER9) 위에 설치된 nvidia-docker2 환경에서 Ubuntu 기반의 docker image를 만드는 과정을 정리했습니다. 

먼저, 아래 posting 참조해서 Redhat 7.5 ppc64le에 nvidia-docker2 (정확하게는 nvidia-docker2가 아니라 nvidia-runtime)을 설치합니다.

http://hwengineer.blogspot.com/2019/04/ppc64le-redhat-7-nvidia-docker2-nvidia.html

아래와 같이 ubuntu 관련 apt key값들을 받아둡니다.  이것들은 나중에 ubuntu 기반의 docker image를 만들 때 사용됩니다.

[root@ac922 tmp]# mkdir docker

[root@ac922 tmp]# cd docker

[root@ac922 docker]# curl -fsSL https://download.docker.com/linux/ubuntu/gpg > apt.key1

[root@ac922 docker]# curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/7fa2af80.pub > apt.key2

참고로, 이 key 파일들은 아래와 같이 생긴 text file들입니다.

[root@ac922 docker]# cat apt.key1
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBFit2ioBEADhWpZ8/wvZ6hUTiXOwQHXMAlaFHcPH9hAtr4F1y2+OYdbtMuth
lqqwp028AqyY+PRfVMtSYMbjuQuu5byyKR01BbqYhuS3jtqQmljZ/bJvXqnmiVXh
38UuLa+z077PxyxQhu5BbqntTPQMfiyqEiU+BKbq2WmANUKQf+1AmZY/IruOXbnq
L4C1+gJ8vfmXQt99npCaxEjaNRVYfOS8QcixNzHUYnb6emjlANyEVlZzeqo7XKl7
UrwV5inawTSzWNvtjEjj4nJL8NsLwscpLPQUhTQ+7BbQXAwAmeHCUTQIvvWXqw0N
cmhh4HgeQscQHYgOJjjDVfoY5MucvglbIgCqfzAHW9jxmRL4qbMZj+b1XoePEtht
ku4bIQN1X5P07fNWzlgaRL5Z4POXDDZTlIQ/El58j9kp4bnWRCJW0lya+f8ocodo
vZZ+Doi+fy4D5ZGrL4XEcIQP/Lv5uFyf+kQtl/94VFYVJOleAv8W92KdgDkhTcTD
G7c0tIkVEKNUq48b3aQ64NOZQW7fVjfoKwEZdOqPE72Pa45jrZzvUFxSpdiNk2tZ
XYukHjlxxEgBdC/J3cMMNRE1F4NCA3ApfV1Y7/hTeOnmDuDYwr9/obA8t016Yljj
q5rdkywPf4JF8mXUW5eCN1vAFHxeg9ZWemhBtQmGxXnw9M+z6hWwc6ahmwARAQAB
tCtEb2NrZXIgUmVsZWFzZSAoQ0UgZGViKSA8ZG9ja2VyQGRvY2tlci5jb20+iQI3
BBMBCgAhBQJYrefAAhsvBQsJCAcDBRUKCQgLBRYCAwEAAh4BAheAAAoJEI2BgDwO
v82IsskP/iQZo68flDQmNvn8X5XTd6RRaUH33kXYXquT6NkHJciS7E2gTJmqvMqd
tI4mNYHCSEYxI5qrcYV5YqX9P6+Ko+vozo4nseUQLPH/ATQ4qL0Zok+1jkag3Lgk
jonyUf9bwtWxFp05HC3GMHPhhcUSexCxQLQvnFWXD2sWLKivHp2fT8QbRGeZ+d3m
6fqcd5Fu7pxsqm0EUDK5NL+nPIgYhN+auTrhgzhK1CShfGccM/wfRlei9Utz6p9P
XRKIlWnXtT4qNGZNTN0tR+NLG/6Bqd8OYBaFAUcue/w1VW6JQ2VGYZHnZu9S8LMc
FYBa5Ig9PxwGQOgq6RDKDbV+PqTQT5EFMeR1mrjckk4DQJjbxeMZbiNMG5kGECA8
g383P3elhn03WGbEEa4MNc3Z4+7c236QI3xWJfNPdUbXRaAwhy/6rTSFbzwKB0Jm
ebwzQfwjQY6f55MiI/RqDCyuPj3r3jyVRkK86pQKBAJwFHyqj9KaKXMZjfVnowLh
9svIGfNbGHpucATqREvUHuQbNnqkCx8VVhtYkhDb9fEP2xBu5VvHbR+3nfVhMut5
G34Ct5RS7Jt6LIfFdtcn8CaSas/l1HbiGeRgc70X/9aYx/V/CEJv0lIe8gP6uDoW
FPIZ7d6vH+Vro6xuWEGiuMaiznap2KhZmpkgfupyFmplh0s6knymuQINBFit2ioB
EADneL9S9m4vhU3blaRjVUUyJ7b/qTjcSylvCH5XUE6R2k+ckEZjfAMZPLpO+/tF
M2JIJMD4SifKuS3xck9KtZGCufGmcwiLQRzeHF7vJUKrLD5RTkNi23ydvWZgPjtx
Q+DTT1Zcn7BrQFY6FgnRoUVIxwtdw1bMY/89rsFgS5wwuMESd3Q2RYgb7EOFOpnu
w6da7WakWf4IhnF5nsNYGDVaIHzpiqCl+uTbf1epCjrOlIzkZ3Z3Yk5CM/TiFzPk
z2lLz89cpD8U+NtCsfagWWfjd2U3jDapgH+7nQnCEWpROtzaKHG6lA3pXdix5zG8
eRc6/0IbUSWvfjKxLLPfNeCS2pCL3IeEI5nothEEYdQH6szpLog79xB9dVnJyKJb
VfxXnseoYqVrRz2VVbUI5Blwm6B40E3eGVfUQWiux54DspyVMMk41Mx7QJ3iynIa
1N4ZAqVMAEruyXTRTxc9XW0tYhDMA/1GYvz0EmFpm8LzTHA6sFVtPm/ZlNCX6P1X
zJwrv7DSQKD6GGlBQUX+OeEJ8tTkkf8QTJSPUdh8P8YxDFS5EOGAvhhpMBYD42kQ
pqXjEC+XcycTvGI7impgv9PDY1RCC1zkBjKPa120rNhv/hkVk/YhuGoajoHyy4h7
ZQopdcMtpN2dgmhEegny9JCSwxfQmQ0zK0g7m6SHiKMwjwARAQABiQQ+BBgBCAAJ
BQJYrdoqAhsCAikJEI2BgDwOv82IwV0gBBkBCAAGBQJYrdoqAAoJEH6gqcPyc/zY
1WAP/2wJ+R0gE6qsce3rjaIz58PJmc8goKrir5hnElWhPgbq7cYIsW5qiFyLhkdp
YcMmhD9mRiPpQn6Ya2w3e3B8zfIVKipbMBnke/ytZ9M7qHmDCcjoiSmwEXN3wKYI
mD9VHONsl/CG1rU9Isw1jtB5g1YxuBA7M/m36XN6x2u+NtNMDB9P56yc4gfsZVES
KA9v+yY2/l45L8d/WUkUi0YXomn6hyBGI7JrBLq0CX37GEYP6O9rrKipfz73XfO7
JIGzOKZlljb/D9RX/g7nRbCn+3EtH7xnk+TK/50euEKw8SMUg147sJTcpQmv6UzZ
cM4JgL0HbHVCojV4C/plELwMddALOFeYQzTif6sMRPf+3DSj8frbInjChC3yOLy0
6br92KFom17EIj2CAcoeq7UPhi2oouYBwPxh5ytdehJkoo+sN7RIWua6P2WSmon5
U888cSylXC0+ADFdgLX9K2zrDVYUG1vo8CX0vzxFBaHwN6Px26fhIT1/hYUHQR1z
VfNDcyQmXqkOnZvvoMfz/Q0s9BhFJ/zU6AgQbIZE/hm1spsfgvtsD1frZfygXJ9f
irP+MSAI80xHSf91qSRZOj4Pl3ZJNbq4yYxv0b1pkMqeGdjdCYhLU+LZ4wbQmpCk
SVe2prlLureigXtmZfkqevRz7FrIZiu9ky8wnCAPwC7/zmS18rgP/17bOtL4/iIz
QhxAAoAMWVrGyJivSkjhSGx1uCojsWfsTAm11P7jsruIL61ZzMUVE2aM3Pmj5G+W
9AcZ58Em+1WsVnAXdUR//bMmhyr8wL/G1YO1V3JEJTRdxsSxdYa4deGBBY/Adpsw
24jxhOJR+lsJpqIUeb999+R8euDhRHG9eFO7DRu6weatUJ6suupoDTRWtr/4yGqe
dKxV3qQhNLSnaAzqW/1nA3iUB4k7kCaKZxhdhDbClf9P37qaRW467BLCVO/coL3y
Vm50dwdrNtKpMBh3ZpbB1uJvgi9mXtyBOMJ3v8RZeDzFiG8HdCtg9RvIt/AIFoHR
H3S+U79NT6i0KPzLImDfs8T7RlpyuMc4Ufs8ggyg9v3Ae6cN3eQyxcK3w0cbBwsh
/nQNfsA6uu+9H7NhbehBMhYnpNZyrHzCmzyXkauwRAqoCbGCNykTRwsur9gS41TQ
M8ssD1jFheOJf3hODnkKU+HKjvMROl1DK7zdmLdNzA1cvtZH/nCC9KPj1z8QC47S
xx+dTZSx4ONAhwbS/LN3PoKtn8LPjY9NP9uDWI+TWYquS2U+KHDrBDlsgozDbs/O
jCxcpDzNmXpWQHEtHU7649OXHP7UeNST1mCUCH5qdank0V1iejF6/CfTFU4MfcrG
YT90qFF93M3v01BbxP+EIY2/9tiIPbrd
=0YYh
-----END PGP PUBLIC KEY BLOCK-----

[root@ac922 docker]# cat apt.key2
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1

mQINBFdtt4UBEAC8FDSWMR07GJZ265giLn7kLF+EsJCWESUq6Cd13QN0JQ/tLibi
QlW4ZjeOnEH9VPlqh/mKqNMG4SwRt8S+GHpePMQrr0aOkiRGfCclnAWIZURSAP+t
PLelCt43fkw1BBTopd/0oOzO8kHu8j8WU4A8GHxqghfFWPv54FQs2iaZ2eWR7a6d
79IJrbDKaVCCiQrkhCM8m648pNKHhuoJ9cQXFV+uvwkpfmKWGQ4ultxlOyjLHJLF
vuML2RuAO9IxbdZjzeYNN+T+wjFIBVcPnwEO+WrYgvGkT4r9aqVqTeg3EPb7QclV
sKBVJdxk4jZl0y22HAWqScVi6SJ15uK9pXxywDZkbpuRBWx4ThWiGe/FiUa2igi9
/SIvqN2TBY0g18sRTrylVr1wE1UGa/y7nDx6PoGCP1frBt8YUYt3pkM8Xvb2CRxx
CyWwmuFEQHC6jCEWf7FnoBHBYQwTVGNrU0vkuIeDrm+ZAcv8wx+ie1hlFhqCCJnf
jqeQ0/zA9RPmCPOkLyTdSsNZtlxxk7bzCdTdFFKzBjGTR7Gz3SMSp23d11eIyRiF
HQsp2v0SvnPJ6OcgB95Hmo544vi3RuoVfovtDOdfSBCRxP+GhhxkKSrTleQjD0/r
CGkdG2Kox3m9YllAsvZchLXlS7bZV9mGRF61mVMjF3HJRUQfBBm89VPQ+QARAQAB
tCBjdWRhdG9vbHMgPGN1ZGF0b29sc0BudmlkaWEuY29tPokCNwQTAQgAIQUCV223
hQIbAwULCQgHAwUVCgkICwUWAgMBAAIeAQIXgAAKCRD2D0s9f6KvgNArEAChnfcW
rYItgt7xXXubT6E+KpJyJ0RPrXf51S2mhciFbjDl+3EXRMRjOutVmgWYPWUUZaKR
8Iez3Lz4BRmwYOWBLtdnOLbKoSsQUX95rnPFjfly/DFLfjKxz4NRBmh4r4/rCYWm
2hmnXmOAi8kV7fqx3g5XMpJ//N6+T8ctEol2iZ82GrXjadcRWE4rAe7UyuEzJ74y
6ZKIzk5ijdgEKtcaBhzEWvoV5Pr9nkn7ByGsdehKR/gNnjPMYXrklSHGfphJIsS2
S32lMk/kuRjihBcWcYBXIPEQ7CV+PNW2TlkZj/YqTg637sZHwkhcjcNzxeqKvRYG
8V7Ju5hTDxL1UQBmgDS3cRx1lw7tYRG5bS67tbC2dc/CpPkG5agiZ/WyoHQDnn4r
1fRuOFx694QR6+0rAP6171xEEoNAPaH7gdJdhWKiYiJD0T2EEbW7wBUi/EupeKRv
kR12R1jUa1mlpxNtWQxJ7qp98T9+DmkxI1XDmWx0/g4ryuicwLDSqoPgNcRNdSQb
b8YfTqrkqaDdYzwLr/n0YKW3cYIvIeisV0WxRjb6OP7oAlAtaAhImlIc//51qNO7
/WAud6qMtnhFoZayR/BzLKqnCioN5GYr9BAKskpPHe9cDKVS3fg+Qvc1sNJID+jf
k52PqyW24Qsr0A9+5zQyE4tH9dfv120gj9avmg==
=0nKc
-----END PGP PUBLIC KEY BLOCK-----


그리고나서 ppc64le 아키텍처의 Ubuntu 기반 docker base image를 아래와 같이 pull로 당겨옵니다.

[root@ac922 docker]# docker pull ubuntu:18.04

아래와 같이 작은 ubuntu 기반 image가 pull 된 것을 보실 수 있습니다.

[root@ac922 docker]# docker images
REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
ubuntu                              18.04               a7cbdb002963        4 weeks ago         129MB

이게 ppc64le 아키텍처의 image가 맞는지는 아래와 같이 확인하시면 됩니다.

[root@ac922 docker]# docker inspect ubuntu:18.04 | grep -i arch
        "Architecture": "ppc64le",

위에서 받은 base image를 이용하여, cuda10-0 기반의 ubuntu docker image를 만들기 위한 dockerfile을 아래와 같이 작성합니다.

[root@ac922 docker]# cat dockerfile.cuda10-0
FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common gcc g++ zip unzip gzip bzip2 build-essential wget git vim-tiny  libcurl4-openssl-dev libssl-dev libssh2-1-dev
RUN mkdir /tmp/temp
COPY apt.key1 /tmp/temp
RUN apt-key add /tmp/temp/apt.key1
RUN add-apt-repository "deb [arch=ppc64el] https://download.docker.com/linux/ubuntu bionic stable"
COPY cuda-repo-ubuntu1804_10.0.130-1_ppc64el.deb /tmp/temp/
RUN dpkg -i /tmp/temp/cuda-repo-ubuntu1804_10.0.130-1_ppc64el.deb
COPY apt.key2 /tmp/temp
RUN apt-key add /tmp/temp/apt.key2
RUN apt-get update
RUN apt-get install -y `apt-cache pkgnames | grep cuda | grep -v qnx | grep -v armhf | grep -v aarch64 | grep 10-0`
COPY cudnn-10.0-linux-ppc64le-v7.5.0.56.solitairetheme8 /tmp/temp/
COPY nccl_2.4.2-1+cuda10.0_ppc64le.txz /tmp/temp/
LABEL com.nvidia.volumes.needed="nvidia_driver"
LABEL com.nvidia.cuda.version="10.1"
RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf
RUN echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf
RUN tar -xf /tmp/temp/cudnn-10.0-linux-ppc64le-v7.5.0.56.solitairetheme8 -C /usr/local
RUN tar -xf /tmp/temp/nccl_2.4.2-1+cuda10.0_ppc64le.txz -C /usr/local
# set the working directory
ENV LD_LIBRARY_PATH="/usr/local/nvidia/lib64:/usr/local/nvidia/lib:/usr/local/cuda/lib64:/usr/lib:/usr/lib64:/lib:/lib64:/usr/local/lib:/usr/local/lib64"
ENV PATH="/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/bin"
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
ENV NVIDIA_REQUIRE_CUDA "cuda>=10.0"
CMD /bin/bash

위와 같이 작성된 dockerfile을 이용하여 아래 docker build 명령으로 CUDA 10.0이 설치된 docker image를 build합니다.

[root@ac922 docker]# docker build -f dockerfile.cuda10-0 -t bsyu/ubuntu18.04_cuda10-0_ppc64le:v0.2 .

이제 아래와 같이 bsyu/ubuntu18.04_cuda10-0_ppc64le:v0.2 라는 이미지가 build 되었습니다.

[root@ac922 docker]# docker images
REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
bsyu/ubuntu18.04_cuda10-0_ppc64le   v0.2                c723faffd5c5        About an hour ago   5.47GB
ubuntu                              18.04               a7cbdb002963        4 weeks ago         129MB

이 image를 아래와 같이 run 시켜 봅니다. 

[root@ac922 docker]# docker run --runtime=nvidia -ti --rm bsyu/ubuntu18.04_cuda10-0_ppc64le:v0.2 bash

아래와 같이 nvidia-smi 명령을 수행해보면 일단 잘 돌아가는 것을 보실 수 있습니다.

root@19b69f2a09a3:/# nvidia-smi -l 2
Fri Apr 12 12:55:21 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.40.04    Driver Version: 418.40.04    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000004:04:00.0 Off |                    0 |
| N/A   40C    P0    65W / 300W |  31606MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000004:05:00.0 Off |                    0 |
| N/A   44C    P0    71W / 300W |  31606MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000035:03:00.0 Off |                    0 |
| N/A   40C    P0    65W / 300W |  31606MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000035:04:00.0 Off |                    0 |
| N/A   45C    P0    55W / 300W |  31606MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+


하지만 이 경우는 운이 좋은 것에 불과합니다.   Parent OS인 Redhat 7.5 ppc64le에도 nvidia driver 버전이 Ubuntu image속 버전과 동일한 418.40.04 이었기 때문에 이렇게 잘 수행되는 것입니다.  원래, nvidia-docker2에서는 image 속에 nvidia driver가 설치되어 있으면 안 됩니다.

만약 parent OS의 nvidia-driver 버전이 다른 경우, 아래와 같은 error message가 나면서 dicker run이 안 될 것입니다.  아래는 Ubuntu image 속의 driver가 410.104 버전인 경우입니다.

[root@ac922 docker]# docker run --runtime=nvidia -ti --rm bsyu/ubuntu18.04_cuda9-2_ppc64le:v0.1 bash

docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:385: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig --device=all --compute --utility --pid=44664 /var/lib/docker/overlay2/65906de889a32465e87e680b8968834abaa1f4c4069beadab01bb7691f43523f/merged]\\\\nnvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/65906de889a32465e87e680b8968834abaa1f4c4069beadab01bb7691f43523f/merged/usr/bin/nvidia-smi: file exists\\\\n\\\"\"": unknown.

이를 해결하기 위해서는 docker image 속에 CUDA를 설치할 때 함께 설치된 nvidia driver를 제거해줘야 합니다.   먼저, 아래와 같이 nvidia-docker가 아닌 그냥 docker로 수행되도록 --runtime=nvidia 옵션을 빼고 docker를 구동합니다.  그렇게 하면 최소한 위의 'file exists' error는 나지 않을 것입니다.

[root@ac922 docker]# docker run -ti --rm bsyu/ubuntu18.04_cuda9-2_ppc64le:v0.1 bash             

이어서 image 속에 들어가보면 아래와 같은 nvidia driver들이 보일 것입니다. 

root@45fb663f025c:/# dpkg -l | grep nvidia
ii  nvidia-410                      410.104-0ubuntu1                  ppc64el      NVIDIA binary driver - version 410.104
ii  nvidia-410-dev                  410.104-0ubuntu1                  ppc64el      NVIDIA binary Xorg driver development files
ii  nvidia-modprobe                 410.104-0ubuntu1                  ppc64el      Load the NVIDIA kernel driver and create device files
ii  nvidia-opencl-icd-410           410.104-0ubuntu1                  ppc64el      NVIDIA OpenCL ICD
ii  nvidia-prime                    0.8.8.2                           all          Tools to enable NVIDIA's Prime
ii  nvidia-settings                 410.104-0ubuntu1                  ppc64el      Tool for configuring the NVIDIA graphics driver

아래와 같이 이 관련 fileset들을 다 지워버린 뒤, 새로운 tag로 해당 container를 commit 합니다.

root@45fb663f025c:/# apt-get remove -y nvidia-*

Docker image 속의 저 명령이 완료되면 다른 창에서 parent OS에 접속하여 다음고 같이 commit 합니다.

먼저 저 container의 ID를 확인하고...

[root@ac922 docker]# docker ps -a
CONTAINER ID        IMAGE                                   COMMAND             CREATED             STATUS              PORTS               NAMES
45fb663f025c        bsyu/ubuntu18.04_cuda9-2_ppc64le:v0.1   "bash"              42 seconds ago      Up 39 seconds                           elastic_gates

이어서 v0.2라는 새로운 tag로 commit 합니다.

[root@ac922 docker]# docker commit 45fb663f025c bsyu/ubuntu18.04_cuda9-2_ppc64le:v0.2
sha256:172cdbe6ba2f5f21ce8352a7be964d921d0340c82c1d1d91d54bf28d2c5e2c33


아래와 같이 여러가지 image들을 준비하여 public docker hub에 push로 올려 놓았습니다.  아래 이름을 참조하여 필요에 따라 pull 하여 사용하시기 바랍니다.


[root@ac922 docker]# docker images | grep latest
bsyu/ubuntu18.04_cuda9-2_python368_pytorch1.01_ppc64le    latest              b6695ad4cb7f        8 days ago          8.19GB
bsyu/ubuntu18.04_cuda10-0_ppc64le                         latest              a4f8442230bf        8 days ago          5.54GB
bsyu/ubuntu18.04_cuda10-0_python368_ppc64le               latest              817f3211acdb        8 days ago          7.29GB
bsyu/ubuntu18.04_cuda10-0_python352_ppc64le               latest              cadd3fdb9653        8 days ago          8.6GB
bsyu/ubuntu18.04_cuda10-0_python352_tf1.12_ppc64le        latest              929db5ab1260        8 days ago          18.2GB
bsyu/ubuntu18.04_cuda10-0_python368_pytorch1.01_ppc64le   latest              00fb9697002a        8 days ago          9.45GB
bsyu/ubuntu18.04_cuda10-0_python368_tf1.12_ppc64le        latest              cf2342b0c1b8        8 days ago          16.8GB
bsyu/ubuntu18.04_cuda10-0_python352_pytorch1.01_ppc64le   latest              e8d0a0897362        8 days ago          9.24GB
bsyu/ubuntu18.04_cuda9-2_python352_pytorch1.01_ppc64le    latest              9bf2cafb6836        8 days ago          8.92GB
bsyu/ubuntu18.04_cuda9-2_python352_tf1.12_ppc64le         latest              c1d6c56b6f11        8 days ago          17.4GB
bsyu/ubuntu18.04_cuda9-2_python352_ppc64le                latest              79d3a19ae9ad        8 days ago          8.35GB
bsyu/ubuntu18.04_cuda9-2_python368_tf1.12_ppc64le         latest              e80932cafae5        8 days ago          18.5GB
bsyu/ubuntu18.04_cuda9-2_ppc64le                          latest              eeae2d1f6fd4        8 days ago          5.26GB
bsyu/ubuntu18.04_cuda10-0_python368_cudf0.7.0_ppc64le     latest              86930ab12fe2        8 days ago          10.4GB

Ubuntu ppc64le 환경에서 nvidia-docker를 source로부터 build하기

최근에 nvidia-docker를 source에서 build 해보니 기존에 있던 ppc64le branch가 없어져서 그냥 하면 error가 납니다.   물론 nvidia-docker는 다음과 같이 apt repository에서 편리하게 설치가 가능합니다.

$ sudo vi /etc/apt/sources.list.d/nvidia-docker.list
deb https://nvidia.github.io/libnvidia-container/ubuntu16.04/$(ARCH) /
deb https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/$(ARCH) /
deb https://nvidia.github.io/nvidia-docker/ubuntu16.04/$(ARCH) /

$ sudo vi /etc/apt/sources.list.d/docker.list
deb [arch=ppc64el] https://download.docker.com/linux/ubuntu xenial stable


그래도 필요에 의해서 nvidia-docker를 source로부터 build해야 할 경우가 있습니다.   이 경우에도 근성만 있으면 아래와 같이 build가 가능합니다.

minsky_test@minsky:~/files$ git clone https://github.com/NVIDIA/nvidia-docker.git

minsky_test@minsky:~/files$ cd nvidia-docker

minsky_test@minsky:~/files/nvidia-docker$ git fetch --all

minsky_test@minsky:~/files/nvidia-docker$ git checkout tags/v1.0.1

여기서 그냥 make를 하시면 결국 다음과 같은 "exec format error"가 납니다.

minsky_test@minsky:~/files/nvidia-docker$ sudo make
...
http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64 /" > /etc/apt/sources.list.d/cuda.list
 ---> Running in 9bc30333d07a
standard_init_linux.go:178: exec user process caused "exec format error"
...
make: *** [tools] Error 2

이건 기본적으로 이 build를 위해 pull 해오는 golang container image가 ppc64le가 아닌 AMD64 아키텍처의 것이기 때문입니다.

minsky_test@minsky:~/files/nvidia-docker$ sudo docker inspect golang:1.5 | grep Arch
        "Architecture": "amd64",

이 문제와 기타 다른 문제들을 해결하기 위해 아래와 같이 Makefile과 Dockerfile.build을 일부 수정해주면 됩니다.

minsky_test@minsky:~/files/nvidia-docker$ vi Makefile
...
#ifneq ($(MAKECMDGOALS),rpm)
#PKG_ARCH := amd64
#else
#PKG_ARCH := x86_64
#endif
PKG_ARCH := ppc64le
...


minsky_test@minsky:~/files/nvidia-docker$ vi Dockerfile.build
#FROM golang:1.5
FROM ppc64le/golang:1.9
...
#    apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/7fa2af80.pub && \
    apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/ppc64el/7fa2af80.pub && \
...
#    echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64 /" > /etc/apt/sources.list.d/cuda.list
    echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/ppc64el /" > /etc/apt/sources.list.d/cuda.list
...
#        cuda-cudart-dev-6-5=6.5-14 \
        cuda-cudart-dev-8-0=8.0.61-1 \
#        cuda-misc-headers-6-5=6.5-19 && \
        cuda-misc-headers-8-0=8.0.61-1 && \
...
#RUN objcopy --redefine-sym memcpy=memcpy@GLIBC_2.2.5 /usr/local/cuda-6.5/lib64/libcudart_static.a   #으로 comment-out합니다
...
#ENV CGO_CFLAGS "-I /usr/local/cuda-6.5/include -I /usr/include/nvidia/gdk"
ENV CGO_CFLAGS "-I /usr/local/cuda-8.0/include -I /usr/include/nvidia/gdk"
#ENV CGO_LDFLAGS "-L /usr/local/cuda-6.5/lib64"
ENV CGO_LDFLAGS "-L /usr/local/cuda-8.0/lib64"

이제 build를 하면 됩니다.  make tarball을 수행하면 nvidia-docker와 nvidia-docker-plugin binary를 tar.xz로 묶어 줍니다.  이걸 /usr/local 등 적절한 위치에 풀어주기만 하면 됩니다.

minsky_test@minsky:~/files/nvidia-docker$ sudo make tarball
...
Find tarball at /home/minsky_test/files/nvidia-docker/dist

minsky_test@minsky:~/files/nvidia-docker$ tar -tvf /home/minsky_test/files/nvidia-docker/dist/nvidia-docker_1.0.1_ppc64le.tar.xz
-rwxr-xr-x root/miruware 5247984 2018-05-16 16:57 nvidia-docker/nvidia-docker
-rwxr-xr-x root/miruware 6489232 2018-05-16 16:57 nvidia-docker/nvidia-docker-plugin

물론 이렇게 source에서 build한 nvidia-docker는 service로 등록하기 전엔 아래처럼 손으로 start 해주고 kill로 죽여야 합니다.

minsky_test@minsky:~/files/nvidia-docker/bin$ sudo ./nvidia-docker-plugin &

minsky_test@minsky:~/files/nvidia-docker/bin$ sudo ./nvidia-docker run -ti bsyu/tf1.3-ppc64le:v0.1 bash

root@78d0b67e2c8d:/# nvidia-smi
Wed May 16 08:05:31 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.111                Driver Version: 384.111                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-SXM2...  Off  | 00000002:01:00.0 Off |                    0 |
| N/A   35C    P0    31W / 300W |     10MiB / 16276MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P100-SXM2...  Off  | 00000003:01:00.0 Off |                    0 |
| N/A   35C    P0    30W / 300W |     10MiB / 16276MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla P100-SXM2...  Off  | 0000000A:01:00.0 Off |                    0 |
| N/A   35C    P0    32W / 300W |     10MiB / 16276MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla P100-SXM2...  Off  | 0000000B:01:00.0 Off |                    0 |
| N/A   30C    P0    31W / 300W |     10MiB / 16276MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

아래에 이렇게 만들어진 nvidia-docker tarball을 google drive에 올려놓았습니다.

https://drive.google.com/open?id=1XoXXgpdsQPKHo-IPx3iNbKuJYHKgpihz

Ubuntu ppc64le 환경에서의 CUDA 및 CPU 버전의 HPL compile 및 수행

지난번에 올린 posting에서는 HPL (High Performance Linpack)을 POWER9 AC922 Redhat 환경에서 CUDA를 이용하여 수행하는 방법을 정리했지요.  이번에는 같은 HPL CUDA를 POWER8 Minsky Ubuntu 환경에서 수행하는 방법을 정리하고, 이어서 CPU 버전의 평범한 HPL도 수행해보겠습니다.  

먼저 아래와 같이 필요한 package들을 설치합니다.   Redhat에서는 가령 lapack-devel 패키지 등이 기본 DVD에 들어있지 않아서 사용하지 못했는데, Ubuntu에서는 다 제공되므로 편리하게 그것들까지 다 설치할 수 있습니다.

minsky@minsky:~$ sudo apt-get install openmpi-bin openmpi-common libopenmpi-dev mpich libmpich12 libmpich-dev libopenblas-dev libopenblas libopenblas-base libatlas3-base libatlas-base-dev libatlas-test libatlas-dev liblapack3 liblapack-dev liblapack-test liblapacke-dev liblapacke

지난번과 마찬가지로 NVIDIA site에서 (x86_64 버전이긴 하지만) HPL의 CUDA 버전 소스코드를 받습니다.  

https://developer.nvidia.com/rdp/assets/cuda-accelerated-linpack-linux64

위에서 license 등에 동의하면 아래와 같이 hpl-2.0_FERMI_v15.solitairetheme8을 download 받을 수 있습니다.  이는 tar.gz 형태의 파일입니다.

minsky@minsky:~/files$ tar -zxvf hpl-2.0_FERMI_v15.solitairetheme8

minsky@minsky:~/files$ cd hpl-2.0_FERMI_v15

먼저, Intel MKL compiler에 편향된 cuda_dgemm.c의 source를 약간 수정해야 합니다.

minsky@minsky:~/files/hpl-2.0_FERMI_v15$ vi ./src/cuda/cuda_dgemm.c
...
//      handle2 = dlopen ("libmkl_intel_lp64.so", RTLD_LAZY);
      handle2 = dlopen ("libopenblas.so", RTLD_LAZY);
...
//      dgemm_mkl = (void(*)())dlsym(handle, "dgemm");
      dgemm_mkl = (void(*)())dlsym(handle, "dgemm_");
...
//      handle = dlopen ("libmkl_intel_lp64.so", RTLD_LAZY);
      handle = dlopen ("libopenblas.so", RTLD_LAZY);
...
//      mkl_dtrsm = (void(*)())dlsym(handle2, "dtrsm");
      mkl_dtrsm = (void(*)())dlsym(handle2, "dtrsm_");
...

위의 수정들을 하지 않으면 run_linpack 수행시 다음과 같은 runtime error가 납니다.  이는 ppc64le 아키텍처 상에서는 libmkl_intel_lp64 대신 오픈소스인 openblas를 사용하기 때문입니다.

libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
libopenblas.so.0: undefined symbol: dtrsm
libopenblas.so.0: undefined symbol: dgemm

이제 compile을 위해 Make.CUDA를 수정합니다.   ppc64le 아키텍처라고 해서 크게 바뀔 건 없습니다.  Redhat과는 달리 Ubuntu에서는 libatlas-dev와 libmpich-dev 등이 제공되므로, 그냥 libatlas.a와 libmpich.a 등을 쓸 수 있습니다.  -lmkl 대신 -lopenblas를 쓴 것에 주목하십시요.

minsky@minsky:~/files/hpl-2.0_FERMI_v15$ vi Make.CUDA
...
#TOPdir = /home/mfatica/hpl-2.0_FERMI_v15
TOPdir = /home/minsky/files/hpl-2.0_FERMI_v15
...
#MPdir        = /opt/intel/mpi/3.0
#MPinc        = -I$(MPdir)/include64
#MPlib        = $(MPdir)/lib64/libmpi.a
#MPlib        = $(MPdir)/lib64/libmpich.a
MPdir        = /usr/lib/openmpi/lib
MPinc        = -I /usr/lib/openmpi/include
MPlib        = -L /usr/lib/openmpi/lib -lmpi /usr/lib/powerpc64le-linux-gnu/libmpich.a
...
#LAdir        = $(TOPdir)/../../lib/em64t
#LAdir        = /share/apps/intel/mkl/10.2.4.032/libem64t
#LAinc        =
# CUDA
#LAlib        = -L /home/cuda/Fortran_Cuda_Blas  -ldgemm -L/usr/local/cuda/lib -lcublas  -L$(LAdir) -lmkl -lguide -lpthread
#LAlib        = -L $(TOPdir)/src/cuda  -ldgemm -L/usr/local/cuda/lib64 -lcuda -lcudart -lcublas -L$(LAdir) -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5
LAdir        = /usr/lib
LAinc        = -I /usr/include/atlas -I /usr/include/openblas -I /usr/include
#LAlib        = /usr/lib/libatlas.a ${LAdir}/libopenblas.a /usr/lib/atlas-base/atlas/libblas.a /usr/lib/atlas-base/atlas/liblapack.a
LAlib        = -L $(TOPdir)/src/cuda -ldgemm -L /usr/local/cuda/targets/ppc64le-linux/lib/stubs -lcuda -lcublas -L /usr/local/cuda/lib64 -lcudart -L$(LAdir) -lpthread -lm /usr/lib/libatlas.a /usr/lib/atlas-base/atlas/liblapack.a /usr/lib/atlas-base/atlas/libblas.a ${LAdir}/libopenblas.a /usr/lib/gcc/powerpc64le-linux-gnu/5/libgfortran.a
...

이제 아래와 같이 환경변수를 맞춰주고, make arch=CUDA를 수행하면 일사천리로 compile이 수행됩니다.

minsky@minsky:~/files/hpl-2.0_FERMI_v15$ export LD_LIBRARY_PATH=/usr/lib/openmpi/lib:/usr/lib/mpich/lib:$LD_LIBRARY_PATH

minsky@minsky:~/files/hpl-2.0_FERMI_v15$ make arch=CUDA
...
mpicc -o HPL_pdtest.o -c -DAdd__ -DF77_INTEGER=int -DStringSunStyle -DCUDA -I/home/minsky/files/hpl-2.0_FERMI_v15/include -I/home/minsky/files/hpl-2.0_FERMI_v15/include/CUDA -I /usr/include/atlas -I /usr/include/openblas -I /usr/include -I /usr/include/openmpi-ppc64le -I/usr/local/cuda/include -fomit-frame-pointer -O3 -funroll-loops -W -Wall -fopenmp  ../HPL_pdtest.c
mpicc -DAdd__ -DF77_INTEGER=int -DStringSunStyle -DCUDA -I/home/minsky/files/hpl-2.0_FERMI_v15/include -I/home/minsky/files/hpl-2.0_FERMI_v15/include/CUDA -I /usr/include/atlas -I /usr/include/openblas -I /usr/include -I /usr/include/openmpi-ppc64le -I/usr/local/cuda/include -fomit-frame-pointer -O3 -funroll-loops -W -Wall -fopenmp  -o /home/minsky/files/hpl-2.0_FERMI_v15/bin/CUDA/xhpl HPL_pddriver.o         HPL_pdinfo.o           HPL_pdtest.o /home/minsky/files/hpl-2.0_FERMI_v15/lib/CUDA/libhpl.a  -L /home/minsky/files/hpl-2.0_FERMI_v15/src/cuda -ldgemm -L /usr/local/cuda/targets/ppc64le-linux/lib/stubs -lcuda -lcublas -L /usr/local/cuda/lib64 -lcudart -L/usr/lib -lpthread -lm /usr/lib/libatlas.a /usr/lib/atlas-base/atlas/liblapack.a /usr/lib/atlas-base/atlas/libblas.a /usr/lib/libopenblas.a /usr/lib/gcc/powerpc64le-linux-gnu/5/libgfortran.a -L /usr/lib/openmpi/lib -lmpi /usr/lib/powerpc64le-linux-gnu/libmpich.a
make TOPdir=/home/minsky/files/hpl-2.0_FERMI_v15 /home/minsky/files/hpl-2.0_FERMI_v15/bin/CUDA/HPL.dat
make[3]: Entering directory '/home/minsky/files/hpl-2.0_FERMI_v15/testing/ptest/CUDA'
make[3]: '/home/minsky/files/hpl-2.0_FERMI_v15/bin/CUDA/HPL.dat' is up to date.
make[3]: Leaving directory '/home/minsky/files/hpl-2.0_FERMI_v15/testing/ptest/CUDA'
touch dexe.grd
make[2]: Leaving directory '/home/minsky/files/hpl-2.0_FERMI_v15/testing/ptest/CUDA'
make[1]: Leaving directory '/home/minsky/files/hpl-2.0_FERMI_v15'

실행 파일은 아래와 같이 bin/CUDA 밑에 xhpl이라는 이름으로 만들어져 있습니다.  

minsky@minsky:~/files/hpl-2.0_FERMI_v15$ cd bin/CUDA

minsky@minsky:~/files/hpl-2.0_FERMI_v15/bin/CUDA$ vi run_linpack
...
#HPL_DIR=/home/mfatica/hpl-2.0_FERMI_v15
HPL_DIR=/home/minsky/files/hpl-2.0_FERMI_v15
...
#CPU_CORES_PER_GPU=4
CPU_CORES_PER_GPU=8
...
#export CUDA_DGEMM_SPLIT=0.80
export CUDA_DGEMM_SPLIT=0.99
...
#export CUDA_DTRSM_SPLIT=0.70
export CUDA_DTRSM_SPLIT=0.99

그리고 input 파일이라고 할 수 있는 HPL.dat 파일을 수정해야 합니다.  이에 대해서는 아래 URL을 참조하여 수정합니다.

http://www.netlib.org/benchmark/hpl/tuning.html

HPL.dat의 주요 input 항목의 의미에 대해서는 아래 URL을 참조하시면 됩니다.

http://www.netlib.org/benchmark/hpl/tuning.html

여기서 중요한 것은 problem size(Ns)를 얼마로 두느냐와 이걸 어떤 process grid(P x Q)에 어떤 block size (NBs)로 태우느냐입니다.

problem size(Ns)를 구하는 방법은 대략 다음과 같습니다.   여기서는 16GB memory가 장착된 P100 GPU 4장이 장착되어 있으니 다음과 같이 하면 됩니다.

sqrt(GPU mem size * # of GPUs * 적정 mem% / double precision 64-bit in byte)
sqrt(64 * 1024^3 * 4 * 0.8 / 8) = 15852

process grid(P x Q)는 Minsky에 장착된 GPU 개수에 맞추면 됩니다.  2 x 2 =4로 하든, 1 x 4 =4로 하든, 또는 둘 다 수행하든 택하면 됩니다.   실제로 해보면  1 x 4로 하는 것이 성능은 좀더 잘 나오는데, 대신 큰 Ns를 사용하는 경우 검증 과정에서 fail 나는 경우가 종종 있습니다.  여기서는 그냥 flat grid인 1 x 4로 하겠습니다.  

Process grid에 어떤 Block size(NBs)로 태울 것인가 하는 것은, CPU인 경우 32 ~ 256 정도에서 택하되 CUDA인 경우 1000 단위로 크게 하라는데, 2048보다는 1024가 더 나은 것 같습니다.

minsky@minsky:~/files/hpl-2.0_FERMI_v15/bin/CUDA$ vi HPL.dat
HPLinpack benchmark input file
Innovative Computing Laboratory, University of Tennessee
HPL.out      output file name (if any)
6            device out (6=stdout,7=stderr,file)
1            # of problems sizes (N)
120000       Ns
1            # of NBs
1024         NBs
0            PMAP process mapping (0=Row-,1=Column-major)
1            # of process grids (P x Q)
1          Ps
4          Qs
16.0         threshold
1            # of panel fact
0            PFACTs (0=left, 1=Crout, 2=Right)
1            # of recursive stopping criterium
2            NBMINs (>= 1)
1            # of panels in recursion
2            NDIVs
1            # of recursive panel fact.
2            RFACTs (0=left, 1=Crout, 2=Right)
1            # of broadcast
3            BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)
1            # of lookahead depth
0            DEPTHs (>=0)
1            SWAP (0=bin-exch,1=long,2=mix)
64           swapping threshold
1            L1 in (0=transposed,1=no-transposed) form
0            U  in (0=transposed,1=no-transposed) form
1            Equilibration (0=no,1=yes)
32           memory alignment in double (> 0)

이제 수행하면 됩니다.  PxQ = 4이므로 여기서는 mpirun을 이용하여 4개를 수행합니다.

minsky@minsky:~/files/hpl-2.0_FERMI_v15/bin/CUDA$ nohup time mpirun -np 4 ./run_linpack > linpack_out1.txt &

결과가 궁금하실텐데, 여기서 제가 개발새발 수행한 것을 공개하는 것은 곤란하군요.  다만, 이 hpl-2.0_FERMI_v15로 구현된 것은 2011년 정도에 당시 GPU에 맞춰서 HPL을 CUDA로 변환한 것이라서, 현대적인 P100이나 V100 GPU에서는 제 성능을 내지 못 합니다.  (https://devtalk.nvidia.com/default/topic/991058/cuda-programming-and-performance/poor-results-from-cuda-linpack-on-k80/post/5074677/ 참조)  신규 GPU에 맞춰 NVIDIA가 작성한 HPL-CUDA가 있을텐데, 그건 일반 공개되지는 않는다고 합니다.  실제로 제가 돌려본 결과도 이론치(Rpeak)에 훨씬 미치지 못 합니다.

지난번 AC922 Redhat 환경에서와는 다른 점이 있습니다.   여기서도 대부분의 구간에서 기본적으로 np 당 1개씩의 CPU core만 100% 쓰지만, 일부 구간에서는 각각의 xhpl process가 multi-thread 형태로 여러 개의 CPU core들을 사용합니다.  

lnmonq14gqqqqqq[H for help]qqqHostname=minskyqqqqqqqRefresh= 2secs qqq00:07.57qq
x CPU Utilisation qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
x---------------------------+-------------------------------------------------+
xCPU  User%  Sys% Wait% Idle|0          |25         |50          |75       100|
x  1  50.2   9.9   0.0  39.9|UUUUUUUUUUUUUUUUUUUUUUUUUssss                    >
x  2  51.7  10.3   0.0  37.9|UUUUUUUUUUUUUUUUUUUUUUUUUsssss                   >
x  3  46.8   2.0   0.0  51.2|UUUUUUUUUUUUUUUUUUUUUUU                          >
x  4  60.3  13.2   0.0  26.5|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUssssss             >
x  5  42.0  11.9   0.0  46.1|UUUUUUUUUUUUUUUUUUUUsssss                        >
x  6  51.3   9.1   0.0  39.6|UUUUUUUUUUUUUUUUUUUUUUUUUssss                  > |
x  7  83.3   3.9   0.0  12.7|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUs       >
x  8  39.9   5.9   0.0  54.2|UUUUUUUUUUUUUUUUUUUss                            >
x  9  47.3  11.3   0.0  41.4|UUUUUUUUUUUUUUUUUUUUUUUsssss                     >
x 10  48.5  10.4   0.0  41.1|UUUUUUUUUUUUUUUUUUUUUUUUsssss                    >
x 11  43.8   9.9   0.0  46.3|UUUUUUUUUUUUUUUUUUUUUssss                        >
x 12  53.9   8.3   0.5  37.3|UUUUUUUUUUUUUUUUUUUUUUUUUUssss                   >
x 13  95.1   2.9   0.0   2.0|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUs >
x 14  35.0   2.5   0.0  62.6|UUUUUUUUUUUUUUUUUs                               >
x 15  51.5   6.9   0.0  41.6|UUUUUUUUUUUUUUUUUUUUUUUUUsss                     >
x 16  50.2   9.4   0.0  40.4|UUUUUUUUUUUUUUUUUUUUUUUUUssss                    >
x 17  49.3   8.9   0.0  41.9|UUUUUUUUUUUUUUUUUUUUUUUUssss                     >
x 18  49.8  11.8   0.0  38.4|UUUUUUUUUUUUUUUUUUUUUUUUsssss                    >
x 19  46.3   7.9   0.0  45.8|UUUUUUUUUUUUUUUUUUUUUUUsss                       >
x 20  68.6   9.8   0.0  21.6|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUssss           >
x 21  37.3   8.3   0.0  54.4|UUUUUUUUUUUUUUUUUUssss                           >
x 22  48.5   7.9   0.0  43.6|UUUUUUUUUUUUUUUUUUUUUUUUsss                      >
x 23  50.7  12.8   0.0  36.5|UUUUUUUUUUUUUUUUUUUUUUUUUssssss                  >
x 24  44.1   8.3   0.0  47.5|UUUUUUUUUUUUUUUUUUUUUUssss                       >
x 25  49.8   8.4   0.0  41.9|UUUUUUUUUUUUUUUUUUUUUUUUssss                     >
x 26  43.8   7.9   0.0  48.3|UUUUUUUUUUUUUUUUUUUUUsss                         >
x 27  49.0   4.9   0.0  46.1|UUUUUUUUUUUUUUUUUUUUUUUUss                       >
x 28  97.5   1.5   0.0   1.0|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU >
x 29  75.4   1.5   0.0  23.2|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU            >
x 30  52.0   9.3   0.0  38.7|UUUUUUUUUUUUUUUUUUUUUUUUUssss                    >
x 31  41.9   6.4   0.0  51.7|UUUUUUUUUUUUUUUUUUUUsss                          >
x 32  47.2  10.1   0.0  42.7|UUUUUUUUUUUUUUUUUUUUUUUsssss                     >
x---------------------------+-------------------------------------------------+
xAvg  53.2   8.0   0.0  38.8|UUUUUUUUUUUUUUUUUUUUUUUUUUsss                    >
x---------------------------+-------------------------------------------------+
x Top Processes Procs=1090 mode=3 (1=Basic, 3=Perf 4=Size 5=I/O)qqqqqqqqqqqqqqqq
x  PID    %CPU  Size   Res   Res   Res   Res Shared   Faults Command
x          Used    KB   Set  Text  Data   Lib    KB  Min  Maj
x   16175 517.6 52070656 11392128   512 51713408     0 142080   15    0 xhpl   x
x   16178 493.8 51681984 10993664   512 51324736     0 142144   44    0 xhpl   x
x   16179 485.6 51503232 10828096   512 51145984     0 142144   97    0 xhpl   x
x   16177 450.7 52079424 11429888   512 51722176     0 142080   46    0 xhpl   x


lnmonq14gqqqqqqqqqqqqqqqqqqqqqHostname=minskyqqqqqqqRefresh= 2secs qqq03:00.56qk
x CPU +------------------------------------------------------------------------x
x100%-|                            |                                           x
x 95%-|                            |                                           x
x 90%-|                            |                                           x
x 85%-|                            |                                           x
x 80%-|                            |                                           x
x 75%-|                            |                                           x
x 70%-|                            |                                           x
x 65%-|ssss ss  s                  |                                           x
x 60%-|UUssssss ss ss              |                                           x
x 55%-|UUUUUUUssUssss              |                                           x
x 50%-|UUUUUUUUUUssUU              |                                           x
x 45%-|UUUUUUUUUUUUUU              |                                           x
x 40%-|UUUUUUUUUUUUUU              |                                           x
x 35%-|UUUUUUUUUUUUUU              +                                           x
x 30%-|UUUUUUUUUUUUUU              |                                           x
x 25%-|UUUUUUUUUUUUUU              |                                           x
x 20%-|UUUUUUUUUUUUUU              |                                           x
x 15%-|UUUUUUUUUUUUUUUUUUUUUUUUUUUU|                                           x
x 10%-|UUUUUUUUUUUUUUUUUUUUUUUUUUUU|                                           x
x  5%-|UUUUUUUUUUUUUUUUUUUUUUUUUUUU|                                           x
x     +--------------------User---------System---------Wait--------------------x
x Top Processes Procs=1098 mode=3 (1=Basic, 3=Perf 4=Size 5=I/O)qqqqqqqqqqqqqqqx
x  PID    %CPU  Size   Res   Res   Res   Res Shared   Faults Command           x
x          Used    KB   Set  Text  Data   Lib    KB  Min  Maj                  x
x   16175 100.4 51502720 10825152   512 51145472     0 142144   17    0 xhpl   x
x   16177 100.4 51503296 10853824   512 51146048     0 142144   17    0 xhpl   x
x   16179 100.4 50927104 10252096   512 50569856     0 142272   17    0 xhpl   x
x   16178  99.9 51105856 10417600   512 50748608     0 142208   17    0 xhpl   x
x   12926   2.5 11520  8448   192  8128     0  2432    0    0 nmon             x
x   14666   2.5  5696  4608   576   640     0  3584    0    0 nvidia-smi       x
x    3056   0.5 2413632 53440 35776 2366976     0 31296    0    0 dockerd      x
x       1   0.0 10944  9792  1728  2304     0  5376    0    0 systemd          x
x       2   0.0     0     0     0     0     0     0    0    0 kthreadd   


그리고 CPU core의 병목이 이렇게 해소되니, GPU 사용률도 계속 100%를 쓰는 것까지는 아니지만 지난번 Redhat에서보다는 사용률이 훨씬 높습니다.  Ns를 얼마로 주든 간에 GPU memory usage는 아래처럼 언제나 2365MiB로 나옵니다.  대신 Ns를 큰 값으로 주면, 위에서 보시는 바와 같이 xhpl 프로세스가 차지하는 서버 메모리가 크게 나옵니다.


##################################


이어서 CPU만 이용하는 평범한 HPL을 수행해보겠습니다.   이는 아래 site에서 받으실 수 있습니다.

minsky@minsky:~/files$  wget http://www.netlib.org/benchmark/hpl/hpl-2.2.tar.gz

minsky@minsky:~/files$ tar -zxf hpl-2.2.tar.gz

minsky@minsky:~/files$ cd hpl-2.2

Make.{arch} 파일의 sample은 setup directory 밑에 있는 것 중에서 골라 사용하면 되는데, ppc64le에서는 가장 비슷해 보이는 Make.Linux_PII_CBLAS를 아래와 같이 복사해서 편집한 뒤 사용하시면 됩니다.

minsky@minsky:~/files/hpl-2.2$ cp setup/Make.Linux_PII_CBLAS Make.Linux

minsky@minsky:~/files/hpl-2.2$ vi Make.Linux
...
#ARCH         = Linux_PII_CBLAS
ARCH         = Linux
...
#TOPdir       = $(HOME)/hpl
TOPdir       = $(HOME)/files/hpl-2.2
...
#MPdir        = /usr/local/mpi
#MPinc        = -I$(MPdir)/include
#MPlib        = $(MPdir)/lib/libmpich.a
MPdir        = /usr/lib/openmpi/lib
MPinc        = -I /usr/lib/openmpi/include
MPlib        = -L /usr/lib/openmpi/lib -lmpi /usr/lib/powerpc64le-linux-gnu/libmpich.a
...
#LAdir        = $(HOME)/netlib/ARCHIVES/Linux_PII
#LAinc        =
#LAlib        = $(LAdir)/libcblas.a $(LAdir)/libatlas.a
LAdir        = /usr/lib
LAinc        = -I /usr/include/atlas -I /usr/include/openblas -I /usr/include
LAlib        = -L$(LAdir) -lpthread -lm /usr/lib/libatlas.a /usr/lib/atlas-base/atlas/liblapack.a /usr/lib/atlas-base/atlas/libblas.a ${LAdir}/libopenblas.a /usr/lib/gcc/powerpc64le-linux-gnu/5/libgfortran.a
...
#CC           = /usr/bin/gcc
CC           = /usr/bin/mpicc
...
#LINKER       = /usr/bin/g77
LINKER       = /usr/bin/mpicc
...

이제 다음과 같이 make를 수행하시어 compile 하시면 역시 그 결과물인 xhpl 파일이 bin/Linux 밑에 생성됩니다.

minsky@minsky:~/files/hpl-2.2$ make arch=Linux

minsky@minsky:~/files/hpl-2.2$ cd bin/Linux

minsky@minsky:~/files/hpl-2.2/bin/Linux$ vi run_linpack
export HPL_DIR=/home/minsky/files/hpl-2.2
# FOR OMP
export OMP_NUM_THREADS=8
export LD_LIBRARY_PATH=$HPL_DIR/lib/Linux:$LD_LIBRARY_PATH
$HPL_DIR/bin/Linux/xhpl

minsky@minsky:~/files/hpl-2.2/bin/Linux$ chmod a+x run_linpack

HPL.dat은 역시 비슷한 방법으로 작성하면 됩니다.  다만 여기서는 RAM이 512GB니까 훨씬 더 큰 Ns 값을 줄 수 있습니다.  그리고 core 수도 물리적 core 수인 16 또는 SMT-8에 의한 값인 16 * 8 = 128을 줘 볼 수도 있습니다.

minsky@minsky:~/files/hpl-2.2/bin/Linux$ cat HPL.dat
HPLinpack benchmark input file
Innovative Computing Laboratory, University of Tennessee
HPL.out      output file name (if any)
6            device out (6=stdout,7=stderr,file)
1            # of problems sizes (N)
234000  Ns
1            # of NBs
128      NBs
0            PMAP process mapping (0=Row-,1=Column-major)
1            # of process grids (P x Q)
1        Ps
16       Qs
16.0         threshold
3            # of panel fact
0 1 2        PFACTs (0=left, 1=Crout, 2=Right)
2            # of recursive stopping criterium
2 4          NBMINs (>= 1)
1            # of panels in recursion
2            NDIVs
3            # of recursive panel fact.
0 1 2        RFACTs (0=left, 1=Crout, 2=Right)
1            # of broadcast
0            BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)
1            # of lookahead depth
0            DEPTHs (>=0)
2            SWAP (0=bin-exch,1=long,2=mix)
64           swapping threshold
0            L1 in (0=transposed,1=no-transposed) form
0            U  in (0=transposed,1=no-transposed) form
1            Equilibration (0=no,1=yes)
8            memory alignment in double (> 0)

다음과 같이 수행하면 전체 16개 core를 다 사용합니다.  여기서는 SMT를 off 시켜 놓은 상황입니다.

minsky@minsky:~/files/hpl-2.2/bin/Linux$ export HPL_DIR=/home/minsky/files/hpl-2.2
minsky@minsky:~/files/hpl-2.2/bin/Linux$ export OMP_NUM_THREADS=8
minsky@minsky:~/files/hpl-2.2/bin/Linux$ export LD_LIBRARY_PATH=$HPL_DIR/lib/Linux:$LD_LIBRARY_PATH

minsky@minsky:~/files/hpl-2.2/bin/Linux$ nohup mpirun -np 16 -x HPL_DIR -x OMP_NUM_THREADS -x LD_LIBRARY_PATH ./xhpl > linpack3.txt &



lnmonq14gqqqqqq[H for help]qqqHostname=minskyqqqqqqqRefresh= 2secs qqq00:26.53qk
x CPU +------------------------------------------------------------------------x
x100%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 95%-|UUUUUUUUUUUUUUUUUUUUUUUU+UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 90%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 85%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 80%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 75%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 70%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 65%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 60%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 55%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 50%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 45%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 40%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 35%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 30%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 25%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 20%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 15%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x 10%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x  5%-|UUUUUUUUUUUUUUUUUUUUUUUU|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUx
x     +--------------------User---------System---------Wait--------------------x
x Memory Stats qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqx
x                RAM     High      Low     Swap    Page Size=64 KB             x
x Total MB    523135.2     -0.0     -0.0  44075.8                              x
x Free  MB     60951.1     -0.0     -0.0  44075.8                              x
x Free Percent    11.7%   100.0%   100.0%   100.0%                             x
x             MB                  MB                  MB                       x
x                      Cached= 15197.8     Active=453282.8                     x
x Buffers=   367.3 Swapcached=     0.0  Inactive =  5598.4                     x
x Dirty  =     0.1 Writeback =     0.0  Mapped   =   142.3                     x
x Slab   =  1392.9 Commit_AS =445432.4 PageTables=   114.8                     x
x Top Processes Procs=1091 mode=3 (1=Basic, 3=Perf 4=Size 5=I/O)qqqqqqqqqqqqqqqx
x  PID    %CPU  Size   Res   Res   Res   Res Shared   Faults Command           x
x          Used    KB   Set  Text  Data   Lib    KB  Min  Maj                  x
x   74578 100.1 28825024 28579392  1664 28576704     0 14016   25    0 xhpl    x
x   74579 100.1 28825792 28579520  1664 28577472     0 14208   32    0 xhpl    x
x   74580 100.1 28825792 28579392  1664 28577472     0 14016   32    0 xhpl    x
x   74581 100.1 28351360 28107904  1664 28103040     0 14144   31    0 xhpl    x
x   74582 100.1 28584768 28339520  1664 28336448     0 14208   26    0 xhpl    x
x   74583 100.1 28585792 28339328  1664 28337472     0 14016   41    0 xhpl    x
x   74584 100.1 28584768 28339520  1664 28336448     0 14208   34    0 xhpl    x
x   74589 100.1 28584768 28339328  1664 28336448     0 14016   32    0 xhpl    x


아마 CUDA 버전의 HPL과 CPU 버전의 HPL의 성능 차이가 궁금하실 것입니다.   구체적인 수치를 공개하기는 그렇습니다만, 아무리 최신 GPU에 최적화되지 않은 버전의 HPL-CUDA라고 해도, 확실히 서버 전체의 CPU와 메모리를 100% 쓰는 경우보다는 훠얼~씬 빠릅니다.