ppc64le (IBM POWER9) nvidia-docker2 환경에서 Ubuntu 기반의 docker image 만들기

Redhat 7.5 ppc64le (IBM POWER9) 위에 설치된 nvidia-docker2 환경에서 Ubuntu 기반의 docker image를 만드는 과정을 정리했습니다. 

먼저, 아래 posting 참조해서 Redhat 7.5 ppc64le에 nvidia-docker2 (정확하게는 nvidia-docker2가 아니라 nvidia-runtime)을 설치합니다.

http://hwengineer.blogspot.com/2019/04/ppc64le-redhat-7-nvidia-docker2-nvidia.html

아래와 같이 ubuntu 관련 apt key값들을 받아둡니다.  이것들은 나중에 ubuntu 기반의 docker image를 만들 때 사용됩니다.

[root@ac922 tmp]# mkdir docker

[root@ac922 tmp]# cd docker

[root@ac922 docker]# curl -fsSL https://download.docker.com/linux/ubuntu/gpg > apt.key1

[root@ac922 docker]# curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/7fa2af80.pub > apt.key2

참고로, 이 key 파일들은 아래와 같이 생긴 text file들입니다.

[root@ac922 docker]# cat apt.key1
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBFit2ioBEADhWpZ8/wvZ6hUTiXOwQHXMAlaFHcPH9hAtr4F1y2+OYdbtMuth
lqqwp028AqyY+PRfVMtSYMbjuQuu5byyKR01BbqYhuS3jtqQmljZ/bJvXqnmiVXh
38UuLa+z077PxyxQhu5BbqntTPQMfiyqEiU+BKbq2WmANUKQf+1AmZY/IruOXbnq
L4C1+gJ8vfmXQt99npCaxEjaNRVYfOS8QcixNzHUYnb6emjlANyEVlZzeqo7XKl7
UrwV5inawTSzWNvtjEjj4nJL8NsLwscpLPQUhTQ+7BbQXAwAmeHCUTQIvvWXqw0N
cmhh4HgeQscQHYgOJjjDVfoY5MucvglbIgCqfzAHW9jxmRL4qbMZj+b1XoePEtht
ku4bIQN1X5P07fNWzlgaRL5Z4POXDDZTlIQ/El58j9kp4bnWRCJW0lya+f8ocodo
vZZ+Doi+fy4D5ZGrL4XEcIQP/Lv5uFyf+kQtl/94VFYVJOleAv8W92KdgDkhTcTD
G7c0tIkVEKNUq48b3aQ64NOZQW7fVjfoKwEZdOqPE72Pa45jrZzvUFxSpdiNk2tZ
XYukHjlxxEgBdC/J3cMMNRE1F4NCA3ApfV1Y7/hTeOnmDuDYwr9/obA8t016Yljj
q5rdkywPf4JF8mXUW5eCN1vAFHxeg9ZWemhBtQmGxXnw9M+z6hWwc6ahmwARAQAB
tCtEb2NrZXIgUmVsZWFzZSAoQ0UgZGViKSA8ZG9ja2VyQGRvY2tlci5jb20+iQI3
BBMBCgAhBQJYrefAAhsvBQsJCAcDBRUKCQgLBRYCAwEAAh4BAheAAAoJEI2BgDwO
v82IsskP/iQZo68flDQmNvn8X5XTd6RRaUH33kXYXquT6NkHJciS7E2gTJmqvMqd
tI4mNYHCSEYxI5qrcYV5YqX9P6+Ko+vozo4nseUQLPH/ATQ4qL0Zok+1jkag3Lgk
jonyUf9bwtWxFp05HC3GMHPhhcUSexCxQLQvnFWXD2sWLKivHp2fT8QbRGeZ+d3m
6fqcd5Fu7pxsqm0EUDK5NL+nPIgYhN+auTrhgzhK1CShfGccM/wfRlei9Utz6p9P
XRKIlWnXtT4qNGZNTN0tR+NLG/6Bqd8OYBaFAUcue/w1VW6JQ2VGYZHnZu9S8LMc
FYBa5Ig9PxwGQOgq6RDKDbV+PqTQT5EFMeR1mrjckk4DQJjbxeMZbiNMG5kGECA8
g383P3elhn03WGbEEa4MNc3Z4+7c236QI3xWJfNPdUbXRaAwhy/6rTSFbzwKB0Jm
ebwzQfwjQY6f55MiI/RqDCyuPj3r3jyVRkK86pQKBAJwFHyqj9KaKXMZjfVnowLh
9svIGfNbGHpucATqREvUHuQbNnqkCx8VVhtYkhDb9fEP2xBu5VvHbR+3nfVhMut5
G34Ct5RS7Jt6LIfFdtcn8CaSas/l1HbiGeRgc70X/9aYx/V/CEJv0lIe8gP6uDoW
FPIZ7d6vH+Vro6xuWEGiuMaiznap2KhZmpkgfupyFmplh0s6knymuQINBFit2ioB
EADneL9S9m4vhU3blaRjVUUyJ7b/qTjcSylvCH5XUE6R2k+ckEZjfAMZPLpO+/tF
M2JIJMD4SifKuS3xck9KtZGCufGmcwiLQRzeHF7vJUKrLD5RTkNi23ydvWZgPjtx
Q+DTT1Zcn7BrQFY6FgnRoUVIxwtdw1bMY/89rsFgS5wwuMESd3Q2RYgb7EOFOpnu
w6da7WakWf4IhnF5nsNYGDVaIHzpiqCl+uTbf1epCjrOlIzkZ3Z3Yk5CM/TiFzPk
z2lLz89cpD8U+NtCsfagWWfjd2U3jDapgH+7nQnCEWpROtzaKHG6lA3pXdix5zG8
eRc6/0IbUSWvfjKxLLPfNeCS2pCL3IeEI5nothEEYdQH6szpLog79xB9dVnJyKJb
VfxXnseoYqVrRz2VVbUI5Blwm6B40E3eGVfUQWiux54DspyVMMk41Mx7QJ3iynIa
1N4ZAqVMAEruyXTRTxc9XW0tYhDMA/1GYvz0EmFpm8LzTHA6sFVtPm/ZlNCX6P1X
zJwrv7DSQKD6GGlBQUX+OeEJ8tTkkf8QTJSPUdh8P8YxDFS5EOGAvhhpMBYD42kQ
pqXjEC+XcycTvGI7impgv9PDY1RCC1zkBjKPa120rNhv/hkVk/YhuGoajoHyy4h7
ZQopdcMtpN2dgmhEegny9JCSwxfQmQ0zK0g7m6SHiKMwjwARAQABiQQ+BBgBCAAJ
BQJYrdoqAhsCAikJEI2BgDwOv82IwV0gBBkBCAAGBQJYrdoqAAoJEH6gqcPyc/zY
1WAP/2wJ+R0gE6qsce3rjaIz58PJmc8goKrir5hnElWhPgbq7cYIsW5qiFyLhkdp
YcMmhD9mRiPpQn6Ya2w3e3B8zfIVKipbMBnke/ytZ9M7qHmDCcjoiSmwEXN3wKYI
mD9VHONsl/CG1rU9Isw1jtB5g1YxuBA7M/m36XN6x2u+NtNMDB9P56yc4gfsZVES
KA9v+yY2/l45L8d/WUkUi0YXomn6hyBGI7JrBLq0CX37GEYP6O9rrKipfz73XfO7
JIGzOKZlljb/D9RX/g7nRbCn+3EtH7xnk+TK/50euEKw8SMUg147sJTcpQmv6UzZ
cM4JgL0HbHVCojV4C/plELwMddALOFeYQzTif6sMRPf+3DSj8frbInjChC3yOLy0
6br92KFom17EIj2CAcoeq7UPhi2oouYBwPxh5ytdehJkoo+sN7RIWua6P2WSmon5
U888cSylXC0+ADFdgLX9K2zrDVYUG1vo8CX0vzxFBaHwN6Px26fhIT1/hYUHQR1z
VfNDcyQmXqkOnZvvoMfz/Q0s9BhFJ/zU6AgQbIZE/hm1spsfgvtsD1frZfygXJ9f
irP+MSAI80xHSf91qSRZOj4Pl3ZJNbq4yYxv0b1pkMqeGdjdCYhLU+LZ4wbQmpCk
SVe2prlLureigXtmZfkqevRz7FrIZiu9ky8wnCAPwC7/zmS18rgP/17bOtL4/iIz
QhxAAoAMWVrGyJivSkjhSGx1uCojsWfsTAm11P7jsruIL61ZzMUVE2aM3Pmj5G+W
9AcZ58Em+1WsVnAXdUR//bMmhyr8wL/G1YO1V3JEJTRdxsSxdYa4deGBBY/Adpsw
24jxhOJR+lsJpqIUeb999+R8euDhRHG9eFO7DRu6weatUJ6suupoDTRWtr/4yGqe
dKxV3qQhNLSnaAzqW/1nA3iUB4k7kCaKZxhdhDbClf9P37qaRW467BLCVO/coL3y
Vm50dwdrNtKpMBh3ZpbB1uJvgi9mXtyBOMJ3v8RZeDzFiG8HdCtg9RvIt/AIFoHR
H3S+U79NT6i0KPzLImDfs8T7RlpyuMc4Ufs8ggyg9v3Ae6cN3eQyxcK3w0cbBwsh
/nQNfsA6uu+9H7NhbehBMhYnpNZyrHzCmzyXkauwRAqoCbGCNykTRwsur9gS41TQ
M8ssD1jFheOJf3hODnkKU+HKjvMROl1DK7zdmLdNzA1cvtZH/nCC9KPj1z8QC47S
xx+dTZSx4ONAhwbS/LN3PoKtn8LPjY9NP9uDWI+TWYquS2U+KHDrBDlsgozDbs/O
jCxcpDzNmXpWQHEtHU7649OXHP7UeNST1mCUCH5qdank0V1iejF6/CfTFU4MfcrG
YT90qFF93M3v01BbxP+EIY2/9tiIPbrd
=0YYh
-----END PGP PUBLIC KEY BLOCK-----

[root@ac922 docker]# cat apt.key2
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1

mQINBFdtt4UBEAC8FDSWMR07GJZ265giLn7kLF+EsJCWESUq6Cd13QN0JQ/tLibi
QlW4ZjeOnEH9VPlqh/mKqNMG4SwRt8S+GHpePMQrr0aOkiRGfCclnAWIZURSAP+t
PLelCt43fkw1BBTopd/0oOzO8kHu8j8WU4A8GHxqghfFWPv54FQs2iaZ2eWR7a6d
79IJrbDKaVCCiQrkhCM8m648pNKHhuoJ9cQXFV+uvwkpfmKWGQ4ultxlOyjLHJLF
vuML2RuAO9IxbdZjzeYNN+T+wjFIBVcPnwEO+WrYgvGkT4r9aqVqTeg3EPb7QclV
sKBVJdxk4jZl0y22HAWqScVi6SJ15uK9pXxywDZkbpuRBWx4ThWiGe/FiUa2igi9
/SIvqN2TBY0g18sRTrylVr1wE1UGa/y7nDx6PoGCP1frBt8YUYt3pkM8Xvb2CRxx
CyWwmuFEQHC6jCEWf7FnoBHBYQwTVGNrU0vkuIeDrm+ZAcv8wx+ie1hlFhqCCJnf
jqeQ0/zA9RPmCPOkLyTdSsNZtlxxk7bzCdTdFFKzBjGTR7Gz3SMSp23d11eIyRiF
HQsp2v0SvnPJ6OcgB95Hmo544vi3RuoVfovtDOdfSBCRxP+GhhxkKSrTleQjD0/r
CGkdG2Kox3m9YllAsvZchLXlS7bZV9mGRF61mVMjF3HJRUQfBBm89VPQ+QARAQAB
tCBjdWRhdG9vbHMgPGN1ZGF0b29sc0BudmlkaWEuY29tPokCNwQTAQgAIQUCV223
hQIbAwULCQgHAwUVCgkICwUWAgMBAAIeAQIXgAAKCRD2D0s9f6KvgNArEAChnfcW
rYItgt7xXXubT6E+KpJyJ0RPrXf51S2mhciFbjDl+3EXRMRjOutVmgWYPWUUZaKR
8Iez3Lz4BRmwYOWBLtdnOLbKoSsQUX95rnPFjfly/DFLfjKxz4NRBmh4r4/rCYWm
2hmnXmOAi8kV7fqx3g5XMpJ//N6+T8ctEol2iZ82GrXjadcRWE4rAe7UyuEzJ74y
6ZKIzk5ijdgEKtcaBhzEWvoV5Pr9nkn7ByGsdehKR/gNnjPMYXrklSHGfphJIsS2
S32lMk/kuRjihBcWcYBXIPEQ7CV+PNW2TlkZj/YqTg637sZHwkhcjcNzxeqKvRYG
8V7Ju5hTDxL1UQBmgDS3cRx1lw7tYRG5bS67tbC2dc/CpPkG5agiZ/WyoHQDnn4r
1fRuOFx694QR6+0rAP6171xEEoNAPaH7gdJdhWKiYiJD0T2EEbW7wBUi/EupeKRv
kR12R1jUa1mlpxNtWQxJ7qp98T9+DmkxI1XDmWx0/g4ryuicwLDSqoPgNcRNdSQb
b8YfTqrkqaDdYzwLr/n0YKW3cYIvIeisV0WxRjb6OP7oAlAtaAhImlIc//51qNO7
/WAud6qMtnhFoZayR/BzLKqnCioN5GYr9BAKskpPHe9cDKVS3fg+Qvc1sNJID+jf
k52PqyW24Qsr0A9+5zQyE4tH9dfv120gj9avmg==
=0nKc
-----END PGP PUBLIC KEY BLOCK-----


그리고나서 ppc64le 아키텍처의 Ubuntu 기반 docker base image를 아래와 같이 pull로 당겨옵니다.

[root@ac922 docker]# docker pull ubuntu:18.04

아래와 같이 작은 ubuntu 기반 image가 pull 된 것을 보실 수 있습니다.

[root@ac922 docker]# docker images
REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
ubuntu                              18.04               a7cbdb002963        4 weeks ago         129MB

이게 ppc64le 아키텍처의 image가 맞는지는 아래와 같이 확인하시면 됩니다.

[root@ac922 docker]# docker inspect ubuntu:18.04 | grep -i arch
        "Architecture": "ppc64le",

위에서 받은 base image를 이용하여, cuda10-0 기반의 ubuntu docker image를 만들기 위한 dockerfile을 아래와 같이 작성합니다.

[root@ac922 docker]# cat dockerfile.cuda10-0
FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common gcc g++ zip unzip gzip bzip2 build-essential wget git vim-tiny  libcurl4-openssl-dev libssl-dev libssh2-1-dev
RUN mkdir /tmp/temp
COPY apt.key1 /tmp/temp
RUN apt-key add /tmp/temp/apt.key1
RUN add-apt-repository "deb [arch=ppc64el] https://download.docker.com/linux/ubuntu bionic stable"
COPY cuda-repo-ubuntu1804_10.0.130-1_ppc64el.deb /tmp/temp/
RUN dpkg -i /tmp/temp/cuda-repo-ubuntu1804_10.0.130-1_ppc64el.deb
COPY apt.key2 /tmp/temp
RUN apt-key add /tmp/temp/apt.key2
RUN apt-get update
RUN apt-get install -y `apt-cache pkgnames | grep cuda | grep -v qnx | grep -v armhf | grep -v aarch64 | grep 10-0`
COPY cudnn-10.0-linux-ppc64le-v7.5.0.56.solitairetheme8 /tmp/temp/
COPY nccl_2.4.2-1+cuda10.0_ppc64le.txz /tmp/temp/
LABEL com.nvidia.volumes.needed="nvidia_driver"
LABEL com.nvidia.cuda.version="10.1"
RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf
RUN echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf
RUN tar -xf /tmp/temp/cudnn-10.0-linux-ppc64le-v7.5.0.56.solitairetheme8 -C /usr/local
RUN tar -xf /tmp/temp/nccl_2.4.2-1+cuda10.0_ppc64le.txz -C /usr/local
# set the working directory
ENV LD_LIBRARY_PATH="/usr/local/nvidia/lib64:/usr/local/nvidia/lib:/usr/local/cuda/lib64:/usr/lib:/usr/lib64:/lib:/lib64:/usr/local/lib:/usr/local/lib64"
ENV PATH="/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/bin"
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
ENV NVIDIA_REQUIRE_CUDA "cuda>=10.0"
CMD /bin/bash

위와 같이 작성된 dockerfile을 이용하여 아래 docker build 명령으로 CUDA 10.0이 설치된 docker image를 build합니다.

[root@ac922 docker]# docker build -f dockerfile.cuda10-0 -t bsyu/ubuntu18.04_cuda10-0_ppc64le:v0.2 .

이제 아래와 같이 bsyu/ubuntu18.04_cuda10-0_ppc64le:v0.2 라는 이미지가 build 되었습니다.

[root@ac922 docker]# docker images
REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
bsyu/ubuntu18.04_cuda10-0_ppc64le   v0.2                c723faffd5c5        About an hour ago   5.47GB
ubuntu                              18.04               a7cbdb002963        4 weeks ago         129MB

이 image를 아래와 같이 run 시켜 봅니다. 

[root@ac922 docker]# docker run --runtime=nvidia -ti --rm bsyu/ubuntu18.04_cuda10-0_ppc64le:v0.2 bash

아래와 같이 nvidia-smi 명령을 수행해보면 일단 잘 돌아가는 것을 보실 수 있습니다.

root@19b69f2a09a3:/# nvidia-smi -l 2
Fri Apr 12 12:55:21 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.40.04    Driver Version: 418.40.04    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000004:04:00.0 Off |                    0 |
| N/A   40C    P0    65W / 300W |  31606MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000004:05:00.0 Off |                    0 |
| N/A   44C    P0    71W / 300W |  31606MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000035:03:00.0 Off |                    0 |
| N/A   40C    P0    65W / 300W |  31606MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000035:04:00.0 Off |                    0 |
| N/A   45C    P0    55W / 300W |  31606MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+


하지만 이 경우는 운이 좋은 것에 불과합니다.   Parent OS인 Redhat 7.5 ppc64le에도 nvidia driver 버전이 Ubuntu image속 버전과 동일한 418.40.04 이었기 때문에 이렇게 잘 수행되는 것입니다.  원래, nvidia-docker2에서는 image 속에 nvidia driver가 설치되어 있으면 안 됩니다.

만약 parent OS의 nvidia-driver 버전이 다른 경우, 아래와 같은 error message가 나면서 dicker run이 안 될 것입니다.  아래는 Ubuntu image 속의 driver가 410.104 버전인 경우입니다.

[root@ac922 docker]# docker run --runtime=nvidia -ti --rm bsyu/ubuntu18.04_cuda9-2_ppc64le:v0.1 bash

docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:385: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig --device=all --compute --utility --pid=44664 /var/lib/docker/overlay2/65906de889a32465e87e680b8968834abaa1f4c4069beadab01bb7691f43523f/merged]\\\\nnvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/65906de889a32465e87e680b8968834abaa1f4c4069beadab01bb7691f43523f/merged/usr/bin/nvidia-smi: file exists\\\\n\\\"\"": unknown.

이를 해결하기 위해서는 docker image 속에 CUDA를 설치할 때 함께 설치된 nvidia driver를 제거해줘야 합니다.   먼저, 아래와 같이 nvidia-docker가 아닌 그냥 docker로 수행되도록 --runtime=nvidia 옵션을 빼고 docker를 구동합니다.  그렇게 하면 최소한 위의 'file exists' error는 나지 않을 것입니다.

[root@ac922 docker]# docker run -ti --rm bsyu/ubuntu18.04_cuda9-2_ppc64le:v0.1 bash             

이어서 image 속에 들어가보면 아래와 같은 nvidia driver들이 보일 것입니다. 

root@45fb663f025c:/# dpkg -l | grep nvidia
ii  nvidia-410                      410.104-0ubuntu1                  ppc64el      NVIDIA binary driver - version 410.104
ii  nvidia-410-dev                  410.104-0ubuntu1                  ppc64el      NVIDIA binary Xorg driver development files
ii  nvidia-modprobe                 410.104-0ubuntu1                  ppc64el      Load the NVIDIA kernel driver and create device files
ii  nvidia-opencl-icd-410           410.104-0ubuntu1                  ppc64el      NVIDIA OpenCL ICD
ii  nvidia-prime                    0.8.8.2                           all          Tools to enable NVIDIA's Prime
ii  nvidia-settings                 410.104-0ubuntu1                  ppc64el      Tool for configuring the NVIDIA graphics driver

아래와 같이 이 관련 fileset들을 다 지워버린 뒤, 새로운 tag로 해당 container를 commit 합니다.

root@45fb663f025c:/# apt-get remove -y nvidia-*

Docker image 속의 저 명령이 완료되면 다른 창에서 parent OS에 접속하여 다음고 같이 commit 합니다.

먼저 저 container의 ID를 확인하고...

[root@ac922 docker]# docker ps -a
CONTAINER ID        IMAGE                                   COMMAND             CREATED             STATUS              PORTS               NAMES
45fb663f025c        bsyu/ubuntu18.04_cuda9-2_ppc64le:v0.1   "bash"              42 seconds ago      Up 39 seconds                           elastic_gates

이어서 v0.2라는 새로운 tag로 commit 합니다.

[root@ac922 docker]# docker commit 45fb663f025c bsyu/ubuntu18.04_cuda9-2_ppc64le:v0.2
sha256:172cdbe6ba2f5f21ce8352a7be964d921d0340c82c1d1d91d54bf28d2c5e2c33


아래와 같이 여러가지 image들을 준비하여 public docker hub에 push로 올려 놓았습니다.  아래 이름을 참조하여 필요에 따라 pull 하여 사용하시기 바랍니다.


[root@ac922 docker]# docker images | grep latest
bsyu/ubuntu18.04_cuda9-2_python368_pytorch1.01_ppc64le    latest              b6695ad4cb7f        8 days ago          8.19GB
bsyu/ubuntu18.04_cuda10-0_ppc64le                         latest              a4f8442230bf        8 days ago          5.54GB
bsyu/ubuntu18.04_cuda10-0_python368_ppc64le               latest              817f3211acdb        8 days ago          7.29GB
bsyu/ubuntu18.04_cuda10-0_python352_ppc64le               latest              cadd3fdb9653        8 days ago          8.6GB
bsyu/ubuntu18.04_cuda10-0_python352_tf1.12_ppc64le        latest              929db5ab1260        8 days ago          18.2GB
bsyu/ubuntu18.04_cuda10-0_python368_pytorch1.01_ppc64le   latest              00fb9697002a        8 days ago          9.45GB
bsyu/ubuntu18.04_cuda10-0_python368_tf1.12_ppc64le        latest              cf2342b0c1b8        8 days ago          16.8GB
bsyu/ubuntu18.04_cuda10-0_python352_pytorch1.01_ppc64le   latest              e8d0a0897362        8 days ago          9.24GB
bsyu/ubuntu18.04_cuda9-2_python352_pytorch1.01_ppc64le    latest              9bf2cafb6836        8 days ago          8.92GB
bsyu/ubuntu18.04_cuda9-2_python352_tf1.12_ppc64le         latest              c1d6c56b6f11        8 days ago          17.4GB
bsyu/ubuntu18.04_cuda9-2_python352_ppc64le                latest              79d3a19ae9ad        8 days ago          8.35GB
bsyu/ubuntu18.04_cuda9-2_python368_tf1.12_ppc64le         latest              e80932cafae5        8 days ago          18.5GB
bsyu/ubuntu18.04_cuda9-2_ppc64le                          latest              eeae2d1f6fd4        8 days ago          5.26GB
bsyu/ubuntu18.04_cuda10-0_python368_cudf0.7.0_ppc64le     latest              86930ab12fe2        8 days ago          10.4GB