nvidia-docker를 위한 Redhat 7.5 ppc64le docker image의 build

** 이 문서에서 서술하는 방식으로 만든 이미지들의 완성본을 다음의 주소에서 받으실 수 있습니다.   IBM GPU 서버인 AC922이나 Minsky 서버를 위한 Redhat 7.5 ppc64le + CUDA 9.2 + cudnn 7.2.1 + NCCL 2.2 + PowerAI 5.2 + Anaconda 2 (py2) or Anaconda3 (py3) 기반으로 만들어진 이미지들입니다.

https://hub.docker.com/r/bsyu/rhel75_ppc64le_cuda92_all_py2_powerai52/
https://hub.docker.com/r/bsyu/rhel75_ppc64le_cuda92_all_py3_powerai52/

Pull 할 때는 다음과 같이 하시면 됩니다.

# docker pull bsyu/rhel75_ppc64le_cuda92_all_py2_powerai52:v0.4     (python2.7)
# docker pull bsyu/rhel75_ppc64le_cuda92_all_py3_powerai52:v0.3     (python3.6)

아래는 이 docker image들을 만들 때의 과정입니다.

----------------------

먼저, 현재 host 서버의 OS(여기서는 Redhat 7.5 ALT ppc64le)를 기반으로 docker base image를 만듭니다.  아래 github에서 제공되는 mkimage-yum.sh를 사용하면 쉽습니다.

https://github.com/moby/moby/blob/master/contrib/mkimage-yum.sh

[root@ING ~]# cd docker

[root@ING docker]# vi mkimage-yum.sh    #  https://github.com/moby/moby/blob/master/contrib/mkimage-yum.sh의 script를 copy & paste

[root@ING docker]# chmod +x mkimage-yum.sh

[root@ING docker]# ./mkimage-yum.sh baserehel75alt
...
Running transaction
  Installing : libgcc-4.8.5-28.el7_5.1.ppc64le                                                                         1/243
  Installing : redhat-release-server-7.5-7.el7a.ppc64le                                                                2/243
  Installing : setup-2.8.71-9.el7.noarch                                                                               3/243
  Installing : filesystem-3.2-25.el7.ppc64le                                                                           4/243
  Installing : tzdata-2018e-3.el7.noarch                                                                               5/243
  Installing : basesystem-10.0-7.el7.noarch                                                                            6/243
...
  xz-libs.ppc64le 0:5.2.2-1.el7                                       yum-metadata-parser.ppc64le 0:1.1.4-10.el7
  zlib.ppc64le 0:1.2.7-17.el7

Complete!
+ [[ -n '' ]]
+ yum -c /etc/yum.conf --installroot=/tmp/mkimage-yum.sh.Oh71td -y clean all
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
There are no enabled repos.
 Run "yum repolist all" to see the repos you have.
 To enable Red Hat Subscription Management repositories:
     subscription-manager repos --enable <repo>
 To enable custom repositories:
     yum-config-manager --enable <repo>

Script가 위와 같이 성공적으로 완료된 뒤에 message에 표시된 /tmp 밑의 directory로 가보면 아래와 같이 mini-OS image가 마련되어 있는 것을 보실 수 있습니다. 
이제 이걸 tar로 말아올린 뒤 docker import 명령으로 import 해주면 됩니다.

[root@ING docker]# cd /tmp/mkimage-yum.sh.Oh71td

[root@ING mkimage-yum.sh.Oh71td]# ls -ltr
total 12
dr-xr-xr-x  2 root root    6 Dec 15  2017 sys
drwxr-xr-x  2 root root    6 Dec 15  2017 srv
dr-xr-xr-x  2 root root    6 Dec 15  2017 proc
drwxr-xr-x  2 root root    6 Dec 15  2017 opt
drwxr-xr-x  2 root root    6 Dec 15  2017 mnt
drwxr-xr-x  2 root root    6 Dec 15  2017 media
drwxr-xr-x  2 root root    6 Dec 15  2017 home
drwxr-xr-x  2 root root  180 Dec 15  2017 dev
dr-xr-xr-x  2 root root    6 Dec 15  2017 boot
lrwxrwxrwx  1 root root    7 Aug  9 09:47 bin -> usr/bin
lrwxrwxrwx  1 root root    8 Aug  9 09:47 sbin -> usr/sbin
lrwxrwxrwx  1 root root    9 Aug  9 09:47 lib64 -> usr/lib64
lrwxrwxrwx  1 root root    7 Aug  9 09:47 lib -> usr/lib
drwxr-xr-x 13 root root  207 Aug  9 09:47 usr
drwxrwxrwt  7 root root  117 Aug  9 09:47 tmp
drwxr-xr-x 19 root root  335 Aug  9 09:47 var
drwxr-xr-x 15 root root  261 Aug  9 09:47 run
drwxr-xr-x 68 root root 8192 Aug  9 09:47 etc
dr-xr-x---  2 root root   91 Aug  9 09:47 root

[root@ING mkimage-yum.sh.Oh71td]# tar -zcf /tmp/rhel7_ppc64le.tar.gz .

[root@ING mkimage-yum.sh.Oh71td]# ls -l /tmp/rhel7_ppc64le.tar.gz
-rw-r--r-- 1 root root 224266718 Aug  9 10:15 /tmp/rhel7_ppc64le.tar.gz

[root@ING mkimage-yum.sh.Oh71td]# docker import /tmp/rhel7_ppc64le.tar.gz bsyu/rhel75_ppc64le:base

[root@ING mkimage-yum.sh.Oh71td]# docker images
REPOSITORY                 TAG                         IMAGE ID            CREATED             SIZE
bsyu/rhel75_ppc64le        base                        3bf1baa60960        4 seconds ago       715 MB

이제 이 base image를 기반으로 dockerfile을 편집해서 CUDA 명령을 쓸 수 있는 nvidia-docker image를 만들면 됩니다.

그러기 위해서는 먼저 이 docker base image에서도 yum 명령을 써야 하는데, 기본적으로 Redhat에서 제공되는 /etc/yum.repos.d/redhat.repo 파일은 매번 yum을 기동할 때마다 자동으로 blank로 reset 되어버립니다.   따라서 이 파일 이름말고, 다른 이름, 가령 /etc/yum.repos.d/new.repo라는 이름으로 host OS에 있는 /etc/yum.repos.d/redhat.repo 파일을 그대로 저장해두면 host OS에서처럼 외부 YUM repository를 자유롭게 쓸 수 있습니다.   물론 이때 /etc/pki/entitlement/*-key.pem 등의 필요 ceritificate 파일들도 함께 docker image 속의 /etc/pki/entitlement에 copy 해두어야 합니다.   가령 다음과 같이 하면 됩니다.

[bsyu@p57a22 ~]# docker run -ti --rm -v /etc:/mnt bsyu/rhel75_ppc64le:base bash

----inside docker container------

[root@7e3b00f3fac2 ~]# cp /mnt/pki/entitlement/*.pem /etc/pki/entitlement

[root@7e3b00f3fac2 ~]# cp /mnt/yum.repos.d/redhat.repo /etc/yum.repos.d/new.repo    # 절대 redhat.repo라는 이름으로 copy하면 안 됩니다.

[root@7e3b00f3fac2 ~]# rpm -Uvh /mnt/epel-release-7-11.noarch.rpm   # epel-release-7-11.noarch.rpm 파일의 경우 host의 /etc 밑에 미리 copy해둡니다.

[root@7e3b00f3fac2 ~]# yum install wget curl cmake cmake3 make automake autoconf bzip2 zip unzip git gcc gcc-c++ gfortran yum-utils   # 기본적으로 필요한 package들을 미리 설치해둡니다.

----outside docker container------

[bsyu@p57a22 ~]# docker ps
CONTAINER ID        IMAGE                   COMMAND             CREATED             STATUS              PORTS              NAMES
7e3b00f3fac2        bsyu/rhel75_ppc64le:base bash   "bash"              23 hours ago        Up 23 hours                  angry_bose

[bsyu@p57a22 ~]# docker commit 7e3b00f3fac2 bsyu/rhel75_ppc64le:base_v6

이제 dockerfile을 이용하여 nvidia-docker 이미지를 만듭니다.  먼저 (다른 이름도 상관없습니다만) docker라는 directory를 만들고, 거기에 docker image 속에 설치할 cuda*.rpm 등을 미리 copy 해둡니다.  그리고나서 dockerfile.cuda92 파일을 다음과 같이 만듭니다.

[bsyu@p57a22 ~]# mkdir /home/docker && cd /home/docker   

[root@p57a22 docker]# vi dockerfile.cuda92
FROM bsyu/rhel75_ppc64le:base_v6

# Update image
# 'local' means rhel local repository
# To use these repos below in building process, you must set up the RHEL subscription and enable these repos on your host system first.
RUN yum-config-manager --enable rhel-7-for-power-9-rpms/7Server/ppc64le
RUN yum-config-manager --enable rhel-7-for-power-9-optional-rpms/7Server/ppc64le
RUN yum-config-manager --enable rhel-7-for-power-9-extras-rpms/7Server/ppc64le
RUN yum-config-manager --enable epel/ppc64le

RUN mkdir /tmp/temp
COPY cuda-repo-rhel7-9-2-local* /tmp/temp/
RUN rpm -Uvh /tmp/temp/cuda-repo-rhel7-9-2-local*
RUN yum install -y cuda

# nvidia-docker 1.0
LABEL com.nvidia.volumes.needed="nvidia_driver"
LABEL com.nvidia.cuda.version="${CUDA_VERSION}"

RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \
    echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf

ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64/stubs

# nvidia-container-runtime
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
ENV NVIDIA_REQUIRE_CUDA "cuda>=9.2"

RUN yum clean all &&  rm -rf /var/cache/yum/* /tmp/temp
RUN rpm -e cuda-repo-rhel7-9-2-local

이제 위의 dockerfile.cuda92 파일을 이용하여 다음과 같이 bsyu/rhel75_ppc64le_cuda92_all:v0.1라는 이름으로 docker image를 build합니다.

[root@p57a22 docker]# docker build -t bsyu/rhel75_ppc64le_cuda92_all:v0.1 -f dockerfile.cuda92 .
Sending build context to Docker daemon 1.705 GB
Step 1/24 : FROM docker.io/bsyu/rhel75_ppc64le:base_v5
 ---> fc88cbad6b18
....
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
 ---> f26c0a2167a0
Removing intermediate container bb911c868f27
Step 26/26 : RUN rpm -e cuda-repo-rhel7-9-2-local
 ---> Running in 6bceb027f2f1

 ---> deaf06286bf4
Removing intermediate container 6bceb027f2f1
Successfully built deaf06286bf4

다음과 같이 지정한 이름의 docker image가 생성된 것을 확인합니다.

[root@p57a22 docker]# docker images | grep bsyu
bsyu/rhel75_ppc64le_cuda92_all                 v0.1                        deaf06286bf4        45 seconds ago      8.11 GB
bsyu/rhel75_ppc64le                       base_v6                     872374a229f2        8 minutes ago       1.35 GB

이제 nvidia-docker로 이 image를 구동하여 nvidia-smi가 제대로 작동하는지 확인합니다.   그리고 PowerAI나 libcudnn* libnccl2* 등 필요 SW를 더 설치한 뒤 commit 하여 필요한 image를 추가로 만듭니다.

[bsyu@p57a22 data]$ nvidia-docker run -ti --rm -v /home/bsyu/files:/mnt bsyu/rhel75_ppc64le_cuda92_all:v0.1 bash

----inside docker container------

[root@e8a6f36fb132 /]# nvidia-smi
Thu Aug 23 01:28:52 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.26                 Driver Version: 396.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-SXM2...  Off  | 00000002:01:00.0 Off |                    0 |
| N/A   28C    P0    29W / 300W |     10MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P100-SXM2...  Off  | 00000003:01:00.0 Off |                    0 |
| N/A   30C    P0    32W / 300W |     10MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla P100-SXM2...  Off  | 0000000A:01:00.0 Off |                    0 |
| N/A   28C    P0    29W / 300W |     10MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla P100-SXM2...  Off  | 0000000B:01:00.0 Off |                    0 |
| N/A   31C    P0    29W / 300W |     10MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Estimator-based tensorflow training에 LMS 적용한 MNIST python code

이 MNIST training을 위한 tensorflow python code는 원래 /opt/DL/tensorflow/lib/python3.6/site-packages/tensorflow/examples/tutorials/layers/cnn_mnist.py 에 LMS를 적용한 것입니다.  보시다시피 hook를 사용하기 때문에 Estimator-based tensorflow training이고, 그에 따라 LMS가 적용되어 있습니다. 

LMS가 동작하는 message 확인이라든가, 다중 사용자를 위한 permission 등을 위한 부분도 있습니다만 그건 LMS와는 무관한 부분이고, 그 부분들은 빨간색으로 표시를 했습니다.  실제 LMS 구현을 위한 부분은 굵은 파란색으로 표시했습니다.  의외로 간단하다는 것을 보실 수 있습니다.  해당 부분들을 제거하면 그냥 LMS 없는 평범한 MNIST training code가 됩니다.

이 example code도 PowerAI 5.2를 설치하면 딸려오는 /opt/DL/tensorflow/lib/python3.6/site-packages/tensorflow/contrib/lms/examples/cnn_mnist_lms.py 을 그대로 가져다 놓은 것입니다.

실제 수행해보면 다음과 같이 동작하며, 12개의 tensor가 host 서버의 RAM으로 swap-out/in 되는 것을 보실 수 있습니다.

[bsyu@p57a22 ~]$ cd /opt/DL/tensorflow/lib/python3.6/site-packages/tensorflow/contrib/lms/examples

[bsyu@p57a22 examples]$ source /opt/DL/tensorflow/bin/tensorflow-activate

[bsyu@p57a22 examples]$ python cnn_mnist_lms.py
...
INFO:tensorflow:[LMS][1] Tensor sparse_softmax_cross_entropy_loss/Sum_1:0 will be placed on /cpu:0
INFO:tensorflow:[LMS][1] Operation: adam_optimizer/gradients/sparse_softmax_cross_entropy_loss/div_grad/RealDiv_1, order 23, type RealDiv
INFO:tensorflow:[LMS][1] Tensor adam_optimizer/gradients/sparse_softmax_cross_entropy_loss/div_grad/RealDiv_1:0 will be placed on /cpu:0
INFO:tensorflow:[LMS][1] Consuming op adam_optimizer/gradients/sparse_softmax_cross_entropy_loss/div_grad/RealDiv_2 (order 24) swaps in adam_optimizer/gradients/sparse_softmax_cross_entropy_loss/div_grad/RealDiv_1:0
INFO:tensorflow:[LMS][1] No control dependency op needed for swap in of op adam_optimizer/gradients/sparse_softmax_cross_entropy_loss/div_grad/RealDiv_1.
INFO:tensorflow:[LMS][0] Edited model is valid and logically equivalent to the original one
INFO:tensorflow:[LMS][0] Added 25 ops into the model
INFO:tensorflow:[LMS][0] Editing model for LMS, took: 88.58513832092285 ms
INFO:tensorflow:[LMS][0] 12 tensors will be swapped out(in) to(from) the host
INFO:tensorflow:Graph was finalized.
...
{'accuracy': 0.9702, 'loss': 0.098796368, 'global_step': 20000}

전체 code 내용은 아래를 보시기 바랍니다.

[bsyu@p57a22 examples]$ cat /opt/DL/tensorflow/lib/python3.6/site-packages/tensorflow/contrib/lms/examples/cnn_mnist_lms.py

#  Copyright 2016 The TensorFlow Authors. All Rights Reserved.
#
#  Licensed under the Apache License, Version 2.0 (the "License");
#  you may not use this file except in compliance with the License.
#  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.
"""Convolutional Neural Network Estimator for MNIST, built with tf.layers."""

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
import tempfile # Change not related to LMS
import tensorflow as tf

tf.logging.set_verbosity(tf.logging.INFO)  #LMS 기능과는 무관. LMS 메시지를 보기 위한 설정.


def cnn_model_fn(features, labels, mode):
  """Model function for CNN."""
  # Input Layer
  # Reshape X to 4-D tensor: [batch_size, width, height, channels]
  # MNIST images are 28x28 pixels, and have one color channel
  input_layer = tf.reshape(features["x"], [-1, 28, 28, 1])

  # Convolutional Layer #1
  # Computes 32 features using a 5x5 filter with ReLU activation.
  # Padding is added to preserve width and height.
  # Input Tensor Shape: [batch_size, 28, 28, 1]
  # Output Tensor Shape: [batch_size, 28, 28, 32]
  conv1 = tf.layers.conv2d(
      inputs=input_layer,
      filters=32,
      kernel_size=[5, 5],
      padding="same",
      activation=tf.nn.relu)

  # Pooling Layer #1
  # First max pooling layer with a 2x2 filter and stride of 2
  # Input Tensor Shape: [batch_size, 28, 28, 32]
  # Output Tensor Shape: [batch_size, 14, 14, 32]
  pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)

  # Convolutional Layer #2
  # Computes 64 features using a 5x5 filter.
  # Padding is added to preserve width and height.
  # Input Tensor Shape: [batch_size, 14, 14, 32]
  # Output Tensor Shape: [batch_size, 14, 14, 64]
  conv2 = tf.layers.conv2d(
      inputs=pool1,
      filters=64,
      kernel_size=[5, 5],
      padding="same",
      activation=tf.nn.relu)

  # Pooling Layer #2
  # Second max pooling layer with a 2x2 filter and stride of 2
  # Input Tensor Shape: [batch_size, 14, 14, 64]
  # Output Tensor Shape: [batch_size, 7, 7, 64]
  pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)

  # Flatten tensor into a batch of vectors
  # Input Tensor Shape: [batch_size, 7, 7, 64]
  # Output Tensor Shape: [batch_size, 7 * 7 * 64]
  pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])

  # Dense Layer
  # Densely connected layer with 1024 neurons
  # Input Tensor Shape: [batch_size, 7 * 7 * 64]
  # Output Tensor Shape: [batch_size, 1024]
  dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)

  # Add dropout operation; 0.6 probability that element will be kept
  dropout = tf.layers.dropout(
      inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN)

  # Logits layer
  # Input Tensor Shape: [batch_size, 1024]
  # Output Tensor Shape: [batch_size, 10]
  logits = tf.layers.dense(inputs=dropout, units=10)

  predictions = {
      # Generate predictions (for PREDICT and EVAL mode)
      "classes": tf.argmax(input=logits, axis=1),
      # Add `softmax_tensor` to the graph. It is used for PREDICT and by the
      # `logging_hook`.
      "probabilities": tf.nn.softmax(logits, name="softmax_tensor")
  }
  if mode == tf.estimator.ModeKeys.PREDICT:
    return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)

  # Calculate Loss (for both TRAIN and EVAL modes)
  loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)

  # Configure the Training Op (for TRAIN mode)
  if mode == tf.estimator.ModeKeys.TRAIN:
    with tf.name_scope('adam_optimizer'):
      optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)
      train_op = optimizer.minimize(
        loss=loss,
        global_step=tf.train.get_global_step())
    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)

  # Add evaluation metrics (for EVAL mode)
  eval_metric_ops = {
      "accuracy": tf.metrics.accuracy(
          labels=labels, predictions=predictions["classes"])}
  return tf.estimator.EstimatorSpec(
      mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)


def main(unused_argv):
  # Load training and eval data
  mnist = tf.contrib.learn.datasets.load_dataset("mnist")
  train_data = mnist.train.images  # Returns np.array
  train_labels = np.asarray(mnist.train.labels, dtype=np.int32)
  eval_data = mnist.test.images  # Returns np.array
  eval_labels = np.asarray(mnist.test.labels, dtype=np.int32)

  # The graph_location changes are not related to LMS enablement
  # but rather allow multiple users to run the example without
  # having permission issues on temp directories.
  graph_location = tempfile.mkdtemp()
  print('Saving graph to: %s' % graph_location)

  # Create the Estimator
  mnist_classifier = tf.estimator.Estimator(
      model_fn=cnn_model_fn, model_dir=graph_location)

  # Set up logging for predictions
  # Log the values in the "Softmax" tensor with label "probabilities"
  tensors_to_log = {"probabilities": "softmax_tensor"}
  logging_hook = tf.train.LoggingTensorHook(
      tensors=tensors_to_log, every_n_iter=50)

  # Train the model
  train_input_fn = tf.estimator.inputs.numpy_input_fn(
      x={"x": train_data},
      y=train_labels,
      batch_size=100,
      num_epochs=None,
      shuffle=True)

  # Hook for Large Model Support
  from tensorflow.contrib.lms import LMSHook
  lms_hook = LMSHook({'adam_optimizer'}, lb=3, debug=True)

  mnist_classifier.train(
      input_fn=train_input_fn,
      steps=20000,
      hooks=[logging_hook, lms_hook])

  # Evaluate the model and print results
  eval_input_fn = tf.estimator.inputs.numpy_input_fn(
      x={"x": eval_data},
      y=eval_labels,
      num_epochs=1,
      shuffle=False)
  eval_results = mnist_classifier.evaluate(input_fn=eval_input_fn)
  print(eval_results)


if __name__ == "__main__":
  tf.app.run()