2019년 4월 25일 목요일
RAPIDS.ai on IBM POWER9 : ppc64le 환경에서 cuDF를 source로부터 build하기
RAPIDS.ai는 NVIDIA가 요즘 공을 들이고 있는 GPU를 이용한 data science platform입니다. 그 핵심 모듈이 cuDF와 cuML입니다. 이 모듈들은 최근 발표된 IBM PowerAI toolkit에도 포함되어 있습니다. ( https://www.ibm.com/support/knowledgecenter/en/SS5SF7_1.6.0/navigation/pai_getstarted_rapids.html 참조)
다만 IBM PowerAI는 CUDA 10.1만을 지원하는 등 제약 사항이 일부 있기 때문에 모든 환경에서 유연하게 사용하시기가 곤란한 부분이 있을 수 있습니다. 그래도 어차피 RAPIDS는 open source니까, ppc64le 아키텍처 위에서 그냥 source로부터 build하시면 됩니다. 이번 posting에서는 cuDF를 Ubuntu 18.04 ppc64le에서 build하는 방법을 보시겠습니다.
일단 아래와 같이 cuda 10.0을 설치합니다.
~/files$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/cuda-repo-ubuntu1804_10.0.130-1_ppc64el.deb
~/files$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/ppc64el/7fa2af80.pub
~/files$ sudo apt-get update
~/files$ sudo apt-get install cuda
~/files$ sudo tar -xvf ./cudnn-10.0-linux-ppc64le-v7.5.0.56.solitairetheme8 -C /usr/local
~/files$ sudo tar -xvf ./nccl_2.4.2-1+cuda10.0_ppc64le.txz -C /usr/local
Python은 Anaconda3를 설치하여 Python 3.6.8을 사용했습니다.
~/files$ which python
/home/u0017649/anaconda3/bin/python
먼저 conda install 명령으로 아래 package들을 설치합니다.
~/files$ conda install numpy six setuptools cython pandas pytest cmake flatbuffers rapidjson boost boost-cpp thrift snappy zlib gflags brotli lz4-c zstd arrow -c conda-forge
먼저 유의하실 것은 Ubuntu 18.10의 기본 gcc 버전인 gcc-8은 CUDA 10.0을 지원하지 않습니다. 그런데 또 현재 cudf v0.7.0은 아직 CUDA 10.1을 지원하지 않습니다. 그래서 해결책은 Ubuntu에 gcc-7과 g++-7을 설치해야 합니다.
~/files$ sudo apt-get install -y gcc-7 gcc-7-base gcc-7-powerpc-linux-gnu libgcc-7-dev g++-7
~/files$ sudo rm /usr/bin/gcc /usr/bin/g++
~/files$ sudo ln -s /usr/bin/gcc-7 /usr/bin/gcc
~/files$ sudo ln -s /usr/bin/g++-7 /usr/bin/g++
먼저, RMM을 설치합니다.
~/files$ git clone --recursive https://github.com/rapidsai/rmm.git
~/files$ cd rmm
~/files/rmm$ rm -rf build
~/files/rmm$ mkdir build && cd build
~/files/rmm/build$ cmake ..
~/files/rmm/build$ make -j2
~/files/rmm/build$ sudo make install
~/files/rmm/build$ cd ../python
~/files/rmm/python$ python setup.py install
다음으로는 custrings를 설치합니다.
~/files$ git clone --recursive https://github.com/rapidsai/custrings.git
~/files$ cd custrings/cpp
~/files/custrings/cpp$ mkdir build && cd build
~/files/custrings/cpp/build$ cmake ..
~/files/custrings/cpp$ make -j2
~/files/custrings/cpp$ sudo make install
다음으로는 thrift를 설치합니다.
~/files$ git clone --recursive https://github.com/apache/thrift
~/files$ cd thrift/
~/files/thrift$ sudo apt-get install libssl-dev
~/files/thrift$ ./bootstrap.sh
~/files/thrift$ ./configure
~/files/thrift$ cd build
~/files/thrift/build$ cmake ..
~/files/thrift/build$ make -j2
~/files/thrift/build$ sudo make install
다음으로는 Arrow를 설치합니다.
~/files$ sudo apt-get install libdouble-conversion1 libghc-double-conversion-dev libdouble-conversion-dev libgoogle-glog-dev libgoogle-glog0v5 thrift-compiler libthrift-java
~/files$ git clone https://github.com/apache/arrow.git
~/files$ cd arrow/cpp
~/files/arrow/cpp$ mkdir build && cd build
여기서 특정 header 파일을 아래와 같이 일부 수정해야 합니다. 이 error는 꼭 ppc64le 환경에서만 나는 것은 아니고, 나중에 patch가 될 예정입니다. ( error: there are no arguments to ‘DCHECK’ that depend on a template parameter, so a declaration of ‘DCHECK’ must be available 이라는 error 방지용. https://github.com/apache/arrow/issues/4014 참조)
~/files/arrow/cpp/build$ vi ~/files/arrow/cpp/src/arrow/util/sse-util.h
...
#include "logging.h" # 이 include 문을 삽입해야 함
~/files/arrow/cpp/build$ cmake -DCMAKE_BUILD_TYPE=release -DARROW_PYTHON=on -DARROW_PLASMA=on -DARROW_BUILD_TESTS=OFF -DARROW_PARQUET=ON ..
~/files/arrow/cpp/build$ make -j2
~/files/arrow/cpp/build$ sudo make install
~/files/arrow/cpp/build$ cd ../../python
~/files/arrow/python$ export ARROW_HOME=/usr/local
기본으로 설치되는 Cython 버전이 맞지 않으므로, 반드시 0.29 버전으로 새로 설치해야 하더라고요.
~/files/arrow/python$ pip install Cython==0.29
~/files/arrow/python$ MAKEFLAGS=-j2 python setup.py build_ext --build-type=release --with-parquet --inplace
~/files/arrow/python$ MAKEFLAGS=-j2 python setup.py build_ext --build-type=release --with-parquet --bundle-arrow-cpp bdist_wheel
위에서 생성되는 wheel file을 아래와 같이 설치하시면 됩니다.
~/files/arrow/python$ pip install dist/pyarrow-0.13.1.dev40+g6907d972.d20190408-cp37-cp37m-linux_ppc64le.whl
이제 비로소 cudf를 설치할 수 있게 되었습니다.
~/files$ git clone --recursive https://github.com/rapidsai/cudf.git
~/files$ cd cudf
~/files/cudf$ git submodule update --init --remote --recursive
여기서 역시 아래의 file 하나를 수정해야 합니다. 이 역시 ppc64le에 국한된 문제는 아니고, 아래 link에서 해결된 것인데, 나중에 patch로 반영될 예정입니다. ( https://github.com/rapidsai/cudf/issues/1384 참조)
~/files/cudf$ vi cpp/tests/hashing/hash_partition_test.cu
...
// The indices of the columns that will be hashed to determine the partitions
constexpr static const std::array<int, sizeof...(cols)> cols_to_hash{{cols...}};
};
template< typename tuple_of_vectors # 이하 4줄의 line을 추가하시면 됩니다.
gdf_hash_func hash,
int... cols>
constexpr const std::array<int, sizeof...(cols)> TestParameters<tuple_of_vectors, hash, cols...>::cols_to_hash;
그 다음은 아래와 같이 build하시면 됩니다.
~/files/cudf$ cd cpp
~/files/cudf/cpp$ mkdir build && cd build
~/files/cudf/cpp/build$ cmake .. -DCMAKE_INSTALL_PREFIX=~/anaconda3
~/files/cudf/cpp/build$ make -j2
~/files/cudf/cpp/build$ sudo make install
root@8f2f82cf3e22:/mnt/cudf# cd python
root@8f2f82cf3e22:/mnt/cudf/python# python setup.py build_ext --inplace
root@8f2f82cf3e22:/mnt/cudf/python# python setup.py install
...
Best match: numpy 1.16.2
Adding numpy 1.16.2 to easy-install.pth file
Installing f2py script to /opt/anaconda3/bin
Installing f2py3 script to /opt/anaconda3/bin
Installing f2py3.6 script to /opt/anaconda3/bin
Using /opt/anaconda3/lib/python3.6/site-packages
Finished processing dependencies for cudf==0.7.0.dev0+679.g97833593.dirty
2017년 3월 16일 목요일
Ubuntu 16.04 ppc64le에서, source로부터 tensorflow를 build하는 방법
PowerAI 속에 tensorflow가 들어 있긴 합니다만, 간혹 customizing 때문에 source에서 build할 필요도 있을 수 있습니다. 여기서는 CUDA 8.0과 PowerAI가 이미 다 설치된 상태에서, build를 시작하는 것으로 하겠습니다. 먼저, 기반 OS package를 설치합니다.
test@k8002:~$ sudo apt-get install make automake autoconf gfortran-4.8-multilib-powerpc-linux-gnu libgfortran-4.8-dev-powerpc-cross gfortran libgfortran-4.8-dev gcc-powerpc-linux-gnu g++ pkg-config cmake-curses-gui zlib1g-dev zip unzip libtool openjdk-8-jdk python python-all-dev swig python-numpy g++-4.8 g++-4.8-multilib-powerpc-linux-gnu zlibc re2c libre2-dev libcupti-dev
Tensorflow를 github에서 download 받습니다.
test@k8002:~$ git clone --recurse-submodules https://github.com/tensorflow/tensorflow
이제 tensorflow에 어떤 버전의 branch들이 있는지 확인합니다.
test@k8002:~$ cd tensorflow
test@k8002:~/tensorflow$ git branch -a
* master
remotes/origin/0.6.0
remotes/origin/HEAD -> origin/master
remotes/origin/master
remotes/origin/r0.10
remotes/origin/r0.11
remotes/origin/r0.12
remotes/origin/r0.7
remotes/origin/r0.8
remotes/origin/r0.9
remotes/origin/r1.0
이중에서 일단은 master 버전을 build해보겠습니다.
test@k8002:~/tensorflow$ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
여기서 일단 환경 변수 확인을 해보겠습니다.
test@k8002:~/tensorflow$ env | grep PATH
LD_LIBRARY_PATH=/usr/local/cuDNN:/usr/local/cuda-8.0/lib64
PATH=/home/test/bin:/home/test/.local/bin:/usr/local/cuda-8.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
Github에서 download 받은 tensorflow를 build할 때도, PowerAI에 포함된 bazel을 사용하면 편리합니다. 그를 위해, 현재의 PATH 환경변수들을 PowerAI 기준으로 고쳐줘야 하는데, 그를 위해서는 아래와 같이 PowerAI에 포함된 script 하나만 수행해주면 됩니다.
test@k8002:~/tensorflow$ source /opt/DL/tensorflow/bin/tensorflow-activate
이제 아래와 같이 PATH에 /opt/DL/tensorflow/bin가 추가되고, PYTHONPATH가 추가된 것을 보실 수 있습니다.
test@k8002:~/tensorflow$ env | grep PATH
LD_LIBRARY_PATH=/usr/local/cuDNN:/usr/local/cuda-8.0/lib64
PATH=/home/test/bin:/home/test/.local/bin:/usr/local/cuda-8.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/DL/tensorflow/bin
PYTHONPATH=/opt/DL/tensorflow/lib/python2.7/site-packages
여기에 추가로 bazel의 PATH도 잡아줘야 합니다. Tensorflow의 build를 위해서는 먼저 bazel을 build해야 하는데, 사실 그것이 tensorflow build보다 더 힘듭니다. 여기서는 PowerAI에 포함된 bazel을 간편하게 사용하겠습니다.
test@k8002:~/tensorflow$ export PATH=/opt/DL/bazel/bin:$PATH
이제 configure부터 시작하여 build에 들어갑니다.
test@k8002:~/tensorflow$ ./configure
Please specify the location of python. [Default is /usr/bin/python]:
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:
Do you wish to use jemalloc as the malloc implementation? [Y/n]
jemalloc enabled
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N]
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N]
No Hadoop File System support will be enabled for TensorFlow
Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N]
No XLA support will be enabled for TensorFlow
Found possible Python library paths:
/opt/DL/tensorflow/lib/python2.7/site-packages
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
Please input the desired Python library path to use. Default is [/opt/DL/tensorflow/lib/python2.7/site-packages]
/usr/lib/python2.7/dist-packages
Do you wish to build TensorFlow with OpenCL support? [y/N]
No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with CUDA support? [y/N] y
CUDA support will be enabled for TensorFlow
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Please specify the CUDA SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 8.0
Please specify the location where CUDA 8.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /usr/local/cuda-8.0
Please specify the Cudnn version you want to use. [Leave empty to use system default]: 5
Please specify the location where cuDNN 5 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda-8.0]: /usr/lib/powerpc64le-linux-gnu
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]: 6.0 (--> P100은 6.0, K80의 경우 3.7)
Extracting Bazel installation...
.............
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
............
WARNING: /home/test/.cache/bazel/_bazel_test/3a51eda24c560ebddf29b66b8fa0460e/external/protobuf/protobuf.bzl:90:19: Variables HOST_CFG and DATA_CFG are deprecated in favor of strings "host" and "data" correspondingly.
WARNING: /home/test/.cache/bazel/_bazel_test/3a51eda24c560ebddf29b66b8fa0460e/external/protobuf/protobuf.bzl:96:28: Variables HOST_CFG and DATA_CFG are deprecated in favor of strings "host" and "data" correspondingly.
INFO: Downloading http://pilotfiber.dl.sourceforge.net/project/boost/boost/1.6\
1.0/boost_1_61_0.tar.gz: 104,733,800 bytes
...
INFO: All external dependencies fetched successfully.
Configuration finished
configure가 끝나면 이제 bazel build를 합니다.
test@k8002:~/tensorflow$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
...
Target //tensorflow/tools/pip_package:build_pip_package up-to-date:
bazel-bin/tensorflow/tools/pip_package/build_pip_package
INFO: Elapsed time: 256.806s, Critical Path: 203.45s
윗 단계가 성공적으로 끝나면 다음과 같이 /tmp/tensorflow_pkg directory에 pip package를 build합니다.
test@k8002:~/tensorflow$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
Thu Mar 16 15:03:32 KST 2017 : === Using tmpdir: /tmp/tmp.LCBvnwu8ZZ
~/tensorflow/bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles ~/tensorflow
~/tensorflow
/tmp/tmp.LCBvnwu8ZZ ~/tensorflow
Thu Mar 16 15:03:33 KST 2017 : === Building wheel
warning: no files found matching '*.dll' under directory '*'
warning: no files found matching '*.lib' under directory '*'
~/tensorflow
Thu Mar 16 15:03:44 KST 2017 : === Output wheel file is in: /tmp/tensorflow_pkg
이제 /tmp/tensorflow_pkg directory에 생성된 wheel file을 이용하여 pip install을 하면 tensorflow 설치가 완료됩니다.
test@k8002:~/tensorflow$ sudo pip install /tmp/tensorflow_pkg/tensorflow-1.0.1-cp27-cp27mu-linux_ppc64le.whl
...
Downloading funcsigs-1.0.2-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): pbr>=0.11 in /usr/lib/python2.7/dist-packages (from mock>=2.0.0->tensorflow==1.0.1)
Installing collected packages: werkzeug, protobuf, funcsigs, mock, tensorflow
Found existing installation: protobuf 2.6.1
Not uninstalling protobuf at /usr/lib/python2.7/dist-packages, outside environment /usr
Found existing installation: funcsigs 0.4
Not uninstalling funcsigs at /usr/lib/python2.7/dist-packages, outside environment /usr
Found existing installation: mock 1.3.0
Not uninstalling mock at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed funcsigs-1.0.2 mock-2.0.0 protobuf-3.2.0 tensorflow-1.0.1 werkzeug-0.12.1
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
이제 tensorflow의 버전을 확인해보겠습니다.
test@k8002:~/tensorflow$ pip show tensorflow
---
Metadata-Version: 2.0
Name: tensorflow
Version: 0.12.0
Summary: TensorFlow helps the tensors flow
Home-page: http://tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
Installer: pip
License: Apache 2.0
Location: /opt/DL/tensorflow/lib/python2.7/site-packages
Requires: numpy, mock, six, protobuf, wheel
Classifiers:
Development Status :: 4 - Beta
Intended Audience :: Developers
Intended Audience :: Education
Intended Audience :: Science/Research
License :: OSI Approved :: Apache Software License
Programming Language :: Python :: 2.7
Topic :: Scientific/Engineering :: Mathematics
Topic :: Software Development :: Libraries :: Python Modules
Topic :: Software Development :: Libraries
Entry-points:
[console_scripts]
tensorboard = tensorflow.tensorboard.tensorboard:main
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
위와 같이 version이 방금 설치한 v1.0이 아니라 v0.12로 나오는 것은 환경변수 PYTHONPATH가 아직 /opt/DL/tensorflow/lib/python2.7/site-packages으로 잡혀 있기 때문에, 이전에 PowerAI 설치 때 함께 설치된 tensorflow 0.12가 보이는 것입니다. 이를 configure할 때 주었던 값인 /usr/local/lib/python2.7/dist-packages으로 바꾼 뒤 다시 해보겠습니다.
test@k8002:~/tensorflow$ export PYTHONPATH=/usr/local/lib/python2.7/dist-packages:$PYTHONPATH
test@k8002:~/tensorflow$ pip show tensorflow
---
Metadata-Version: 2.0
Name: tensorflow
Version: 1.0.1
Summary: TensorFlow helps the tensors flow
Home-page: http://tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
Installer: pip
License: Apache 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: werkzeug, six, protobuf, wheel, numpy, mock
Classifiers:
Development Status :: 4 - Beta
Intended Audience :: Developers
Intended Audience :: Education
Intended Audience :: Science/Research
License :: OSI Approved :: Apache Software License
Programming Language :: Python :: 2.7
Topic :: Scientific/Engineering :: Mathematics
Topic :: Software Development :: Libraries :: Python Modules
Topic :: Software Development :: Libraries
Entry-points:
[console_scripts]
tensorboard = tensorflow.tensorboard.tensorboard:main
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
위와 같이 v1.0.1로 제대로 보이는 것을 보실 수 있습니다.
이번에는 r0.12를 build해보겠습니다.
test@k8002:~/tensorflow$ git checkout r0.12
Switched to branch 'r0.12'
Your branch is up-to-date with 'origin/r0.12'.
동일하게, configure부터 시작하여 build에 들어갑니다.
test@k8002:~/tensorflow$ ./configure
~/tensorflow ~/tensorflow
Please specify the location of python. [Default is /usr/bin/python]:
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N]
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N]
No Hadoop File System support will be enabled for TensorFlow
Found possible Python library paths:
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
Please input the desired Python library path to use. Default is [/usr/local/lib/python2.7/dist-packages]
/usr/lib/python2.7/dist-packages
Do you wish to build TensorFlow with OpenCL support? [y/N]
No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with GPU support? [y/N] y
GPU support will be enabled for TensorFlow
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 8.0
Please specify the location where CUDA 8.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /usr/local/cuda-8.0
Please specify the Cudnn version you want to use. [Leave empty to use system default]: 5
Please specify the location where cuDNN 5 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda-8.0]: /usr/lib/powerpc64le-linux-gnu
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]: 6.0 (--> P100은 6.0, K80의 경우 3.7)
............
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
............
INFO: All external dependencies fetched successfully.
Configuration finished
test@k8002:~/tensorflow$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
... ^
At global scope:
cc1plus: warning: unrecognized command line option '-Wno-self-assign'
Target //tensorflow/tools/pip_package:build_pip_package up-to-date:
bazel-bin/tensorflow/tools/pip_package/build_pip_package
INFO: Elapsed time: 306.555s, Critical Path: 273.68s
test@k8002:~/tensorflow$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
Fri Mar 17 16:16:24 KST 2017 : === Using tmpdir: /tmp/tmp.L1dwTgVhwv
~/tensorflow/bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles ~/tensorflow
~/tensorflow
/tmp/tmp.L1dwTgVhwv ~/tensorflow
Fri Mar 17 16:16:25 KST 2017 : === Building wheel
~/tensorflow
Fri Mar 17 16:16:41 KST 2017 : === Output wheel file is in: /tmp/tensorflow_pkg
이제 /tmp/tensorflow_pkg를 보시면 아까 build했던 1.0.1 버전의 wheel file과 함께 새로 0.12 버전의 wheel file이 만들어진 것을 보실 수 있습니다.
test@k8002:~/tensorflow$ ls -l /tmp/tensorflow_pkg/
total 109520
-rw-rw-r-- 1 test test 61576804 Mar 17 16:16 tensorflow-0.12.1-cp27-cp27mu-linux_ppc64le.whl
-rw-rw-r-- 1 test test 50565190 Mar 16 15:03 tensorflow-1.0.1-cp27-cp27mu-linux_ppc64le.whl
그것을 pip install 명령으로 설치합니다. 아래와 같이, 기존의 tensorflow-1.0.1을 uninstall하면서 0.12가 설치됩니다.
test@k8002:~/tensorflow$ sudo pip install /tmp/tensorflow_pkg/tensorflow-0.12.1-cp27-cp27mu-linux_ppc64le.whl
The directory '/home/test/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/test/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /tmp/tensorflow_pkg/tensorflow-0.12.1-cp27-cp27mu-linux_ppc64le.whl
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.11.0 in /home/test/.local/lib/python2.7/site-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): wheel in /usr/lib/python2.7/dist-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): mock>=2.0.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): six>=1.10.0 in /home/test/.local/lib/python2.7/site-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): protobuf>=3.1.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): funcsigs>=1; python_version < "3.3" in /usr/local/lib/python2.7/dist-packages (from mock>=2.0.0->tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): pbr>=0.11 in /usr/lib/python2.7/dist-packages (from mock>=2.0.0->tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/lib/python2.7/dist-packages (from protobuf>=3.1.0->tensorflow==0.12.1)
Installing collected packages: tensorflow
Found existing installation: tensorflow 1.0.1
Uninstalling tensorflow-1.0.1:
Successfully uninstalled tensorflow-1.0.1
Successfully installed tensorflow-0.12.1
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
다음과 같이 tensorflow의 버전이 0.12.1로 변경된 것을 확인하실 수 있습니다.
test@k8002:~/tensorflow$ pip show tensorflow
---
Metadata-Version: 2.0
Name: tensorflow
Version: 0.12.1
Summary: TensorFlow helps the tensors flow
Home-page: http://tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
Installer: pip
License: Apache 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: numpy, wheel, mock, six, protobuf
Classifiers:
Development Status :: 4 - Beta
Intended Audience :: Developers
Intended Audience :: Education
Intended Audience :: Science/Research
License :: OSI Approved :: Apache Software License
Programming Language :: Python :: 2.7
Topic :: Scientific/Engineering :: Mathematics
Topic :: Software Development :: Libraries :: Python Modules
Topic :: Software Development :: Libraries
Entry-points:
[console_scripts]
tensorboard = tensorflow.tensorboard.tensorboard:main
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
test@k8002:~$ sudo apt-get install make automake autoconf gfortran-4.8-multilib-powerpc-linux-gnu libgfortran-4.8-dev-powerpc-cross gfortran libgfortran-4.8-dev gcc-powerpc-linux-gnu g++ pkg-config cmake-curses-gui zlib1g-dev zip unzip libtool openjdk-8-jdk python python-all-dev swig python-numpy g++-4.8 g++-4.8-multilib-powerpc-linux-gnu zlibc re2c libre2-dev libcupti-dev
Tensorflow를 github에서 download 받습니다.
test@k8002:~$ git clone --recurse-submodules https://github.com/tensorflow/tensorflow
이제 tensorflow에 어떤 버전의 branch들이 있는지 확인합니다.
test@k8002:~$ cd tensorflow
test@k8002:~/tensorflow$ git branch -a
* master
remotes/origin/0.6.0
remotes/origin/HEAD -> origin/master
remotes/origin/master
remotes/origin/r0.10
remotes/origin/r0.11
remotes/origin/r0.12
remotes/origin/r0.7
remotes/origin/r0.8
remotes/origin/r0.9
remotes/origin/r1.0
이중에서 일단은 master 버전을 build해보겠습니다.
test@k8002:~/tensorflow$ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
여기서 일단 환경 변수 확인을 해보겠습니다.
test@k8002:~/tensorflow$ env | grep PATH
LD_LIBRARY_PATH=/usr/local/cuDNN:/usr/local/cuda-8.0/lib64
PATH=/home/test/bin:/home/test/.local/bin:/usr/local/cuda-8.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
Github에서 download 받은 tensorflow를 build할 때도, PowerAI에 포함된 bazel을 사용하면 편리합니다. 그를 위해, 현재의 PATH 환경변수들을 PowerAI 기준으로 고쳐줘야 하는데, 그를 위해서는 아래와 같이 PowerAI에 포함된 script 하나만 수행해주면 됩니다.
test@k8002:~/tensorflow$ source /opt/DL/tensorflow/bin/tensorflow-activate
이제 아래와 같이 PATH에 /opt/DL/tensorflow/bin가 추가되고, PYTHONPATH가 추가된 것을 보실 수 있습니다.
test@k8002:~/tensorflow$ env | grep PATH
LD_LIBRARY_PATH=/usr/local/cuDNN:/usr/local/cuda-8.0/lib64
PATH=/home/test/bin:/home/test/.local/bin:/usr/local/cuda-8.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/DL/tensorflow/bin
PYTHONPATH=/opt/DL/tensorflow/lib/python2.7/site-packages
여기에 추가로 bazel의 PATH도 잡아줘야 합니다. Tensorflow의 build를 위해서는 먼저 bazel을 build해야 하는데, 사실 그것이 tensorflow build보다 더 힘듭니다. 여기서는 PowerAI에 포함된 bazel을 간편하게 사용하겠습니다.
test@k8002:~/tensorflow$ export PATH=/opt/DL/bazel/bin:$PATH
이제 configure부터 시작하여 build에 들어갑니다.
test@k8002:~/tensorflow$ ./configure
Please specify the location of python. [Default is /usr/bin/python]:
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:
Do you wish to use jemalloc as the malloc implementation? [Y/n]
jemalloc enabled
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N]
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N]
No Hadoop File System support will be enabled for TensorFlow
Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N]
No XLA support will be enabled for TensorFlow
Found possible Python library paths:
/opt/DL/tensorflow/lib/python2.7/site-packages
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
Please input the desired Python library path to use. Default is [/opt/DL/tensorflow/lib/python2.7/site-packages]
/usr/lib/python2.7/dist-packages
Do you wish to build TensorFlow with OpenCL support? [y/N]
No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with CUDA support? [y/N] y
CUDA support will be enabled for TensorFlow
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Please specify the CUDA SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 8.0
Please specify the location where CUDA 8.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /usr/local/cuda-8.0
Please specify the Cudnn version you want to use. [Leave empty to use system default]: 5
Please specify the location where cuDNN 5 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda-8.0]: /usr/lib/powerpc64le-linux-gnu
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]: 6.0 (--> P100은 6.0, K80의 경우 3.7)
Extracting Bazel installation...
.............
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
............
WARNING: /home/test/.cache/bazel/_bazel_test/3a51eda24c560ebddf29b66b8fa0460e/external/protobuf/protobuf.bzl:90:19: Variables HOST_CFG and DATA_CFG are deprecated in favor of strings "host" and "data" correspondingly.
WARNING: /home/test/.cache/bazel/_bazel_test/3a51eda24c560ebddf29b66b8fa0460e/external/protobuf/protobuf.bzl:96:28: Variables HOST_CFG and DATA_CFG are deprecated in favor of strings "host" and "data" correspondingly.
INFO: Downloading http://pilotfiber.dl.sourceforge.net/project/boost/boost/1.6\
1.0/boost_1_61_0.tar.gz: 104,733,800 bytes
...
INFO: All external dependencies fetched successfully.
Configuration finished
configure가 끝나면 이제 bazel build를 합니다.
test@k8002:~/tensorflow$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
...
Target //tensorflow/tools/pip_package:build_pip_package up-to-date:
bazel-bin/tensorflow/tools/pip_package/build_pip_package
INFO: Elapsed time: 256.806s, Critical Path: 203.45s
윗 단계가 성공적으로 끝나면 다음과 같이 /tmp/tensorflow_pkg directory에 pip package를 build합니다.
test@k8002:~/tensorflow$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
Thu Mar 16 15:03:32 KST 2017 : === Using tmpdir: /tmp/tmp.LCBvnwu8ZZ
~/tensorflow/bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles ~/tensorflow
~/tensorflow
/tmp/tmp.LCBvnwu8ZZ ~/tensorflow
Thu Mar 16 15:03:33 KST 2017 : === Building wheel
warning: no files found matching '*.dll' under directory '*'
warning: no files found matching '*.lib' under directory '*'
~/tensorflow
Thu Mar 16 15:03:44 KST 2017 : === Output wheel file is in: /tmp/tensorflow_pkg
이제 /tmp/tensorflow_pkg directory에 생성된 wheel file을 이용하여 pip install을 하면 tensorflow 설치가 완료됩니다.
test@k8002:~/tensorflow$ sudo pip install /tmp/tensorflow_pkg/tensorflow-1.0.1-cp27-cp27mu-linux_ppc64le.whl
...
Downloading funcsigs-1.0.2-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): pbr>=0.11 in /usr/lib/python2.7/dist-packages (from mock>=2.0.0->tensorflow==1.0.1)
Installing collected packages: werkzeug, protobuf, funcsigs, mock, tensorflow
Found existing installation: protobuf 2.6.1
Not uninstalling protobuf at /usr/lib/python2.7/dist-packages, outside environment /usr
Found existing installation: funcsigs 0.4
Not uninstalling funcsigs at /usr/lib/python2.7/dist-packages, outside environment /usr
Found existing installation: mock 1.3.0
Not uninstalling mock at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed funcsigs-1.0.2 mock-2.0.0 protobuf-3.2.0 tensorflow-1.0.1 werkzeug-0.12.1
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
이제 tensorflow의 버전을 확인해보겠습니다.
test@k8002:~/tensorflow$ pip show tensorflow
---
Metadata-Version: 2.0
Name: tensorflow
Version: 0.12.0
Summary: TensorFlow helps the tensors flow
Home-page: http://tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
Installer: pip
License: Apache 2.0
Location: /opt/DL/tensorflow/lib/python2.7/site-packages
Requires: numpy, mock, six, protobuf, wheel
Classifiers:
Development Status :: 4 - Beta
Intended Audience :: Developers
Intended Audience :: Education
Intended Audience :: Science/Research
License :: OSI Approved :: Apache Software License
Programming Language :: Python :: 2.7
Topic :: Scientific/Engineering :: Mathematics
Topic :: Software Development :: Libraries :: Python Modules
Topic :: Software Development :: Libraries
Entry-points:
[console_scripts]
tensorboard = tensorflow.tensorboard.tensorboard:main
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
위와 같이 version이 방금 설치한 v1.0이 아니라 v0.12로 나오는 것은 환경변수 PYTHONPATH가 아직 /opt/DL/tensorflow/lib/python2.7/site-packages으로 잡혀 있기 때문에, 이전에 PowerAI 설치 때 함께 설치된 tensorflow 0.12가 보이는 것입니다. 이를 configure할 때 주었던 값인 /usr/local/lib/python2.7/dist-packages으로 바꾼 뒤 다시 해보겠습니다.
test@k8002:~/tensorflow$ export PYTHONPATH=/usr/local/lib/python2.7/dist-packages:$PYTHONPATH
test@k8002:~/tensorflow$ pip show tensorflow
---
Metadata-Version: 2.0
Name: tensorflow
Version: 1.0.1
Summary: TensorFlow helps the tensors flow
Home-page: http://tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
Installer: pip
License: Apache 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: werkzeug, six, protobuf, wheel, numpy, mock
Classifiers:
Development Status :: 4 - Beta
Intended Audience :: Developers
Intended Audience :: Education
Intended Audience :: Science/Research
License :: OSI Approved :: Apache Software License
Programming Language :: Python :: 2.7
Topic :: Scientific/Engineering :: Mathematics
Topic :: Software Development :: Libraries :: Python Modules
Topic :: Software Development :: Libraries
Entry-points:
[console_scripts]
tensorboard = tensorflow.tensorboard.tensorboard:main
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
위와 같이 v1.0.1로 제대로 보이는 것을 보실 수 있습니다.
이번에는 r0.12를 build해보겠습니다.
test@k8002:~/tensorflow$ git checkout r0.12
Switched to branch 'r0.12'
Your branch is up-to-date with 'origin/r0.12'.
동일하게, configure부터 시작하여 build에 들어갑니다.
test@k8002:~/tensorflow$ ./configure
~/tensorflow ~/tensorflow
Please specify the location of python. [Default is /usr/bin/python]:
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N]
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N]
No Hadoop File System support will be enabled for TensorFlow
Found possible Python library paths:
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
Please input the desired Python library path to use. Default is [/usr/local/lib/python2.7/dist-packages]
/usr/lib/python2.7/dist-packages
Do you wish to build TensorFlow with OpenCL support? [y/N]
No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with GPU support? [y/N] y
GPU support will be enabled for TensorFlow
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 8.0
Please specify the location where CUDA 8.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /usr/local/cuda-8.0
Please specify the Cudnn version you want to use. [Leave empty to use system default]: 5
Please specify the location where cuDNN 5 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda-8.0]: /usr/lib/powerpc64le-linux-gnu
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]: 6.0 (--> P100은 6.0, K80의 경우 3.7)
............
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
............
INFO: All external dependencies fetched successfully.
Configuration finished
test@k8002:~/tensorflow$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
... ^
At global scope:
cc1plus: warning: unrecognized command line option '-Wno-self-assign'
Target //tensorflow/tools/pip_package:build_pip_package up-to-date:
bazel-bin/tensorflow/tools/pip_package/build_pip_package
INFO: Elapsed time: 306.555s, Critical Path: 273.68s
test@k8002:~/tensorflow$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
Fri Mar 17 16:16:24 KST 2017 : === Using tmpdir: /tmp/tmp.L1dwTgVhwv
~/tensorflow/bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles ~/tensorflow
~/tensorflow
/tmp/tmp.L1dwTgVhwv ~/tensorflow
Fri Mar 17 16:16:25 KST 2017 : === Building wheel
~/tensorflow
Fri Mar 17 16:16:41 KST 2017 : === Output wheel file is in: /tmp/tensorflow_pkg
이제 /tmp/tensorflow_pkg를 보시면 아까 build했던 1.0.1 버전의 wheel file과 함께 새로 0.12 버전의 wheel file이 만들어진 것을 보실 수 있습니다.
test@k8002:~/tensorflow$ ls -l /tmp/tensorflow_pkg/
total 109520
-rw-rw-r-- 1 test test 61576804 Mar 17 16:16 tensorflow-0.12.1-cp27-cp27mu-linux_ppc64le.whl
-rw-rw-r-- 1 test test 50565190 Mar 16 15:03 tensorflow-1.0.1-cp27-cp27mu-linux_ppc64le.whl
그것을 pip install 명령으로 설치합니다. 아래와 같이, 기존의 tensorflow-1.0.1을 uninstall하면서 0.12가 설치됩니다.
test@k8002:~/tensorflow$ sudo pip install /tmp/tensorflow_pkg/tensorflow-0.12.1-cp27-cp27mu-linux_ppc64le.whl
The directory '/home/test/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/test/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /tmp/tensorflow_pkg/tensorflow-0.12.1-cp27-cp27mu-linux_ppc64le.whl
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.11.0 in /home/test/.local/lib/python2.7/site-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): wheel in /usr/lib/python2.7/dist-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): mock>=2.0.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): six>=1.10.0 in /home/test/.local/lib/python2.7/site-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): protobuf>=3.1.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): funcsigs>=1; python_version < "3.3" in /usr/local/lib/python2.7/dist-packages (from mock>=2.0.0->tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): pbr>=0.11 in /usr/lib/python2.7/dist-packages (from mock>=2.0.0->tensorflow==0.12.1)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/lib/python2.7/dist-packages (from protobuf>=3.1.0->tensorflow==0.12.1)
Installing collected packages: tensorflow
Found existing installation: tensorflow 1.0.1
Uninstalling tensorflow-1.0.1:
Successfully uninstalled tensorflow-1.0.1
Successfully installed tensorflow-0.12.1
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
다음과 같이 tensorflow의 버전이 0.12.1로 변경된 것을 확인하실 수 있습니다.
test@k8002:~/tensorflow$ pip show tensorflow
---
Metadata-Version: 2.0
Name: tensorflow
Version: 0.12.1
Summary: TensorFlow helps the tensors flow
Home-page: http://tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
Installer: pip
License: Apache 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: numpy, wheel, mock, six, protobuf
Classifiers:
Development Status :: 4 - Beta
Intended Audience :: Developers
Intended Audience :: Education
Intended Audience :: Science/Research
License :: OSI Approved :: Apache Software License
Programming Language :: Python :: 2.7
Topic :: Scientific/Engineering :: Mathematics
Topic :: Software Development :: Libraries :: Python Modules
Topic :: Software Development :: Libraries
Entry-points:
[console_scripts]
tensorboard = tensorflow.tensorboard.tensorboard:main
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
피드 구독하기:
글 (Atom)