올 3월까지만 해도 Continuum사의 Anaconda installer가 Minsky의 아키텍처인 ppc64le 환경에서는 full package로는 없었고, Miniconda installer만 있었습니다. 그런데 그 사이에 ppc64le용 Anaconda installer의 full package가 이미 나왔네요.
https://repo.continuum.io/archive/
Filename Size Last Modified MD5
Anaconda2-4.4.0.1-Linux-ppc64le.sh 271.4M 2017-07-26 16:10:02 ce166de6f116acd08cd313f9c55c04d6
Anaconda3-4.4.0.1-Linux-ppc64le.sh 285.6M 2017-07-26 16:08:42 fe7c87abd9fd70dc0cb4f83cc22d336f
설치는 아래와 같이 하시면 됩니다. Python2.7 환경에서는 아래와 같이 Anaconda2-4.4.0.1-Linux-ppc64le.sh를 받으시면 되고, Python3 환경에서는 Anaconda3-4.4.0.1-Linux-ppc64le.sh를 받으시면 됩니다. 일단 둘다 받아서 설치하시는 것이 좋습니다. 여기서는 Anaconda2에 대한 부분만 써놓습니다.
u0017649@sys-89075:~$ wget https://repo.continuum.io/archive/Anaconda2-4.4.0.1-Linux-ppc64le.sh
--2017-09-14 21:42:44-- https://repo.continuum.io/archive/Anaconda2-4.4.0.1-Linux-ppc64le.sh
Resolving repo.continuum.io (repo.continuum.io)... 104.16.18.10, 104.16.19.10, 2400:cb00:2048:1::6810:120a, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.16.18.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 284629257 (271M) [application/x-sh]
Saving to: ‘Anaconda2-4.4.0.1-Linux-ppc64le.sh’
Anaconda2-4.4.0.1-L 100%[===================>] 271.44M 16.7MB/s in 17s
2017-09-14 21:43:01 (16.4 MB/s) - ‘Anaconda2-4.4.0.1-Linux-ppc64le.sh’ saved [284629257/284629257]
여러 user가 함께 쓰는 서버 환경에 Anaconda를 설치할 때는, 해당 user의 home directory에 설치하면 다른 user들이 쓸 수가 없지요. 따라서 일단은 /opt/anaconda2에 설치해놓고 user별로 clone을 해서 사용하시는 것이 좋습니다. 물론 /opt directory에 설치하려면 sudo 권한이 있어야 합니다.
u0017649@sys-89075:~$ chmod u+x Anaconda2-4.4.0.1-Linux-ppc64le.sh
u0017649@sys-89075:~$ sudo ./Anaconda2-4.4.0.1-Linux-ppc64le.sh
Do you approve the license terms? [yes|no]
>>> yes
Anaconda2 will now be installed into this location:
/home/u0017649/anaconda2
- Press ENTER to confirm the location
- Press CTRL-C to abort the installation
- Or specify a different location below
[/home/u0017649/anaconda2] >>> /opt/anaconda2
...
installing: zict-0.1.2-py27_0 ...
installing: zlib-1.2.8-3 ...
installing: anaconda-4.4.0-np112py27_0 ...
installing: conda-4.3.21-py27_0 ...
installing: conda-env-2.6.0-0 ...
Python 2.7.13 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
Do you wish the installer to prepend the Anaconda2 install location
to PATH in your /home/u0017649/.bashrc ? [yes|no]
[no] >>> yes
Prepending PATH=/opt/anaconda2/bin to PATH in /home/u0017649/.bashrc
A backup will be made to: /home/u0017649/.bashrc-anaconda2.bak
For this change to become active, you have to open a new terminal.
Thank you for installing Anaconda2!
Share your notebooks and packages on Anaconda Cloud!
Sign up for free: https://anaconda.org
이제 환경변수를 적용하고, conda install을 해봅니다.
u0017649@sys-89075:~$ . ./.bashrc
u0017649@sys-89075:~$ env | grep PATH
PATH=/opt/anaconda2/bin:/home/u0017649/bin:/home/u0017649/.local/bin:/opt/ibm/xlC/current/bin:/opt/ibm/xlf/current/bin:/opt/at10.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
u0017649@sys-89075:~$ conda install scikit-learn
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /opt/anaconda2:
The following packages will be UPDATED:
scikit-learn: 0.18.1-np112py27_1 --> 0.19.0-np112py27_0
Proceed ([y]/n)? y
CondaIOError: Missing write permissions in: /opt/anaconda2
#
# You don't appear to have the necessary permissions to install packages
# into the install area '/opt/anaconda2'.
# However you can clone this environment into your home directory and
# then make changes to it.
# This may be done using the command:
#
# $ conda create -n my_root --clone="/opt/anaconda2"
보시다시피 /opt/anaconda2에 대해서 일반 user들은 write 권한이 없기 때문에 error가 납니다. 그래서 각 user별로 cloning을 해서 사용하는 것입니다. 아래와 같이 하시면 됩니다.
u0017649@sys-89075:~$ conda create -n anaconda2 --clone="/opt/anaconda2"
Source: /opt/anaconda2
Destination: /home/u0017649/.conda/envs/anaconda2
The following packages cannot be cloned out of the root environment:
- conda-4.3.21-py27_0
- conda-env-2.6.0-0
Packages: 186
Files: 11
#
# To activate this environment, use:
# > source activate anaconda2
#
# To deactivate this environment, use:
# > source deactivate anaconda2
#
이제 cloning 환경으로 들어가서 위에서 실패했던 scikit-learn 패키지의 설치를 다시 해보겠습니다.
u0017649@sys-89075:~$ source activate anaconda2
(anaconda2) u0017649@sys-89075:~$ conda install scikit-learn
이건 잘 됩니다. 이것이 잘 되는 이유는 다음과 같이 각 user의 home directory 밑인 ~/.conda/envs에 cloning을 해놓고 쓰기 때문입니다. 물론 이는 disk 저장공간을 사용하는 작업이므로, 많은 user들이 각자 모두 cloning을 할 경우 disk 공간의 낭비가 꽤 있게 되는 문제가 있긴 합니다. 가령 Anaconda2를 1번 clone할 때마다 약 1.1 GB의 공간이 소모됩니다. 따라서 모든 user가 일괄적으로 이렇게 cloning을 할 필요는 없고, 정말 꼭 자신만의 변경된 package 환경을 가져야 할 경우에만 하실 것을 권고합니다.
(anaconda2) u0017649@sys-89075:~/.conda/envs/anaconda2$ du -sm .
1135 .
2017년 9월 15일 금요일
2017년 6월 14일 수요일
ppc64le 환경에서의 anaconda python2&3 환경 동시 설정 - XGBoost, OpenCV, KoNLPy, MeCab 등
먼저, ppc64le 환경에서의 miniconda 설치에 대해서는 아래 post를 참조하십시요.
http://hwengineer.blogspot.kr/2017/05/minsky-continuum-anaconda.html
여기서는 miniconda3 (python3.6)이 이미 설치된 환경에서 시작하며, ppc64le 환경에서 anaconda를 사용하실 때 흔히 있을 수 있는 질문과 응답 형식으로 정리했습니다.
Q1. Anaconda 대신 Miniconda3로 가야하는 경우에는 다음 패키지도 conda install로 설치 가능한지 ?
numpy, pandas, sklearn(scikit-learn), joblib, flask, bs4(beautiful soup), MKL(math kernel library),
A1. 예, 아래와 같이 다 conda로 install 잘 됩니다. 단 하나, MKL은 ppc64le에는 없습니다만, 그와 수반되어 사용되는 numpy 및 scipy는 아래와 같이 conda로 잘 설치되므로 꼭 MKL이 있어야 할 필요는 없습니다.
u0017496@sys-87576:~$ conda install joblib
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
joblib: 0.11-py36_0
Proceed ([y]/n)? y
joblib-0.11-py 100% |#########################################################| Time: 0:00:00 7.17 MB/s
u0017496@sys-87576:~$ conda install scikit-learn
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
scikit-learn: 0.18.1-np112py36_1
Proceed ([y]/n)? y
scikit-learn-0 100% |#########################################################| Time: 0:00:00 14.48 MB/s
u0017496@sys-87576:~$ conda install pandas
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
pandas: 0.20.1-np112py36_0
python-dateutil: 2.6.0-py36_0
pytz: 2017.2-py36_0
Proceed ([y]/n)? y
pytz-2017.2-py 100% |#########################################################| Time: 0:00:00 7.85 MB/s
python-dateuti 100% |#########################################################| Time: 0:00:00 8.43 MB/s
pandas-0.20.1- 100% |#########################################################| Time: 0:00:02 10.59 MB/s
u0017496@sys-87576:~$ conda install flask
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
click: 6.7-py36_0
flask: 0.12.2-py36_0
itsdangerous: 0.24-py36_0
jinja2: 2.9.6-py36_0
markupsafe: 0.23-py36_2
werkzeug: 0.12.2-py36_0
Proceed ([y]/n)? y
click-6.7-py36 100% |#########################################################| Time: 0:00:00 7.14 MB/s
itsdangerous-0 100% |#########################################################| Time: 0:00:00 11.12 MB/s
markupsafe-0.2 100% |#########################################################| Time: 0:00:00 15.42 MB/s
werkzeug-0.12. 100% |#########################################################| Time: 0:00:00 5.71 MB/s
jinja2-2.9.6-p 100% |#########################################################| Time: 0:00:00 14.30 MB/s
flask-0.12.2-p 100% |#########################################################| Time: 0:00:00 7.08 MB/s
u0017496@sys-87576:~$ conda install beautifulsoup4
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
beautifulsoup4: 4.6.0-py36_0
Proceed ([y]/n)? y
beautifulsoup4 100% |#########################################################| Time: 0:00:00 7.12 MB/s
u0017496@sys-87576:~$ conda install numpy scipy
Fetching package metadata .........
Solving package specifications: .
# All requested packages already installed.
# packages in environment at /home/u0017496/miniconda3:
#
numpy 1.12.1 py36_0
scipy 0.19.0 np112py36_0
Q2. Python2 & 3을 위해서 Anaconda2, Anaconda3을 각각 설치하는 대신에 아래 URL에 나오는 것처럼 Anaconda3 하나만 설치하고 그 내부에 python2 가상환경 구축 가능한지?
www.continuum.io/blog/developer-blog/python-3-support-anaconda
A2. 예, 잘 됩니다.
u0017496@sys-87576:~$ conda create -n py2k python=2 anaconda
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3/envs/py2k:
The following NEW packages will be INSTALLED:
alabaster: 0.7.10-py27_0
anaconda: 4.4.0-np112py27_0
anaconda-client: 1.6.3-py27_0
....
...
jupyter-1.0.0- 100% |#########################################################| Time: 0:00:00 10.50 MB/s
anaconda-4.4.0 100% |#########################################################| Time: 0:00:00 5.53 MB/s
#
# To activate this environment, use:
# > source activate py2k
#
# To deactivate this environment, use:
# > source deactivate py2k
#
py2k, 즉 python2 가상환경에 들어가려면 다음과 같이 하시면 됩니다.
u0017496@sys-87576:~$ source activate py2k
python이 이제 2.7.13 버전이 구동되는 것을 보실 수 있습니다.
(py2k) u0017496@sys-87576:~$ python
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Mar 16 2017, 18:34:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>>
Q3. XGBoost의 경우, pip install을 통한 설치는 CPU 버전 패키지만 지원합니다.
GPU 사용가능한 XGBoost는 아래 URL의 Github 소스코드 + CUB 관련 파일 컴파일이 필요한데 ppc64le에서도 가능한지?
github.com/dmlc/xgboost/tree/master/plugin/updater_gpu
A3. 예, 잘 됩니다. git clone으로 위의 package를 download 받아 그 안의 다음 4개의 Makefile에서 intel-specifc한 option인 -msse2만 제거하면 잘 됩니다.
u0017496@sys-87576:~/$ git clone --recursive https://github.com/dmlc/xgboost.git
u0017496@sys-87576:~/$ cd xgboost
u0017496@sys-87576:~/xgboost$ vi ./rabit/guide/Makefile ./rabit/Makefile ./dmlc-core/Makefile ./Makefile
u0017496@sys-87576:~/xgboost$ ./build.sh
...
g++ -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -mcpu=power8 -fPIC -fopenmp -o xgboost build/cli_main.o build/logging.o build/learner.o build/common/common.o build/common/hist_util.o build/metric/metric.o build/metric/rank_metric.o build/metric/elementwise_metric.o build/metric/multiclass_metric.o build/objective/multiclass_obj.o build/objective/objective.o build/objective/rank_obj.o build/objective/regression_obj.o build/data/sparse_page_dmatrix.o build/data/sparse_page_source.o build/data/sparse_page_writer.o build/data/simple_csr_source.o build/data/data.o build/data/sparse_page_raw_format.o build/data/simple_dmatrix.o build/tree/updater_prune.o build/tree/tree_updater.o build/tree/updater_histmaker.o build/tree/updater_refresh.o build/tree/updater_sync.o build/tree/updater_colmaker.o build/tree/updater_skmaker.o build/tree/tree_model.o build/tree/updater_fast_hist.o build/gbm/gbtree.o build/gbm/gblinear.o build/gbm/gbm.o build/c_api/c_api.o build/c_api/c_api_error.o dmlc-core/libdmlc.a rabit/lib/librabit.a -pthread -lm -fopenmp -lrt -lrt
Successfully build multi-thread xgboost
Q4. konlpy는 JDK 및 Jpype1 관련 설정 및 PATH, JAVA_HOME 등등 경로 지정 필요합니다. python2, python3에서 모두 잘 작동하는지 ?
A4. 잘 됩니다.
먼저 기본 환경인 python3에서 다음과 같이 잘 됩니다.
u0017496@sys-87576:~/xgboost$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el
u0017496@sys-87576:~/xgboost$ python
Python 3.6.0 |Continuum Analytics, Inc.| (default, Mar 16 2017, 19:36:14)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from konlpy.tag import *
>>> Kkma()
<konlpy.tag._kkma.Kkma instance at 0x3fff9561be60>
이어서 위에서 설치했던 가상환경의 python2에서도 잘 됩니다. 다만 환경 변수 등은 가상환경 속에서 다시 또 해줘야 합니다.
u0017496@sys-87576:~$ source activate py2k
(py2k) u0017496@sys-87576:~$ python
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Mar 16 2017, 18:34:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from konlpy.tag import *
>>> Kkma()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: Kkma instance has no __call__ method
위에서 error가 난 이유는 JAVA_HOME 설정이 이 가상환경 속에서는 안 되어 있기 때문입니다. 그걸 해주면 error는 없어집니다.
(py2k) u0017496@sys-87576:~$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el
(py2k) u0017496@sys-87576:~$ python
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Mar 16 2017, 18:34:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from konlpy.tag import *
>>> Kkma()
<konlpy.tag._kkma.Kkma instance at 0x3fff9561be60>
Q5. KoNLPy의 mecab 추가 설치 및 작동이 잘 되는지 ? 가령 아래 code가 python2, python3에서 모두 잘 되는지 ?
from konlpy.tag import *
Mecab()
A5. 다음과 같이 잘 됩니다. 처음에는 error가 나서 당황했는데, 보니 mecab-ko-dic를 따로 설치하면 해결되는 문제입니다.
먼저 mecab-python3을 pip로 설치하고, 그 뒤에 mecab-ko-dic을 download 받아 설치합니다.
u0017496@sys-87576:~$ pip install mecab-python3
u0017496@sys-87576:~$ wget https://bitbucket.org/eunjeon/mecab-ko-dic/downloads/mecab-ko-dic-1.6.1-20140814.tar.gz
u0017496@sys-87576:~$ tar -zxvf mecab-ko-dic-1.6.1-20140814.tar.gz
u0017496@sys-87576:~$ cd mecab-ko-dic-1.6.1-20140814
u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ ./autogen.sh
u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ ./configure
u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ make
u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ sudo make install
u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ python
Python 3.6.0 |Continuum Analytics, Inc.| (default, Mar 16 2017, 19:36:14)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from konlpy.tag import *
>>> Mecab('/usr/lib/mecab/dic/mecab-ko-dic')
<konlpy.tag._mecab.Mecab object at 0x3fffa01773c8>
python2의 가상환경에서도 물론 동일하게 잘 됩니다.
u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ source activate py2k
(py2k) u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ python
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Mar 16 2017, 18:34:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from konlpy.tag import *
>>> Mecab('/usr/lib/mecab/dic/mecab-ko-dic')
<konlpy.tag._mecab.Mecab instance at 0x3fff754e7cb0>
>>>
Q5. OpenCV & GraphViz의 경우, conda install 을 사용하지 않고 설치가 가능하다면, python2&3 안에서 호출 가능한 형태로 설치되는 것인지?
import cv2
import graphviz
A6. 예, 둘다 잘 됩니다. https://repo.continuum.io/pkgs/free/linux-ppc64le에서 opencv-3.1.0-np112py27_2.tar.bz2 등의 opencv package를 받아와서 설치하면 됩니다.
먼저 python3를 위한 opencv package를 설치해서 테스트합니다.
u0017496@sys-87576:~$ wget https://repo.continuum.io/pkgs/free/linux-ppc64le/opencv-3.1.0-np112py36_2.tar.bz2
--2017-06-14 03:35:25-- https://repo.continuum.io/pkgs/free/linux-ppc64le/opencv-3.1.0-np112py36_2.tar.bz2
Resolving repo.continuum.io (repo.continuum.io)... 104.16.19.10, 104.16.18.10, 2400:cb00:2048:1::6810:120a, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.16.19.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13145265 (13M) [application/x-tar]
Saving to: ‘opencv-3.1.0-np112py36_2.tar.bz2’
opencv-3.1.0-np112py36_2.t 100%[=====================================>] 12.54M 9.02MB/s in 1.4s
2017-06-14 03:35:26 (9.02 MB/s) - ‘opencv-3.1.0-np112py36_2.tar.bz2’ saved [13145265/13145265]
u0017496@sys-87576:~$ mkdir opencv-3.1.0-py36
u0017496@sys-87576:~$ cd opencv-3.1.0-py36
u0017496@sys-87576:~/opencv-3.1.0-py36$ tar -jxvf ../opencv-3.1.0-np112py36_2.tar.bz2
u0017496@sys-87576:~$ export PYTHONPATH=$PYTHONPATH:/home/u0017496/opencv-3.1.0-py36/lib/python3.6/site-packages
u0017496@sys-87576:~$ python
Python 3.6.0 |Continuum Analytics, Inc.| (default, Mar 16 2017, 19:36:14)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> import graphviz
>>>
다음으로 python2를 위한 opencv package를 설치해서 테스트합니다.
u0017496@sys-87576:~$ wget https://repo.continuum.io/pkgs/free/linux-ppc64le/opencv-3.1.0-np112py27_2.tar.bz2
--2017-06-14 03:35:38-- https://repo.continuum.io/pkgs/free/linux-ppc64le/opencv-3.1.0-np112py27_2.tar.bz2
Resolving repo.continuum.io (repo.continuum.io)... 104.16.18.10, 104.16.19.10, 2400:cb00:2048:1::6810:130a, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.16.18.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13147104 (13M) [application/x-tar]
Saving to: ‘opencv-3.1.0-np112py27_2.tar.bz2’
opencv-3.1.0-np112py27_2.t 100%[=====================================>] 12.54M 9.76MB/s in 1.3s
2017-06-14 03:35:39 (9.76 MB/s) - ‘opencv-3.1.0-np112py27_2.tar.bz2’ saved [13147104/13147104]
u0017496@sys-87576:~$ mkdir opencv-3.1.0-py27
u0017496@sys-87576:~$ cd opencv-3.1.0-py27
u0017496@sys-87576:~/opencv-3.1.0-py27$ tar -jxvf ../opencv-3.1.0-np112py27_2.tar.bz2
u0017496@sys-87576:~$ source activate py2k
(py2k) u0017496@sys-87576:~$ export PYTHONPATH=$PYTHONPATH:/home/u0017496/opencv-3.1.0-py27/lib/python2.7/site-packages
(py2k) u0017496@sys-87576:~$ python
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Mar 16 2017, 18:34:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import cv2
>>> import graphviz
>>>
http://hwengineer.blogspot.kr/2017/05/minsky-continuum-anaconda.html
여기서는 miniconda3 (python3.6)이 이미 설치된 환경에서 시작하며, ppc64le 환경에서 anaconda를 사용하실 때 흔히 있을 수 있는 질문과 응답 형식으로 정리했습니다.
Q1. Anaconda 대신 Miniconda3로 가야하는 경우에는 다음 패키지도 conda install로 설치 가능한지 ?
numpy, pandas, sklearn(scikit-learn), joblib, flask, bs4(beautiful soup), MKL(math kernel library),
A1. 예, 아래와 같이 다 conda로 install 잘 됩니다. 단 하나, MKL은 ppc64le에는 없습니다만, 그와 수반되어 사용되는 numpy 및 scipy는 아래와 같이 conda로 잘 설치되므로 꼭 MKL이 있어야 할 필요는 없습니다.
u0017496@sys-87576:~$ conda install joblib
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
joblib: 0.11-py36_0
Proceed ([y]/n)? y
joblib-0.11-py 100% |#########################################################| Time: 0:00:00 7.17 MB/s
u0017496@sys-87576:~$ conda install scikit-learn
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
scikit-learn: 0.18.1-np112py36_1
Proceed ([y]/n)? y
scikit-learn-0 100% |#########################################################| Time: 0:00:00 14.48 MB/s
u0017496@sys-87576:~$ conda install pandas
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
pandas: 0.20.1-np112py36_0
python-dateutil: 2.6.0-py36_0
pytz: 2017.2-py36_0
Proceed ([y]/n)? y
pytz-2017.2-py 100% |#########################################################| Time: 0:00:00 7.85 MB/s
python-dateuti 100% |#########################################################| Time: 0:00:00 8.43 MB/s
pandas-0.20.1- 100% |#########################################################| Time: 0:00:02 10.59 MB/s
u0017496@sys-87576:~$ conda install flask
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
click: 6.7-py36_0
flask: 0.12.2-py36_0
itsdangerous: 0.24-py36_0
jinja2: 2.9.6-py36_0
markupsafe: 0.23-py36_2
werkzeug: 0.12.2-py36_0
Proceed ([y]/n)? y
click-6.7-py36 100% |#########################################################| Time: 0:00:00 7.14 MB/s
itsdangerous-0 100% |#########################################################| Time: 0:00:00 11.12 MB/s
markupsafe-0.2 100% |#########################################################| Time: 0:00:00 15.42 MB/s
werkzeug-0.12. 100% |#########################################################| Time: 0:00:00 5.71 MB/s
jinja2-2.9.6-p 100% |#########################################################| Time: 0:00:00 14.30 MB/s
flask-0.12.2-p 100% |#########################################################| Time: 0:00:00 7.08 MB/s
u0017496@sys-87576:~$ conda install beautifulsoup4
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
beautifulsoup4: 4.6.0-py36_0
Proceed ([y]/n)? y
beautifulsoup4 100% |#########################################################| Time: 0:00:00 7.12 MB/s
u0017496@sys-87576:~$ conda install numpy scipy
Fetching package metadata .........
Solving package specifications: .
# All requested packages already installed.
# packages in environment at /home/u0017496/miniconda3:
#
numpy 1.12.1 py36_0
scipy 0.19.0 np112py36_0
Q2. Python2 & 3을 위해서 Anaconda2, Anaconda3을 각각 설치하는 대신에 아래 URL에 나오는 것처럼 Anaconda3 하나만 설치하고 그 내부에 python2 가상환경 구축 가능한지?
www.continuum.io/blog/developer-blog/python-3-support-anaconda
A2. 예, 잘 됩니다.
u0017496@sys-87576:~$ conda create -n py2k python=2 anaconda
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3/envs/py2k:
The following NEW packages will be INSTALLED:
alabaster: 0.7.10-py27_0
anaconda: 4.4.0-np112py27_0
anaconda-client: 1.6.3-py27_0
....
...
jupyter-1.0.0- 100% |#########################################################| Time: 0:00:00 10.50 MB/s
anaconda-4.4.0 100% |#########################################################| Time: 0:00:00 5.53 MB/s
#
# To activate this environment, use:
# > source activate py2k
#
# To deactivate this environment, use:
# > source deactivate py2k
#
py2k, 즉 python2 가상환경에 들어가려면 다음과 같이 하시면 됩니다.
u0017496@sys-87576:~$ source activate py2k
python이 이제 2.7.13 버전이 구동되는 것을 보실 수 있습니다.
(py2k) u0017496@sys-87576:~$ python
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Mar 16 2017, 18:34:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>>
Q3. XGBoost의 경우, pip install을 통한 설치는 CPU 버전 패키지만 지원합니다.
GPU 사용가능한 XGBoost는 아래 URL의 Github 소스코드 + CUB 관련 파일 컴파일이 필요한데 ppc64le에서도 가능한지?
github.com/dmlc/xgboost/tree/master/plugin/updater_gpu
A3. 예, 잘 됩니다. git clone으로 위의 package를 download 받아 그 안의 다음 4개의 Makefile에서 intel-specifc한 option인 -msse2만 제거하면 잘 됩니다.
u0017496@sys-87576:~/$ git clone --recursive https://github.com/dmlc/xgboost.git
u0017496@sys-87576:~/$ cd xgboost
u0017496@sys-87576:~/xgboost$ ./build.sh
...
g++ -std=c++11 -Wall -Wno-unknown-pragmas -Iinclude -Idmlc-core/include -Irabit/include -I/include -O3 -funroll-loops -mcpu=power8 -fPIC -fopenmp -o xgboost build/cli_main.o build/logging.o build/learner.o build/common/common.o build/common/hist_util.o build/metric/metric.o build/metric/rank_metric.o build/metric/elementwise_metric.o build/metric/multiclass_metric.o build/objective/multiclass_obj.o build/objective/objective.o build/objective/rank_obj.o build/objective/regression_obj.o build/data/sparse_page_dmatrix.o build/data/sparse_page_source.o build/data/sparse_page_writer.o build/data/simple_csr_source.o build/data/data.o build/data/sparse_page_raw_format.o build/data/simple_dmatrix.o build/tree/updater_prune.o build/tree/tree_updater.o build/tree/updater_histmaker.o build/tree/updater_refresh.o build/tree/updater_sync.o build/tree/updater_colmaker.o build/tree/updater_skmaker.o build/tree/tree_model.o build/tree/updater_fast_hist.o build/gbm/gbtree.o build/gbm/gblinear.o build/gbm/gbm.o build/c_api/c_api.o build/c_api/c_api_error.o dmlc-core/libdmlc.a rabit/lib/librabit.a -pthread -lm -fopenmp -lrt -lrt
Successfully build multi-thread xgboost
Q4. konlpy는 JDK 및 Jpype1 관련 설정 및 PATH, JAVA_HOME 등등 경로 지정 필요합니다. python2, python3에서 모두 잘 작동하는지 ?
A4. 잘 됩니다.
먼저 기본 환경인 python3에서 다음과 같이 잘 됩니다.
u0017496@sys-87576:~/xgboost$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el
u0017496@sys-87576:~/xgboost$ python
Python 3.6.0 |Continuum Analytics, Inc.| (default, Mar 16 2017, 19:36:14)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from konlpy.tag import *
>>> Kkma()
<konlpy.tag._kkma.Kkma instance at 0x3fff9561be60>
이어서 위에서 설치했던 가상환경의 python2에서도 잘 됩니다. 다만 환경 변수 등은 가상환경 속에서 다시 또 해줘야 합니다.
u0017496@sys-87576:~$ source activate py2k
(py2k) u0017496@sys-87576:~$ python
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Mar 16 2017, 18:34:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from konlpy.tag import *
>>> Kkma()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: Kkma instance has no __call__ method
위에서 error가 난 이유는 JAVA_HOME 설정이 이 가상환경 속에서는 안 되어 있기 때문입니다. 그걸 해주면 error는 없어집니다.
(py2k) u0017496@sys-87576:~$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el
(py2k) u0017496@sys-87576:~$ python
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Mar 16 2017, 18:34:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from konlpy.tag import *
>>> Kkma()
<konlpy.tag._kkma.Kkma instance at 0x3fff9561be60>
Q5. KoNLPy의 mecab 추가 설치 및 작동이 잘 되는지 ? 가령 아래 code가 python2, python3에서 모두 잘 되는지 ?
from konlpy.tag import *
Mecab()
A5. 다음과 같이 잘 됩니다. 처음에는 error가 나서 당황했는데, 보니 mecab-ko-dic를 따로 설치하면 해결되는 문제입니다.
먼저 mecab-python3을 pip로 설치하고, 그 뒤에 mecab-ko-dic을 download 받아 설치합니다.
u0017496@sys-87576:~$ pip install mecab-python3
u0017496@sys-87576:~$ wget https://bitbucket.org/eunjeon/mecab-ko-dic/downloads/mecab-ko-dic-1.6.1-20140814.tar.gz
u0017496@sys-87576:~$ tar -zxvf mecab-ko-dic-1.6.1-20140814.tar.gz
u0017496@sys-87576:~$ cd mecab-ko-dic-1.6.1-20140814
u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ ./autogen.sh
u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ ./configure
u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ make
u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ sudo make install
u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ python
Python 3.6.0 |Continuum Analytics, Inc.| (default, Mar 16 2017, 19:36:14)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from konlpy.tag import *
>>> Mecab('/usr/lib/mecab/dic/mecab-ko-dic')
<konlpy.tag._mecab.Mecab object at 0x3fffa01773c8>
python2의 가상환경에서도 물론 동일하게 잘 됩니다.
u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ source activate py2k
(py2k) u0017496@sys-87576:~/mecab-ko-dic-1.6.1-20140814$ python
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Mar 16 2017, 18:34:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from konlpy.tag import *
>>> Mecab('/usr/lib/mecab/dic/mecab-ko-dic')
<konlpy.tag._mecab.Mecab instance at 0x3fff754e7cb0>
>>>
Q5. OpenCV & GraphViz의 경우, conda install 을 사용하지 않고 설치가 가능하다면, python2&3 안에서 호출 가능한 형태로 설치되는 것인지?
import cv2
import graphviz
A6. 예, 둘다 잘 됩니다. https://repo.continuum.io/pkgs/free/linux-ppc64le에서 opencv-3.1.0-np112py27_2.tar.bz2 등의 opencv package를 받아와서 설치하면 됩니다.
먼저 python3를 위한 opencv package를 설치해서 테스트합니다.
u0017496@sys-87576:~$ wget https://repo.continuum.io/pkgs/free/linux-ppc64le/opencv-3.1.0-np112py36_2.tar.bz2
--2017-06-14 03:35:25-- https://repo.continuum.io/pkgs/free/linux-ppc64le/opencv-3.1.0-np112py36_2.tar.bz2
Resolving repo.continuum.io (repo.continuum.io)... 104.16.19.10, 104.16.18.10, 2400:cb00:2048:1::6810:120a, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.16.19.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13145265 (13M) [application/x-tar]
Saving to: ‘opencv-3.1.0-np112py36_2.tar.bz2’
opencv-3.1.0-np112py36_2.t 100%[=====================================>] 12.54M 9.02MB/s in 1.4s
2017-06-14 03:35:26 (9.02 MB/s) - ‘opencv-3.1.0-np112py36_2.tar.bz2’ saved [13145265/13145265]
u0017496@sys-87576:~$ mkdir opencv-3.1.0-py36
u0017496@sys-87576:~$ cd opencv-3.1.0-py36
u0017496@sys-87576:~/opencv-3.1.0-py36$ tar -jxvf ../opencv-3.1.0-np112py36_2.tar.bz2
u0017496@sys-87576:~$ export PYTHONPATH=$PYTHONPATH:/home/u0017496/opencv-3.1.0-py36/lib/python3.6/site-packages
u0017496@sys-87576:~$ python
Python 3.6.0 |Continuum Analytics, Inc.| (default, Mar 16 2017, 19:36:14)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> import graphviz
>>>
다음으로 python2를 위한 opencv package를 설치해서 테스트합니다.
u0017496@sys-87576:~$ wget https://repo.continuum.io/pkgs/free/linux-ppc64le/opencv-3.1.0-np112py27_2.tar.bz2
--2017-06-14 03:35:38-- https://repo.continuum.io/pkgs/free/linux-ppc64le/opencv-3.1.0-np112py27_2.tar.bz2
Resolving repo.continuum.io (repo.continuum.io)... 104.16.18.10, 104.16.19.10, 2400:cb00:2048:1::6810:130a, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.16.18.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13147104 (13M) [application/x-tar]
Saving to: ‘opencv-3.1.0-np112py27_2.tar.bz2’
opencv-3.1.0-np112py27_2.t 100%[=====================================>] 12.54M 9.76MB/s in 1.3s
2017-06-14 03:35:39 (9.76 MB/s) - ‘opencv-3.1.0-np112py27_2.tar.bz2’ saved [13147104/13147104]
u0017496@sys-87576:~$ mkdir opencv-3.1.0-py27
u0017496@sys-87576:~$ cd opencv-3.1.0-py27
u0017496@sys-87576:~/opencv-3.1.0-py27$ tar -jxvf ../opencv-3.1.0-np112py27_2.tar.bz2
u0017496@sys-87576:~$ source activate py2k
(py2k) u0017496@sys-87576:~$ export PYTHONPATH=$PYTHONPATH:/home/u0017496/opencv-3.1.0-py27/lib/python2.7/site-packages
(py2k) u0017496@sys-87576:~$ python
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Mar 16 2017, 18:34:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import cv2
>>> import graphviz
>>>
2017년 6월 13일 화요일
ppc64le 아키텍처용 anaconda package list에 없는 일부 package의 수동 설치 (xgboost의 사례)
이제 ppc64le 아키텍처에서도 Continuum사의 miniconda가 지원됩니다. 다만 모든 package들이 지금 다 available한 것은 아닙니다. 가령 아래 URL에 가서 확인해보면 몇몇 package들은 ppc64le에서는 지원되지 않는 것을 확인하실 수 있습니다. 대표적인 예가 xgboost 입니다.
https://repo.continuum.io/pkgs/free/linux-ppc64le/
그러나 여기에 포함되어 있지 않다고 해서 ppc64le에서는 정말 사용할 수 없느냐 하면 그건 아닙니다. 매우 간단히 설치가 가능합니다.
먼저, 그냥 그대로 pip로 xgboost를 설치할 때 어떤 error가 벌어지는지 보시지요. 먼저, pip가 anaconda에서 제공하는 pip인지 확인합니다.
u0017496@sys-87576:~$ which pip
/home/u0017496/miniconda3/bin/pip
그 다음에 이 pip를 이용하여 xgboost 설치를 시도해 봅니다.
u0017496@sys-87576:~$ pip install xgboost
...
Using cached xgboost-0.6a2.tar.gz
Complete output from command python setup.py egg_info:
rm -f -rf build build_plugin lib bin *~ */*~ */*/*~ */*/*/*~ */*.o */*/*.o */*/*/*.o xgboost
g++ -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude -Idmlc-core/include -Irabit/include -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
g++ -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude -Idmlc-core/include -Irabit/include -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
g++ -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude -Idmlc-core/include -Irabit/include -fPIC -fopenmp -MM -MT build/common/common.o src/common/common.cc >build/common/common.d
g++ -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude -Idmlc-core/include -Irabit/include -fPIC -fopenmp -MM -MT build/metric/metric.o src/metric/metric.cc >build/metric/metric.d
g++: error: unrecognized command line option ‘-msse2’
결국 intel x86 아키텍처에만 있는 SSE2 instruction 관련 option이 문제인 것을 보실 수 있습니다. 이는 source를 download 받은 뒤 직접 python setup.py를 수행함으로써 간단히 해결 가능합니다.
다음과 같이 pip download 명령으로 xgboost의 source를 download 받습니다.
u0017496@sys-87576:~$ pip download -d "./" xgboost
이것의 압축을 풀고, 관련 Makefile들을 수정합니다. 그냥 -msse2 부분만 빼줘도 되는데, 여기서는 하는 김에 CPU 아키텍처가 POWER8이라는 것을 지정하겠습니다.
u0017496@sys-87576:~$ tar -zxvf xgboost-0.6a2.tar.gz
u0017496@sys-87576:~$ cd xgboost-0.6a2
u0017496@sys-87576:~/xgboost-0.6a2$ vi ./xgboost/Makefile
export CFLAGS= -std=c++0x -Wall -O3 -mcpu=power8 -Wno-unknown-pragmas -funroll-loops -Iinclude $(ADD_CFLAGS) $(PLUGIN_CFLAGS)
#export CFLAGS= -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude $(ADD_CFLAGS) $(PLUGIN_CFLAGS)
u0017496@sys-87576:~/xgboost-0.6a2$ vi ./xgboost/dmlc-core/Makefile
export CFLAGS = -O3 -Wall -mcpu=power8 -Wno-unknown-pragmas -Iinclude -std=c++0x
#export CFLAGS = -O3 -Wall -msse2 -Wno-unknown-pragmas -Iinclude -std=c++0x
u0017496@sys-87576:~/xgboost-0.6a2$ vi ./xgboost/rabit/Makefile
export CFLAGS = -O3 -mcpu=power8 $(WARNFLAGS)
#export CFLAGS = -O3 -msse2 $(WARNFLAGS)
그 다음에 'python setup.py install' 명령을 직접 수행합니다.
u0017496@sys-87576:~/xgboost-0.6a2$ python setup.py install
...
Searching for scipy==0.19.0
Best match: scipy 0.19.0
Adding scipy 0.19.0 to easy-install.pth file
Using /home/u0017496/miniconda3/lib/python3.6/site-packages
Searching for numpy==1.13.0
Best match: numpy 1.13.0
Adding numpy 1.13.0 to easy-install.pth file
Using /home/u0017496/miniconda3/lib/python3.6/site-packages
Finished processing dependencies for xgboost==0.6a2
결과적으로 잘 설치되었습니다. PYTHONPATH로 되어 있는 /home/u0017496/miniconda3/lib/python3.6/site-packages 디렉토리를 확인하면 다음과 같이 해당 directory가 생성된 것을 보실 수 있습니다.
u0017496@sys-87576:~/xgboost-0.6a2$ ls /home/u0017496/miniconda3/lib/python3.6/site-packages | grep xgboost
xgboost-0.6a2-py3.6.egg
또한 conda list 명령으로 보면 pip 명령으로 해당 package가 설치된 것으로 display 되는 것을 확인하실 수 있습니다.
u0017496@sys-87576:~/xgboost-0.6a2$ conda list | grep xgboost
xgboost 0.6a2 <pip>
pip 명령으로 삭제도 정상적으로 됩니다.
u0017496@sys-87576:~/xgboost-0.6a2$ pip uninstall xgboost
Uninstalling xgboost-0.6a2:
/home/u0017496/miniconda3/lib/python3.6/site-packages/xgboost-0.6a2-py3.6.egg
https://repo.continuum.io/pkgs/free/linux-ppc64le/
그러나 여기에 포함되어 있지 않다고 해서 ppc64le에서는 정말 사용할 수 없느냐 하면 그건 아닙니다. 매우 간단히 설치가 가능합니다.
먼저, 그냥 그대로 pip로 xgboost를 설치할 때 어떤 error가 벌어지는지 보시지요. 먼저, pip가 anaconda에서 제공하는 pip인지 확인합니다.
u0017496@sys-87576:~$ which pip
/home/u0017496/miniconda3/bin/pip
u0017496@sys-87576:~$ pip install xgboost
...
Using cached xgboost-0.6a2.tar.gz
Complete output from command python setup.py egg_info:
rm -f -rf build build_plugin lib bin *~ */*~ */*/*~ */*/*/*~ */*.o */*/*.o */*/*/*.o xgboost
g++ -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude -Idmlc-core/include -Irabit/include -fPIC -fopenmp -MM -MT build/logging.o src/logging.cc >build/logging.d
g++ -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude -Idmlc-core/include -Irabit/include -fPIC -fopenmp -MM -MT build/learner.o src/learner.cc >build/learner.d
g++ -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude -Idmlc-core/include -Irabit/include -fPIC -fopenmp -MM -MT build/common/common.o src/common/common.cc >build/common/common.d
g++ -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude -Idmlc-core/include -Irabit/include -fPIC -fopenmp -MM -MT build/metric/metric.o src/metric/metric.cc >build/metric/metric.d
g++: error: unrecognized command line option ‘-msse2’
다음과 같이 pip download 명령으로 xgboost의 source를 download 받습니다.
u0017496@sys-87576:~$ pip download -d "./" xgboost
이것의 압축을 풀고, 관련 Makefile들을 수정합니다. 그냥 -msse2 부분만 빼줘도 되는데, 여기서는 하는 김에 CPU 아키텍처가 POWER8이라는 것을 지정하겠습니다.
u0017496@sys-87576:~$ tar -zxvf xgboost-0.6a2.tar.gz
u0017496@sys-87576:~$ cd xgboost-0.6a2
u0017496@sys-87576:~/xgboost-0.6a2$ vi ./xgboost/Makefile
export CFLAGS= -std=c++0x -Wall -O3 -mcpu=power8 -Wno-unknown-pragmas -funroll-loops -Iinclude $(ADD_CFLAGS) $(PLUGIN_CFLAGS)
#export CFLAGS= -std=c++0x -Wall -O3 -msse2 -Wno-unknown-pragmas -funroll-loops -Iinclude $(ADD_CFLAGS) $(PLUGIN_CFLAGS)
u0017496@sys-87576:~/xgboost-0.6a2$ vi ./xgboost/dmlc-core/Makefile
export CFLAGS = -O3 -Wall -mcpu=power8 -Wno-unknown-pragmas -Iinclude -std=c++0x
#export CFLAGS = -O3 -Wall -msse2 -Wno-unknown-pragmas -Iinclude -std=c++0x
u0017496@sys-87576:~/xgboost-0.6a2$ vi ./xgboost/rabit/Makefile
export CFLAGS = -O3 -mcpu=power8 $(WARNFLAGS)
#export CFLAGS = -O3 -msse2 $(WARNFLAGS)
그 다음에 'python setup.py install' 명령을 직접 수행합니다.
u0017496@sys-87576:~/xgboost-0.6a2$ python setup.py install
...
Searching for scipy==0.19.0
Best match: scipy 0.19.0
Adding scipy 0.19.0 to easy-install.pth file
Using /home/u0017496/miniconda3/lib/python3.6/site-packages
Searching for numpy==1.13.0
Best match: numpy 1.13.0
Adding numpy 1.13.0 to easy-install.pth file
Using /home/u0017496/miniconda3/lib/python3.6/site-packages
Finished processing dependencies for xgboost==0.6a2
결과적으로 잘 설치되었습니다. PYTHONPATH로 되어 있는 /home/u0017496/miniconda3/lib/python3.6/site-packages 디렉토리를 확인하면 다음과 같이 해당 directory가 생성된 것을 보실 수 있습니다.
u0017496@sys-87576:~/xgboost-0.6a2$ ls /home/u0017496/miniconda3/lib/python3.6/site-packages | grep xgboost
xgboost-0.6a2-py3.6.egg
또한 conda list 명령으로 보면 pip 명령으로 해당 package가 설치된 것으로 display 되는 것을 확인하실 수 있습니다.
u0017496@sys-87576:~/xgboost-0.6a2$ conda list | grep xgboost
xgboost 0.6a2 <pip>
pip 명령으로 삭제도 정상적으로 됩니다.
u0017496@sys-87576:~/xgboost-0.6a2$ pip uninstall xgboost
Uninstalling xgboost-0.6a2:
/home/u0017496/miniconda3/lib/python3.6/site-packages/xgboost-0.6a2-py3.6.egg
Proceed (y/n)? y
Successfully uninstalled xgboost-0.6a2
2017년 5월 15일 월요일
Minsky 서버에 Continuum 아나콘다(Anaconda) 설치하기 + Tensorflow로 inception v3 training 해보기
Continuum에서 내놓은 아나콘다(Anaconda)는 여태까지는 x86용으로만 존재했으나, 최근 ARM processor와 POWER8 processor를 위한 min-conda를 내놓았습니다. 아래 site에 해당 package를 download 받기 위한 link와 설치 방법이 정리되어 있습니다.
https://www.continuum.io/content/conda-support-raspberry-pi-2-and-power8-le
불행히도 ppc64le를 위한 link는 잘못 지정되어 있어 '404 Not Found'가 나옵니다만, 실제로는 link만 잘못된 것이고 아래와 같이 file은 실제로 존재합니다. 위의 것은 python version2 용이고, 아래 것은 python 3용입니다.
https://repo.continuum.io/miniconda/Miniconda2-4.3.14-Linux-ppc64le.sh
https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
여기서는 python 3용을 설치해보겠습니다.
u0017496@sys-87250:~$ wget https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
--2017-05-14 21:54:26-- https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
Resolving repo.continuum.io (repo.continuum.io)... 104.16.19.10, 104.16.18.10, 2400:cb00:2048:1::6810:120a, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.16.19.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34765794 (33M) [application/x-sh]
Saving to: ‘Miniconda3-4.3.14-Linux-ppc64le.sh’
Miniconda3-4.3.14-Linux-pp 100%[=====================================>] 33.15M 10.2MB/s in 3.3s
2017-05-17 22:05:18 (10.0 MB/s) - ‘Miniconda3-4.3.14-Linux-ppc64le.sh’ saved [34765794/34765794]
u0017496@sys-87250:~$ chmod a+x Miniconda3-4.3.14-Linux-ppc64le.sh
이제 이 shell script를 수행하면 license 동의 등에 답을 해야 하며, 그 외에도 여러가지 입력 값을 넣어야 합니다. 대부분 그냥 enter를 누르시면 됩니다.
u0017496@sys-87250:~$ ./Miniconda3-4.3.14-Linux-ppc64le.sh
Welcome to Miniconda3 4.3.14 (by Continuum Analytics, Inc.)
In order to continue the installation process, please review the license
agreement.
Please, press ENTER to continue
>>>
(중략)
[/home/u0017496/miniconda3] >>>
PREFIX=/home/u0017496/miniconda3
installing: python-3.6.0-0 ...
installing: cffi-1.9.1-py36_0 ...
installing: conda-env-2.6.0-0 ...
installing: cryptography-1.7.1-py36_0 ...
installing: idna-2.2-py36_0 ...
installing: libffi-3.2.1-1 ...
installing: openssl-1.0.2k-1 ...
installing: pyasn1-0.2.3-py36_0 ...
installing: pycosat-0.6.2-py36_0 ...
installing: pycparser-2.17-py36_0 ...
installing: pyopenssl-16.2.0-py36_0 ...
installing: requests-2.13.0-py36_0 ...
installing: ruamel_yaml-0.11.14-py36_1 ...
installing: setuptools-27.2.0-py36_0 ...
installing: six-1.10.0-py36_0 ...
installing: sqlite-3.13.0-0 ...
installing: xz-5.2.2-1 ...
installing: yaml-0.1.6-0 ...
installing: zlib-1.2.8-3 ...
installing: conda-4.3.14-py36_0 ...
installing: pip-9.0.1-py36_1 ...
installing: wheel-0.29.0-py36_0 ...
Python 3.6.0 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
Do you wish the installer to prepend the Miniconda3 install location
to PATH in your /home/u0017496/.bashrc ? [yes|no]
[no] >>> yes
Prepending PATH=/home/u0017496/miniconda3/bin to PATH in /home/u0017496/.bashrc
A backup will be made to: /home/u0017496/.bashrc-miniconda3.bak
For this change to become active, you have to open a new terminal.
Thank you for installing Miniconda2!
Share your notebooks and packages on Anaconda Cloud!
Sign up for free: https://anaconda.org
이제 mini-conda를 설치했으니 conda 명령을 쓸 수 있어야 합니다. 그러나 보시다시피 conda가 없습니다.
u0017496@sys-87250:~$ which conda
이는 mini-conda 설치시 ~/.bashrc에 conda의 PATH 정보가 자동으로 들어가긴 했지만 .bashrc가 수행되지 않았기 때문에 그런 것입니다. 수행하시면 conda의 PATH가 잡혀 있는 것을 보실 수 있습니다.
u0017496@sys-87250:~$ . ~/.bashrc
u0017496@sys-87250:~$ which conda
/home/u0017496/miniconda3/bin/conda
이제 다음과 같이 ananconda에 포함된 python library들을 보실 수 있습니다.
u0017496@sys-87250:~$ conda list
# packages in environment at /home/u0017496/miniconda3:
#
cffi 1.9.1 py36_0
conda 4.3.14 py36_0
conda-env 2.6.0 0
cryptography 1.7.1 py36_0
idna 2.2 py36_0
libffi 3.2.1 1
openssl 1.0.2k 1
pip 9.0.1 py36_1
pyasn1 0.2.3 py36_0
pycosat 0.6.2 py36_0
pycparser 2.17 py36_0
pyopenssl 16.2.0 py36_0
python 3.6.0 0
requests 2.13.0 py36_0
ruamel_yaml 0.11.14 py36_1
setuptools 27.2.0 py36_0
six 1.10.0 py36_0
sqlite 3.13.0 0
wheel 0.29.0 py36_0
xz 5.2.2 1
yaml 0.1.6 0
zlib 1.2.8 3
현재까지 Continuum에서 빌드해놓은 package들을 모조리 다 설치하는 명령을 다음에 정리했습니다.
u0017496@sys-87250:~$ for i in `conda list | awk '{print $1}' | grep -v \#`
> do
> conda install $i
> done
이제 위에서 설치한 패키지 중 pip가 제대로 설치되었는지 conda search로 확인해보겠습니다. 아래와 같이 * 표시가 된 것이 설치된 것입니다.
u0017496@sys-87250:~$ conda search pip
Fetching package metadata .........
pip 7.1.0 py27_0 defaults
7.1.0 py34_0 defaults
7.1.0 py27_1 defaults
7.1.0 py34_1 defaults
7.1.2 py27_0 defaults
7.1.2 py34_0 defaults
8.1.0 py27_0 defaults
8.1.0 py34_0 defaults
8.1.0 py35_0 defaults
8.1.2 py27_0 defaults
8.1.2 py34_0 defaults
8.1.2 py35_0 defaults
9.0.0 py27_0 defaults
9.0.0 py34_0 defaults
9.0.0 py35_0 defaults
9.0.1 py27_1 defaults
9.0.1 py35_1 defaults
* 9.0.1 py36_1 defaults
이렇게 conda에서 제공하는 pip로 keras 2.0.4를 설치해보겠습니다.
u0017496@sys-87250:~/miniconda3/lib$ pip install keras==2.0.4
Collecting keras==2.0.4
Downloading Keras-2.0.4.tar.gz (199kB)
100% |████████████████████████████████| 204kB 3.1MB/s
Collecting theano (from keras==2.0.4)
Downloading Theano-0.9.0.tar.gz (3.1MB)
100% |████████████████████████████████| 3.1MB 310kB/s
Collecting pyyaml (from keras==2.0.4)
Downloading PyYAML-3.12.tar.gz (253kB)
100% |████████████████████████████████| 256kB 3.6MB/s
Requirement already satisfied: six in ./python3.6/site-packages (from keras==2.0.4)
Requirement already satisfied: numpy>=1.9.1 in ./python3.6/site-packages (from theano->keras==2.0.4)
Requirement already satisfied: scipy>=0.14 in ./python3.6/site-packages (from theano->keras==2.0.4)
Building wheels for collected packages: keras, theano, pyyaml
Running setup.py bdist_wheel for keras ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/48/82/42/f06a8c03a8f95ada523a81ba723e89f059693e6ad868d09727
Running setup.py bdist_wheel for theano ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/d5/5b/93/433299b86e3e9b25f0f600e4e4ebf18e38eb7534ea518eba13
Running setup.py bdist_wheel for pyyaml ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
Successfully built keras theano pyyaml
Installing collected packages: theano, pyyaml, keras
Successfully installed keras-2.0.4 pyyaml-3.12 theano-0.9.0
또 gensim 2.0.0과 KoNLPy를 설치해보겠습니다.
u0017496@sys-87250:~$ pip install gensim==2.0.0
Collecting gensim==2.0.0
Downloading gensim-2.0.0.tar.gz (14.1MB)
100% |████████████████████████████████| 14.2MB 88kB/s
Requirement already satisfied: numpy>=1.3 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: scipy>=0.7.0 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: six>=1.5.0 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: smart_open>=1.2.1 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: boto>=2.32 in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Requirement already satisfied: bz2file in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Requirement already satisfied: requests in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Building wheels for collected packages: gensim
Running setup.py bdist_wheel for gensim ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/e9/5f/e7/4ff23a3fe4b181b44f37eed5602f179c1cc92a0a34f337e745
Successfully built gensim
Installing collected packages: gensim
Found existing installation: gensim 1.0.1
Uninstalling gensim-1.0.1:
Successfully uninstalled gensim-1.0.1
Successfully installed gensim-2.0.0
이제 conda 명령을 통해 추가로 numpy와 matplotlib, scipy와 scikit-learn를 설치해보겠습니다. matplotlib의 prerequisite이 numpy이고, scikit-learn의 prerequisite이 scipy라서 그것들은 자동으로 설치되니까, 실제로는 conda 명령은 두번만 쓰면 됩니다.
u0017496@sys-87250:~$ conda install matplotlib
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
cycler: 0.10.0-py36_0
freetype: 2.5.5-2
libpng: 1.6.27-0
matplotlib: 2.0.2-np112py36_0
numpy: 1.12.1-py36_0
openblas: 0.2.19-0
python-dateutil: 2.6.0-py36_0
pytz: 2017.2-py36_0
Proceed ([y]/n)? y
openblas-0.2.1 100% |###########################################################| Time: 0:00:00 10.21 MB/s
libpng-1.6.27- 100% |###########################################################| Time: 0:00:00 12.75 MB/s
freetype-2.5.5 100% |###########################################################| Time: 0:00:00 10.53 MB/s
numpy-1.12.1-p 100% |###########################################################| Time: 0:00:00 15.12 MB/s
pytz-2017.2-py 100% |###########################################################| Time: 0:00:00 13.25 MB/s
cycler-0.10.0- 100% |###########################################################| Time: 0:00:00 15.61 MB/s
python-dateuti 100% |###########################################################| Time: 0:00:00 6.43 MB/s
matplotlib-2.0 100% |###########################################################| Time: 0:00:00 14.62 MB/s
u0017496@sys-87250:~$ conda install scikit-learn
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
scikit-learn: 0.18.1-np112py36_1
scipy: 0.19.0-np112py36_0
Proceed ([y]/n)? y
scipy-0.19.0-n 100% |###########################################################| Time: 0:00:02 14.75 MB/s
scikit-learn-0 100% |###########################################################| Time: 0:00:00 15.56 MB/s
이렇게 설치된 것들은 아래와 같이 /home/u0017496/miniconda3/lib/python3.6/site-packages 에 들어갑니다.
u0017496@sys-87250:~$ ls /home/u0017496/miniconda3/lib/python3.6/site-packages/
asn1crypto mpl_toolkits python_dateutil-2.6.0-py3.6.egg-info
asn1crypto-0.22.0-py3.6.egg-info numpy-1.12.1.dist-info pytz
cffi OpenSSL pytz-2017.2-py3.6.egg-info
cffi-1.10.0-py3.6.egg-info packaging README.txt
_cffi_backend.so packaging-16.8-py3.6.egg-info requests
conda pip requests-2.14.2-py3.6.egg-info
conda-4.3.18-py3.6.egg-info pip-9.0.1-py3.6.egg-info ruamel_yaml
conda_env pyasn1 scikit_learn-0.18.1-py3.6.egg-info
cryptography pyasn1-0.2.3-py3.6.egg-info scipy
cryptography-1.8.1-py3.6.egg-info __pycache__ scipy-0.19.0-py3.6.egg-info
cycler-0.10.0-py3.6.egg-info pycosat-0.6.2-py3.6.egg-info setuptools-27.2.0-py3.6.egg
cycler.py pycosat.cpython-36m-powerpc64le-linux-gnu.so setuptools.pth
dateutil pycparser six-1.10.0-py3.6.egg-info
easy-install.pth pycparser-2.17-py3.6.egg-info six.py
idna pylab.py sklearn
idna-2.5-py3.6.egg-info pyOpenSSL-17.0.0-py3.6.egg-info test_pycosat.py
matplotlib pyparsing-2.1.4-py3.6.egg-info wheel
matplotlib-2.0.2-py3.6.egg-info pyparsing.py wheel-0.29.0-py3.6.egg-info
따라서 이것들을 사용하기 위해서는 PYTHONPATH는 다음과 같이 설정하시면 됩니다.
u0017496@sys-87250:~$ export PYTHONPATH=/home/u0017496/miniconda3/lib/python3.6/site-packages:$PYTHONPATH
이제 여기에 (PowerAI에 포함된 tensorflow 말고) conda로 bazel, tensorflow 및 tensorflow-gpu도 설치해보겠습니다.
u0017496@sys-87250:~$ conda install bazel
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
bazel: 0.4.5-0
Proceed ([y]/n)? y
bazel-0.4.5-0. 100% |#############################################| Time: 0:00:09 13.37 MB/s
u0017496@sys-87250:~$ conda install tensorflow
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
libprotobuf: 3.2.0-0
protobuf: 3.2.0-py36_0
tensorflow: 1.1.0-np112py36_0
werkzeug: 0.12.2-py36_0
Proceed ([y]/n)? y
libprotobuf-3. 100% |#############################################| Time: 0:00:00 13.84 MB/s
werkzeug-0.12. 100% |#############################################| Time: 0:00:00 18.67 MB/s
protobuf-3.2.0 100% |#############################################| Time: 0:00:00 10.39 MB/s
tensorflow-1.1 100% |#############################################| Time: 0:00:01 15.16 MB/s
u0017496@sys-87250:~$ conda install tensorflow-gpu
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
cudatoolkit: 8.0-0
cudnn: 6.0.21-0
tensorflow-gpu: 1.1.0-np112py36_0
Proceed ([y]/n)? y
cudatoolkit-8. 100% |#############################################| Time: 0:00:29 11.24 MB/s
cudnn-6.0.21-0 100% |#############################################| Time: 0:00:11 15.97 MB/s
tensorflow-gpu 100% |#############################################| Time: 0:00:06 14.27 MB/s
conda list 명령으로 보면 다음과 같은 것들이 설치된 것을 보실 수 있습니다.
u0017496@sys-87250:~$ conda list
# packages in environment at /home/u0017496/miniconda3:
#
asn1crypto 0.22.0 py36_0
bazel 0.4.5 0
boto 2.46.1 py36_0
bz2file 0.98 py36_0
cffi 1.10.0 py36_0
conda 4.3.18 py36_0
conda-env 2.6.0 0
cryptography 1.8.1 py36_0
cudatoolkit 8.0 0
cudnn 6.0.21 0
cycler 0.10.0 py36_0
freetype 2.5.5 2
gensim 1.0.1 np112py36_0
gensim 2.0.0 <pip>
idna 2.5 py36_0
Keras 2.0.4 <pip>
konlpy 0.4.4 <pip>
libffi 3.2.1 1
libpng 1.6.27 0
libprotobuf 3.2.0 0
matplotlib 2.0.2 np112py36_0
numpy 1.12.1 <pip>
numpy 1.12.1 py36_0
openblas 0.2.19 0
openssl 1.0.2k 2
packaging 16.8 py36_0
pip 9.0.1 py36_1
protobuf 3.2.0 py36_0
pyasn1 0.2.3 py36_0
pycosat 0.6.2 py36_0
pycparser 2.17 py36_0
pyopenssl 17.0.0 py36_0
pyparsing 2.1.4 py36_0
python 3.6.1 2
python-dateutil 2.6.0 py36_0
pytz 2017.2 py36_0
PyYAML 3.12 <pip>
requests 2.14.2 py36_0
ruamel_yaml 0.11.14 py36_1
scikit-learn 0.18.1 np112py36_1
scipy 0.19.0 np112py36_0
setuptools 27.2.0 py36_0
six 1.10.0 py36_0
smart_open 1.5.2 py36_0
sqlite 3.13.0 0
tensorflow 1.1.0 np112py36_0
tensorflow-gpu 1.1.0 np112py36_0
Theano 0.9.0 <pip>
werkzeug 0.12.2 py36_0
wheel 0.29.0 py36_0
xz 5.2.2 1
yaml 0.1.6 0
zlib 1.2.8 3
설치하는 김에, 이렇게 conda로 설치한 tensorflow를 이용하여 inception v3 model을 training 해보겠습니다. 다음 순서대로 따라 하시면 됩니다.
u0017496@sys-87250:~/inception$ pwd
/home/u0017496/inception
u0017496@sys-87250:~/inception$ export INCEPTION_DIR=/home/u0017496/inception
u0017496@sys-87250:~/inception$ curl -O http://download.tensorflow.org/models/image/imagenet/inception-v3-2016-03-01.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 380M 100 380M 0 0 5918k 0 0:01:05 0:01:05 --:--:-- 4233k
u0017496@sys-87250:~/inception$ tar -xvf inception-v3-2016-03-01.tar.gz
inception-v3/
inception-v3/checkpoint
inception-v3/README.txt
inception-v3/model.ckpt-157585
u0017496@sys-87250:~/inception$ git clone https://github.com/tensorflow/models.git
Cloning into 'models'...
remote: Counting objects: 4703, done.
remote: Compressing objects: 100% (43/43), done.
remote: Total 4703 (delta 17), reused 31 (delta 11), pack-reused 4649
Receiving objects: 100% (4703/4703), 153.34 MiB | 5.62 MiB/s, done.
Resolving deltas: 100% (2374/2374), done.
Checking connectivity... done.
u0017496@sys-87250:~/inception/models/inception$ export FLOWERS_DIR=/home/u0017496/inception/models/inception
u0017496@sys-87250:~/inception/models/inception$ mkdir -p $FLOWERS_DIR/data
u0017496@sys-87250:~/inception/models/inception$ which bazel
/home/u0017496/miniconda3/bin/bazel
u0017496@sys-87250:~/inception/models/inception$ bazel build inception/download_and_preprocess_flowers
Extracting Bazel installation...
....................
INFO: Found 1 target...
Target //inception:download_and_preprocess_flowers up-to-date:
bazel-bin/inception/download_and_preprocess_flowers
INFO: Elapsed time: 6.943s, Critical Path: 0.05s
u0017496@sys-87250:~/inception/models/inception$ export TEST_TMPDIR=/home/u0017496/.cache
u0017496@sys-87250:~/inception/models/inception$ bazel build inception/download_and_preprocess_flowers
INFO: $TEST_TMPDIR defined: output root default is '/home/u0017496/.cache'.
Extracting Bazel installation...
.............
INFO: Found 1 target...
Target //inception:download_and_preprocess_flowers up-to-date:
bazel-bin/inception/download_and_preprocess_flowers
INFO: Elapsed time: 4.867s, Critical Path: 0.03s
u0017496@sys-87250:~/inception/models/inception$ bazel-bin/inception/download_and_preprocess_flowers $FLOWERS_DIR/data
Downloading flower data set.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 218M 100 218M 0 0 9372k 0 0:00:23 0:00:23 --:--:-- 10.1M
(중략)
Found 3170 JPEG files across 5 labels inside /home/u0017496/inception/models/inception/data/raw-data/train.
Launching 2 threads for spacings: [[0, 1585], [1585, 3170]]
2017-05-19 05:33:44.191446 [thread 0]: Processed 1000 of 1585 images in thread batch.
2017-05-19 05:33:44.213856 [thread 1]: Processed 1000 of 1585 images in thread batch.
2017-05-19 05:33:54.902070 [thread 1]: Wrote 1585 images to /home/u0017496/inception/models/inception/data/train-00001-of-00002
2017-05-19 05:33:54.902172 [thread 1]: Wrote 1585 images to 1585 shards.
2017-05-19 05:33:54.911283 [thread 0]: Wrote 1585 images to /home/u0017496/inception/models/inception/data/train-00000-of-00002
2017-05-19 05:33:54.911360 [thread 0]: Wrote 1585 images to 1585 shards.
2017-05-19 05:33:55.171141: Finished writing all 3170 images in data set.
아래에서 보시다시피 이 inception v3는 꽃 사진을 분류하는 neural network입니다.
u0017496@sys-87250:~/inception/models/inception$ du -sm data/raw-data/train/*
29 data/raw-data/train/daisy
44 data/raw-data/train/dandelion
1 data/raw-data/train/LICENSE.txt
33 data/raw-data/train/roses
47 data/raw-data/train/sunflowers
48 data/raw-data/train/tulips
u0017496@sys-87250:~/inception/models/inception$ bazel build inception/flowers_train
INFO: $TEST_TMPDIR defined: output root default is '/home/u0017496/.cache'.
............................
INFO: Found 1 target...
Target //inception:flowers_train up-to-date:
bazel-bin/inception/flowers_train
INFO: Elapsed time: 6.502s, Critical Path: 0.03s
이제 비로소 inception v3의 training 준비가 끝났습니다. 이제 다음 명령으로 training을 시작합니다.
u0017496@sys-87250:~/inception/models/inception$ time bazel-bin/inception/flowers_train --train_dir=$FLOWERS_DIR/train --data_dir=$FLOWERS_DIR/data --pretrained_model_checkpoint_path=$INCEPTION_DIR/inception-v3/model.ckpt-157585 --fine_tune=True --initial_learning_rate=0.001 -input_queue_memory_factor=1 --max_steps=50 --num_gpus 1 --batch_size=32
NVIDIA: no NVIDIA devices found
2017-05-19 05:41:03.740213: E tensorflow/stream_executor/cuda/cuda_driver.cc:405] failed call to cuInit: CUDA_ERROR_UNKNOWN
2017-05-19 05:41:03.740670: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:145] kernel driver does not appear to be running on this host (sys-87250): /proc/driver/nvidia/version does not exist
2017-05-19 05:41:51.947244: Pre-trained model restored from /home/u0017496/inception/inception-v3/model.ckpt-157585
2017-05-19 05:47:22.023602: step 0, loss = 2.79 (0.2 examples/sec; 182.713 sec/batch)
2017-05-19 06:05:58.942671: step 10, loss = 2.53 (0.4 examples/sec; 78.882 sec/batch)
2017-05-19 06:19:26.875533: step 20, loss = 2.40 (0.4 examples/sec; 82.410 sec/batch)
2017-05-19 06:33:10.333275: step 30, loss = 2.20 (0.4 examples/sec; 77.844 sec/batch)
2017-05-19 06:48:27.688993: step 40, loss = 2.24 (0.3 examples/sec; 96.148 sec/batch)
real 84m30.882s
user 135m20.864s
sys 2m30.832s
이제 와서 고백하지만 제가 설치 demo를 보여드린 이 서버는 사실 GPU가 달려 있지 않은 POWER8 서버입니다. GPU가 없는 경우 CPU를 이용하게 되는데, 그런 경우 이 training의 완료는 보시다시피 매우, 매우 오래 걸립니다. 저 output을 보면 초당 example 0.4개 처리로 나옵니다만, P100을 이용하는 경우 (GPU 개수 및 batch size에 따라) 초당 50개~200개 단위로 처리가 됩니다.
아래는 전에 PowerAI를 설치한 Minsky 서버에서 수행했던 inception v3의 결과 log 일부입니다.
2017-05-16 03:48:46.352210: Pre-trained model restored from /gpfs/gpfs_gl4_16mb/b7p088za/inception-v3/model.ckpt-157585
2017-05-16 03:52:44.322381: step 0, loss = 2.72 (17.6 examples/sec; 21.830 sec/batch)
2017-05-16 03:55:29.550791: step 10, loss = 2.57 (213.6 examples/sec; 1.797 sec/batch)
2017-05-16 03:55:47.619990: step 20, loss = 2.35 (212.1 examples/sec; 1.810 sec/batch)
2017-05-16 03:56:05.953991: step 30, loss = 2.17 (206.6 examples/sec; 1.859 sec/batch)
2017-05-16 03:56:24.306742: step 40, loss = 1.98 (209.4 examples/sec; 1.834 sec/batch)
2017-05-16 03:56:42.490063: step 50, loss = 1.92 (217.8 examples/sec; 1.763 sec/batch)
2017-05-16 03:57:00.444537: step 60, loss = 1.67 (216.6 examples/sec; 1.773 sec/batch)
2017-05-16 03:57:18.366941: step 70, loss = 1.58 (212.7 examples/sec; 1.806 sec/batch)
2017-05-16 03:57:36.467837: step 80, loss = 1.55 (213.6 examples/sec; 1.798 sec/batch)
https://www.continuum.io/content/conda-support-raspberry-pi-2-and-power8-le
불행히도 ppc64le를 위한 link는 잘못 지정되어 있어 '404 Not Found'가 나옵니다만, 실제로는 link만 잘못된 것이고 아래와 같이 file은 실제로 존재합니다. 위의 것은 python version2 용이고, 아래 것은 python 3용입니다.
https://repo.continuum.io/miniconda/Miniconda2-4.3.14-Linux-ppc64le.sh
https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
여기서는 python 3용을 설치해보겠습니다.
u0017496@sys-87250:~$ wget https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
--2017-05-14 21:54:26-- https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
Resolving repo.continuum.io (repo.continuum.io)... 104.16.19.10, 104.16.18.10, 2400:cb00:2048:1::6810:120a, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.16.19.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34765794 (33M) [application/x-sh]
Saving to: ‘Miniconda3-4.3.14-Linux-ppc64le.sh’
Miniconda3-4.3.14-Linux-pp 100%[=====================================>] 33.15M 10.2MB/s in 3.3s
2017-05-17 22:05:18 (10.0 MB/s) - ‘Miniconda3-4.3.14-Linux-ppc64le.sh’ saved [34765794/34765794]
u0017496@sys-87250:~$ chmod a+x Miniconda3-4.3.14-Linux-ppc64le.sh
이제 이 shell script를 수행하면 license 동의 등에 답을 해야 하며, 그 외에도 여러가지 입력 값을 넣어야 합니다. 대부분 그냥 enter를 누르시면 됩니다.
u0017496@sys-87250:~$ ./Miniconda3-4.3.14-Linux-ppc64le.sh
Welcome to Miniconda3 4.3.14 (by Continuum Analytics, Inc.)
In order to continue the installation process, please review the license
agreement.
Please, press ENTER to continue
>>>
(중략)
[/home/u0017496/miniconda3] >>>
PREFIX=/home/u0017496/miniconda3
installing: python-3.6.0-0 ...
installing: cffi-1.9.1-py36_0 ...
installing: conda-env-2.6.0-0 ...
installing: cryptography-1.7.1-py36_0 ...
installing: idna-2.2-py36_0 ...
installing: libffi-3.2.1-1 ...
installing: openssl-1.0.2k-1 ...
installing: pyasn1-0.2.3-py36_0 ...
installing: pycosat-0.6.2-py36_0 ...
installing: pycparser-2.17-py36_0 ...
installing: pyopenssl-16.2.0-py36_0 ...
installing: requests-2.13.0-py36_0 ...
installing: ruamel_yaml-0.11.14-py36_1 ...
installing: setuptools-27.2.0-py36_0 ...
installing: six-1.10.0-py36_0 ...
installing: sqlite-3.13.0-0 ...
installing: xz-5.2.2-1 ...
installing: yaml-0.1.6-0 ...
installing: zlib-1.2.8-3 ...
installing: conda-4.3.14-py36_0 ...
installing: pip-9.0.1-py36_1 ...
installing: wheel-0.29.0-py36_0 ...
Python 3.6.0 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
Do you wish the installer to prepend the Miniconda3 install location
to PATH in your /home/u0017496/.bashrc ? [yes|no]
Prepending PATH=/home/u0017496/miniconda3/bin to PATH in /home/u0017496/.bashrc
A backup will be made to: /home/u0017496/.bashrc-miniconda3.bak
For this change to become active, you have to open a new terminal.
Thank you for installing Miniconda2!
Share your notebooks and packages on Anaconda Cloud!
Sign up for free: https://anaconda.org
이제 mini-conda를 설치했으니 conda 명령을 쓸 수 있어야 합니다. 그러나 보시다시피 conda가 없습니다.
u0017496@sys-87250:~$ which conda
이는 mini-conda 설치시 ~/.bashrc에 conda의 PATH 정보가 자동으로 들어가긴 했지만 .bashrc가 수행되지 않았기 때문에 그런 것입니다. 수행하시면 conda의 PATH가 잡혀 있는 것을 보실 수 있습니다.
u0017496@sys-87250:~$ . ~/.bashrc
u0017496@sys-87250:~$ which conda
/home/u0017496/miniconda3/bin/conda
이제 다음과 같이 ananconda에 포함된 python library들을 보실 수 있습니다.
u0017496@sys-87250:~$ conda list
# packages in environment at /home/u0017496/miniconda3:
#
cffi 1.9.1 py36_0
conda 4.3.14 py36_0
conda-env 2.6.0 0
cryptography 1.7.1 py36_0
idna 2.2 py36_0
libffi 3.2.1 1
openssl 1.0.2k 1
pip 9.0.1 py36_1
pyasn1 0.2.3 py36_0
pycosat 0.6.2 py36_0
pycparser 2.17 py36_0
pyopenssl 16.2.0 py36_0
python 3.6.0 0
requests 2.13.0 py36_0
ruamel_yaml 0.11.14 py36_1
setuptools 27.2.0 py36_0
six 1.10.0 py36_0
sqlite 3.13.0 0
wheel 0.29.0 py36_0
xz 5.2.2 1
yaml 0.1.6 0
zlib 1.2.8 3
현재까지 Continuum에서 빌드해놓은 package들을 모조리 다 설치하는 명령을 다음에 정리했습니다.
u0017496@sys-87250:~$ for i in `conda list | awk '{print $1}' | grep -v \#`
> do
> conda install $i
> done
(중략)
이제 위에서 설치한 패키지 중 pip가 제대로 설치되었는지 conda search로 확인해보겠습니다. 아래와 같이 * 표시가 된 것이 설치된 것입니다.
u0017496@sys-87250:~$ conda search pip
Fetching package metadata .........
pip 7.1.0 py27_0 defaults
7.1.0 py34_0 defaults
7.1.0 py27_1 defaults
7.1.0 py34_1 defaults
7.1.2 py27_0 defaults
7.1.2 py34_0 defaults
8.1.0 py27_0 defaults
8.1.0 py34_0 defaults
8.1.0 py35_0 defaults
8.1.2 py27_0 defaults
8.1.2 py34_0 defaults
8.1.2 py35_0 defaults
9.0.0 py27_0 defaults
9.0.0 py34_0 defaults
9.0.0 py35_0 defaults
9.0.1 py27_1 defaults
9.0.1 py35_1 defaults
* 9.0.1 py36_1 defaults
u0017496@sys-87250:~$ which pip
/home/u0017496/miniconda3/bin/pip
u0017496@sys-87250:~$ pip --version
pip 9.0.1 from /home/u0017496/miniconda3/lib/python3.6/site-packages (python 3.6)
이렇게 conda에서 제공하는 pip로 keras 2.0.4를 설치해보겠습니다.
u0017496@sys-87250:~/miniconda3/lib$ pip install keras==2.0.4
Collecting keras==2.0.4
Downloading Keras-2.0.4.tar.gz (199kB)
100% |████████████████████████████████| 204kB 3.1MB/s
Collecting theano (from keras==2.0.4)
Downloading Theano-0.9.0.tar.gz (3.1MB)
100% |████████████████████████████████| 3.1MB 310kB/s
Collecting pyyaml (from keras==2.0.4)
Downloading PyYAML-3.12.tar.gz (253kB)
100% |████████████████████████████████| 256kB 3.6MB/s
Requirement already satisfied: six in ./python3.6/site-packages (from keras==2.0.4)
Requirement already satisfied: numpy>=1.9.1 in ./python3.6/site-packages (from theano->keras==2.0.4)
Requirement already satisfied: scipy>=0.14 in ./python3.6/site-packages (from theano->keras==2.0.4)
Building wheels for collected packages: keras, theano, pyyaml
Running setup.py bdist_wheel for keras ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/48/82/42/f06a8c03a8f95ada523a81ba723e89f059693e6ad868d09727
Running setup.py bdist_wheel for theano ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/d5/5b/93/433299b86e3e9b25f0f600e4e4ebf18e38eb7534ea518eba13
Running setup.py bdist_wheel for pyyaml ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
Successfully built keras theano pyyaml
Installing collected packages: theano, pyyaml, keras
Successfully installed keras-2.0.4 pyyaml-3.12 theano-0.9.0
또 gensim 2.0.0과 KoNLPy를 설치해보겠습니다.
u0017496@sys-87250:~$ pip install gensim==2.0.0
Collecting gensim==2.0.0
Downloading gensim-2.0.0.tar.gz (14.1MB)
100% |████████████████████████████████| 14.2MB 88kB/s
Requirement already satisfied: numpy>=1.3 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: scipy>=0.7.0 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: six>=1.5.0 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: smart_open>=1.2.1 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: boto>=2.32 in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Requirement already satisfied: bz2file in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Requirement already satisfied: requests in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Building wheels for collected packages: gensim
Running setup.py bdist_wheel for gensim ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/e9/5f/e7/4ff23a3fe4b181b44f37eed5602f179c1cc92a0a34f337e745
Successfully built gensim
Installing collected packages: gensim
Found existing installation: gensim 1.0.1
Uninstalling gensim-1.0.1:
Successfully uninstalled gensim-1.0.1
Successfully installed gensim-2.0.0
u0017496@sys-87250:~$ pip install konlpy
Collecting konlpy
Downloading konlpy-0.4.4-py2.py3-none-any.whl (22.5MB)
100% |████████████████████████████████| 22.5MB 57kB/s
Installing collected packages: konlpy
Successfully installed konlpy-0.4.4
이제 conda 명령을 통해 추가로 numpy와 matplotlib, scipy와 scikit-learn를 설치해보겠습니다. matplotlib의 prerequisite이 numpy이고, scikit-learn의 prerequisite이 scipy라서 그것들은 자동으로 설치되니까, 실제로는 conda 명령은 두번만 쓰면 됩니다.
u0017496@sys-87250:~$ conda install matplotlib
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
cycler: 0.10.0-py36_0
freetype: 2.5.5-2
libpng: 1.6.27-0
matplotlib: 2.0.2-np112py36_0
numpy: 1.12.1-py36_0
openblas: 0.2.19-0
python-dateutil: 2.6.0-py36_0
pytz: 2017.2-py36_0
Proceed ([y]/n)? y
openblas-0.2.1 100% |###########################################################| Time: 0:00:00 10.21 MB/s
libpng-1.6.27- 100% |###########################################################| Time: 0:00:00 12.75 MB/s
freetype-2.5.5 100% |###########################################################| Time: 0:00:00 10.53 MB/s
numpy-1.12.1-p 100% |###########################################################| Time: 0:00:00 15.12 MB/s
pytz-2017.2-py 100% |###########################################################| Time: 0:00:00 13.25 MB/s
cycler-0.10.0- 100% |###########################################################| Time: 0:00:00 15.61 MB/s
python-dateuti 100% |###########################################################| Time: 0:00:00 6.43 MB/s
matplotlib-2.0 100% |###########################################################| Time: 0:00:00 14.62 MB/s
u0017496@sys-87250:~$ conda install scikit-learn
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
scikit-learn: 0.18.1-np112py36_1
scipy: 0.19.0-np112py36_0
Proceed ([y]/n)? y
scipy-0.19.0-n 100% |###########################################################| Time: 0:00:02 14.75 MB/s
scikit-learn-0 100% |###########################################################| Time: 0:00:00 15.56 MB/s
이렇게 설치된 것들은 아래와 같이 /home/u0017496/miniconda3/lib/python3.6/site-packages 에 들어갑니다.
u0017496@sys-87250:~$ ls /home/u0017496/miniconda3/lib/python3.6/site-packages/
asn1crypto mpl_toolkits python_dateutil-2.6.0-py3.6.egg-info
asn1crypto-0.22.0-py3.6.egg-info numpy-1.12.1.dist-info pytz
cffi OpenSSL pytz-2017.2-py3.6.egg-info
cffi-1.10.0-py3.6.egg-info packaging README.txt
_cffi_backend.so packaging-16.8-py3.6.egg-info requests
conda pip requests-2.14.2-py3.6.egg-info
conda-4.3.18-py3.6.egg-info pip-9.0.1-py3.6.egg-info ruamel_yaml
conda_env pyasn1 scikit_learn-0.18.1-py3.6.egg-info
cryptography pyasn1-0.2.3-py3.6.egg-info scipy
cryptography-1.8.1-py3.6.egg-info __pycache__ scipy-0.19.0-py3.6.egg-info
cycler-0.10.0-py3.6.egg-info pycosat-0.6.2-py3.6.egg-info setuptools-27.2.0-py3.6.egg
cycler.py pycosat.cpython-36m-powerpc64le-linux-gnu.so setuptools.pth
dateutil pycparser six-1.10.0-py3.6.egg-info
easy-install.pth pycparser-2.17-py3.6.egg-info six.py
idna pylab.py sklearn
idna-2.5-py3.6.egg-info pyOpenSSL-17.0.0-py3.6.egg-info test_pycosat.py
matplotlib pyparsing-2.1.4-py3.6.egg-info wheel
matplotlib-2.0.2-py3.6.egg-info pyparsing.py wheel-0.29.0-py3.6.egg-info
따라서 이것들을 사용하기 위해서는 PYTHONPATH는 다음과 같이 설정하시면 됩니다.
u0017496@sys-87250:~$ export PYTHONPATH=/home/u0017496/miniconda3/lib/python3.6/site-packages:$PYTHONPATH
이제 여기에 (PowerAI에 포함된 tensorflow 말고) conda로 bazel, tensorflow 및 tensorflow-gpu도 설치해보겠습니다.
u0017496@sys-87250:~$ conda install bazel
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
bazel: 0.4.5-0
Proceed ([y]/n)? y
bazel-0.4.5-0. 100% |#############################################| Time: 0:00:09 13.37 MB/s
u0017496@sys-87250:~$ conda install tensorflow
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
libprotobuf: 3.2.0-0
protobuf: 3.2.0-py36_0
tensorflow: 1.1.0-np112py36_0
werkzeug: 0.12.2-py36_0
Proceed ([y]/n)? y
libprotobuf-3. 100% |#############################################| Time: 0:00:00 13.84 MB/s
werkzeug-0.12. 100% |#############################################| Time: 0:00:00 18.67 MB/s
protobuf-3.2.0 100% |#############################################| Time: 0:00:00 10.39 MB/s
tensorflow-1.1 100% |#############################################| Time: 0:00:01 15.16 MB/s
u0017496@sys-87250:~$ conda install tensorflow-gpu
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
cudatoolkit: 8.0-0
cudnn: 6.0.21-0
tensorflow-gpu: 1.1.0-np112py36_0
Proceed ([y]/n)? y
cudatoolkit-8. 100% |#############################################| Time: 0:00:29 11.24 MB/s
cudnn-6.0.21-0 100% |#############################################| Time: 0:00:11 15.97 MB/s
tensorflow-gpu 100% |#############################################| Time: 0:00:06 14.27 MB/s
conda list 명령으로 보면 다음과 같은 것들이 설치된 것을 보실 수 있습니다.
u0017496@sys-87250:~$ conda list
# packages in environment at /home/u0017496/miniconda3:
#
asn1crypto 0.22.0 py36_0
bazel 0.4.5 0
boto 2.46.1 py36_0
bz2file 0.98 py36_0
cffi 1.10.0 py36_0
conda 4.3.18 py36_0
conda-env 2.6.0 0
cryptography 1.8.1 py36_0
cudatoolkit 8.0 0
cudnn 6.0.21 0
cycler 0.10.0 py36_0
freetype 2.5.5 2
gensim 1.0.1 np112py36_0
gensim 2.0.0 <pip>
idna 2.5 py36_0
Keras 2.0.4 <pip>
konlpy 0.4.4 <pip>
libffi 3.2.1 1
libpng 1.6.27 0
libprotobuf 3.2.0 0
matplotlib 2.0.2 np112py36_0
numpy 1.12.1 <pip>
numpy 1.12.1 py36_0
openblas 0.2.19 0
openssl 1.0.2k 2
packaging 16.8 py36_0
pip 9.0.1 py36_1
protobuf 3.2.0 py36_0
pyasn1 0.2.3 py36_0
pycosat 0.6.2 py36_0
pycparser 2.17 py36_0
pyopenssl 17.0.0 py36_0
pyparsing 2.1.4 py36_0
python 3.6.1 2
python-dateutil 2.6.0 py36_0
pytz 2017.2 py36_0
PyYAML 3.12 <pip>
requests 2.14.2 py36_0
ruamel_yaml 0.11.14 py36_1
scikit-learn 0.18.1 np112py36_1
scipy 0.19.0 np112py36_0
setuptools 27.2.0 py36_0
six 1.10.0 py36_0
smart_open 1.5.2 py36_0
sqlite 3.13.0 0
tensorflow 1.1.0 np112py36_0
tensorflow-gpu 1.1.0 np112py36_0
Theano 0.9.0 <pip>
werkzeug 0.12.2 py36_0
wheel 0.29.0 py36_0
xz 5.2.2 1
yaml 0.1.6 0
zlib 1.2.8 3
설치하는 김에, 이렇게 conda로 설치한 tensorflow를 이용하여 inception v3 model을 training 해보겠습니다. 다음 순서대로 따라 하시면 됩니다.
u0017496@sys-87250:~/inception$ pwd
/home/u0017496/inception
u0017496@sys-87250:~/inception$ export INCEPTION_DIR=/home/u0017496/inception
u0017496@sys-87250:~/inception$ curl -O http://download.tensorflow.org/models/image/imagenet/inception-v3-2016-03-01.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 380M 100 380M 0 0 5918k 0 0:01:05 0:01:05 --:--:-- 4233k
u0017496@sys-87250:~/inception$ tar -xvf inception-v3-2016-03-01.tar.gz
inception-v3/
inception-v3/checkpoint
inception-v3/README.txt
inception-v3/model.ckpt-157585
u0017496@sys-87250:~/inception$ git clone https://github.com/tensorflow/models.git
Cloning into 'models'...
remote: Counting objects: 4703, done.
remote: Compressing objects: 100% (43/43), done.
remote: Total 4703 (delta 17), reused 31 (delta 11), pack-reused 4649
Receiving objects: 100% (4703/4703), 153.34 MiB | 5.62 MiB/s, done.
Resolving deltas: 100% (2374/2374), done.
Checking connectivity... done.
u0017496@sys-87250:~/inception/models/inception$ export FLOWERS_DIR=/home/u0017496/inception/models/inception
u0017496@sys-87250:~/inception/models/inception$ mkdir -p $FLOWERS_DIR/data
u0017496@sys-87250:~/inception/models/inception$ which bazel
/home/u0017496/miniconda3/bin/bazel
u0017496@sys-87250:~/inception/models/inception$ bazel build inception/download_and_preprocess_flowers
Extracting Bazel installation...
....................
INFO: Found 1 target...
Target //inception:download_and_preprocess_flowers up-to-date:
bazel-bin/inception/download_and_preprocess_flowers
INFO: Elapsed time: 6.943s, Critical Path: 0.05s
u0017496@sys-87250:~/inception/models/inception$ export TEST_TMPDIR=/home/u0017496/.cache
u0017496@sys-87250:~/inception/models/inception$ bazel build inception/download_and_preprocess_flowers
INFO: $TEST_TMPDIR defined: output root default is '/home/u0017496/.cache'.
Extracting Bazel installation...
.............
INFO: Found 1 target...
Target //inception:download_and_preprocess_flowers up-to-date:
bazel-bin/inception/download_and_preprocess_flowers
INFO: Elapsed time: 4.867s, Critical Path: 0.03s
u0017496@sys-87250:~/inception/models/inception$ bazel-bin/inception/download_and_preprocess_flowers $FLOWERS_DIR/data
Downloading flower data set.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 218M 100 218M 0 0 9372k 0 0:00:23 0:00:23 --:--:-- 10.1M
(중략)
Found 3170 JPEG files across 5 labels inside /home/u0017496/inception/models/inception/data/raw-data/train.
Launching 2 threads for spacings: [[0, 1585], [1585, 3170]]
2017-05-19 05:33:44.191446 [thread 0]: Processed 1000 of 1585 images in thread batch.
2017-05-19 05:33:44.213856 [thread 1]: Processed 1000 of 1585 images in thread batch.
2017-05-19 05:33:54.902070 [thread 1]: Wrote 1585 images to /home/u0017496/inception/models/inception/data/train-00001-of-00002
2017-05-19 05:33:54.902172 [thread 1]: Wrote 1585 images to 1585 shards.
2017-05-19 05:33:54.911283 [thread 0]: Wrote 1585 images to /home/u0017496/inception/models/inception/data/train-00000-of-00002
2017-05-19 05:33:54.911360 [thread 0]: Wrote 1585 images to 1585 shards.
2017-05-19 05:33:55.171141: Finished writing all 3170 images in data set.
아래에서 보시다시피 이 inception v3는 꽃 사진을 분류하는 neural network입니다.
u0017496@sys-87250:~/inception/models/inception$ du -sm data/raw-data/train/*
29 data/raw-data/train/daisy
44 data/raw-data/train/dandelion
1 data/raw-data/train/LICENSE.txt
33 data/raw-data/train/roses
47 data/raw-data/train/sunflowers
48 data/raw-data/train/tulips
u0017496@sys-87250:~/inception/models/inception$ bazel build inception/flowers_train
INFO: $TEST_TMPDIR defined: output root default is '/home/u0017496/.cache'.
............................
INFO: Found 1 target...
Target //inception:flowers_train up-to-date:
bazel-bin/inception/flowers_train
INFO: Elapsed time: 6.502s, Critical Path: 0.03s
이제 비로소 inception v3의 training 준비가 끝났습니다. 이제 다음 명령으로 training을 시작합니다.
u0017496@sys-87250:~/inception/models/inception$ time bazel-bin/inception/flowers_train --train_dir=$FLOWERS_DIR/train --data_dir=$FLOWERS_DIR/data --pretrained_model_checkpoint_path=$INCEPTION_DIR/inception-v3/model.ckpt-157585 --fine_tune=True --initial_learning_rate=0.001 -input_queue_memory_factor=1 --max_steps=50 --num_gpus 1 --batch_size=32
NVIDIA: no NVIDIA devices found
2017-05-19 05:41:03.740213: E tensorflow/stream_executor/cuda/cuda_driver.cc:405] failed call to cuInit: CUDA_ERROR_UNKNOWN
2017-05-19 05:41:03.740670: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:145] kernel driver does not appear to be running on this host (sys-87250): /proc/driver/nvidia/version does not exist
2017-05-19 05:41:51.947244: Pre-trained model restored from /home/u0017496/inception/inception-v3/model.ckpt-157585
2017-05-19 05:47:22.023602: step 0, loss = 2.79 (0.2 examples/sec; 182.713 sec/batch)
2017-05-19 06:05:58.942671: step 10, loss = 2.53 (0.4 examples/sec; 78.882 sec/batch)
2017-05-19 06:19:26.875533: step 20, loss = 2.40 (0.4 examples/sec; 82.410 sec/batch)
2017-05-19 06:33:10.333275: step 30, loss = 2.20 (0.4 examples/sec; 77.844 sec/batch)
2017-05-19 06:48:27.688993: step 40, loss = 2.24 (0.3 examples/sec; 96.148 sec/batch)
real 84m30.882s
user 135m20.864s
sys 2m30.832s
이제 와서 고백하지만 제가 설치 demo를 보여드린 이 서버는 사실 GPU가 달려 있지 않은 POWER8 서버입니다. GPU가 없는 경우 CPU를 이용하게 되는데, 그런 경우 이 training의 완료는 보시다시피 매우, 매우 오래 걸립니다. 저 output을 보면 초당 example 0.4개 처리로 나옵니다만, P100을 이용하는 경우 (GPU 개수 및 batch size에 따라) 초당 50개~200개 단위로 처리가 됩니다.
아래는 전에 PowerAI를 설치한 Minsky 서버에서 수행했던 inception v3의 결과 log 일부입니다.
2017-05-16 03:48:46.352210: Pre-trained model restored from /gpfs/gpfs_gl4_16mb/b7p088za/inception-v3/model.ckpt-157585
2017-05-16 03:52:44.322381: step 0, loss = 2.72 (17.6 examples/sec; 21.830 sec/batch)
2017-05-16 03:55:29.550791: step 10, loss = 2.57 (213.6 examples/sec; 1.797 sec/batch)
2017-05-16 03:55:47.619990: step 20, loss = 2.35 (212.1 examples/sec; 1.810 sec/batch)
2017-05-16 03:56:05.953991: step 30, loss = 2.17 (206.6 examples/sec; 1.859 sec/batch)
2017-05-16 03:56:24.306742: step 40, loss = 1.98 (209.4 examples/sec; 1.834 sec/batch)
2017-05-16 03:56:42.490063: step 50, loss = 1.92 (217.8 examples/sec; 1.763 sec/batch)
2017-05-16 03:57:00.444537: step 60, loss = 1.67 (216.6 examples/sec; 1.773 sec/batch)
2017-05-16 03:57:18.366941: step 70, loss = 1.58 (212.7 examples/sec; 1.806 sec/batch)
2017-05-16 03:57:36.467837: step 80, loss = 1.55 (213.6 examples/sec; 1.798 sec/batch)
피드 구독하기:
글 (Atom)