System admin이 Miniconda를 설치했는데, 별 생각없이 설치하여 root user의 home directory에 설치하는 바람에 /root/miniconda3에 설치된 경우 어떻게 해야 할까요 ?
결론부터 말씀드리면, 원하는 directory로 copy한 뒤 ownership을 바꿔주고, 그리고나서 약 100여개의 file들 속의 기존 directory 이름을 바꿔주면 됩니다.
원래의 정답은 아래의 Continuum 홈페이지에 나왔습니다.
https://docs.continuum.io/anaconda/user-guide/tasks/move-directory
클릭하기 귀찮으신 분을 위해 아래에 copy & paste 해드리면 다음과 같습니다.
To move Anaconda from one directory to another:
1. Delete it from the old directory.
2. Go to the new directory and install it there following the Anaconda installation instructions.
즉, 지우고 새로 설치하라는 것입니다. Anaconda 또는 Miniconda를 한번 설치하고 나면, 그 directory만 새로 옮기는 것은 허용되지 않는다고 합니다.
그러나 Continuum의 그런 정책에 대해서는 많은 불만들이 있을 수 있습니다. 많은 기업들이 보안 등의 문제로 서버를 internet access를 허용하지 않습니다. 따라서 서버를 도입할 때, 먼저 회사 외부의 internet이 자유롭게 access되는 곳에서 OS와 Miniconda를 설치한 뒤, mecab-python3처럼 Miniconda에는 포함되어 있지 않아 internet에서 download 받아야 하는 python package 등을 다 설치한 뒤 회사에 반입하는 경우가 꽤 많습니다. 그런데 그 설치 directory를 /root/miniconda3에 해버리면 다른 일반 유저들이 conda 명령들을 수행할 때 아래와 같은 error가 나버립니다.
u0017496@sys-87926:~$ /root/miniconda3/bin/conda list
-bash: /root/miniconda3/bin/conda: Permission denied
그래서 /root와 같은 민감한 directory를 피해서, /opt 혹은 /usr/local과 같은 곳으로 이 directory를 copy한 뒤 ownership만 변경해주면 좋겠습니다만, 아래와 같이 그것도 안 됩니다.
먼저 /root/miniconda3를 /usr/local/miniconda3로 copy 하고...
root@sys-87926:~# pwd
/root
root@sys-87926:~# cp -r miniconda3 /usr/local/
ownership을 변경해준 뒤
root@sys-87926:~# chown -R u0017496:u0017496 /usr/local/miniconda3
이제 u0017496 사용자로 login해서 환경 변수를 맞춰주고
u0017496@sys-87926:~$ vi ./.bashrc
...
export PATH=/usr/local/miniconda3/bin:$PATH
export PYTHONPATH=/usr/local/miniconda3/lib/python3.6/site-packages
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el
u0017496@sys-87926:~$ . ./.bashrc
u0017496@sys-87926:~$ which conda
/usr/local/miniconda3/bin/conda
간단한 conda 명령을 날려보면 다음과 같이 error가 납니다.
u0017496@sys-87926:~$ conda list
-bash: /usr/local/miniconda3/bin/conda: /root/miniconda3/bin/python: bad interpreter: Permission denied
이는 보시다시피 상당히 많은 file들에 원래의 directory name, 즉 /root/miniconda3이 박혀 있기 때문입니다. 가령 conda라는 명령 자체도 아래와 같이 python script이며, 그 속에 원래 설치 directory의 이름이 박혀 있습니다.
u0017496@sys-87926:~$ vi /usr/local/miniconda3/bin/conda
#!/root/miniconda3/bin/python
if __name__ == '__main__':
import sys
import conda.cli
sys.exit(conda.cli.main())
이에 대해서 아래 URL에 몇몇 user들이 토론을 벌인 바 있습니다. 실은 저 위의 Continuum 홈페이지에 '지우고 새로 설치하라'라고 하는 문서도 아래 github 토론에서 이야기가 나왔다가 Continuum 소속의 개발자가 사람들에게 공표하는 의미로 만든 것입니다.
https://github.com/ContinuumIO/anaconda-issues/issues/270
하지만 꼭 불가능한 것은 아닙니다. 위 github 토론에서도 (결론을 내지는 않았지만) '그냥 그 수많은 file들의 directory name들을 다 바꿔버리면 될 거다'라는 해결책을 제시한 개발자도 있었습니다.
제가 해보니 총 113개의 file만 고치면 되고, 실제로 sed 명령으로 고치면 꽤 쉽게 고칠 수 있으며, 결과적으로 잘 됩니다.
위에서처럼 cp와 chown을 해주고 환경변수 PATH 및 PYTHONPATH도 다 맞춰준 상태에서, 다음과 같이 예전 directory 이름, 즉 /root/miniconda3를 포함하는 text file들을 찾아 list를 만듭니다. 아래에서 find의 -type f는 (directory는 빼고) 파일만 찾으라는 것이고, grep의 -Iq (대문자 아이와 소문자 큐입니다)는 text file이 아닌 것은 즉각 무시하라는 뜻입니다.
u0017496@sys-87926:~$ for i in `find /usr/local/miniconda3 -type f -exec grep -Iq . {} \; -print`
> do
> grep "root\/miniconda3" $i
> if [[ $? -eq 0 ]]
> then
> echo $i >> matchlist.txt
> fi
> done
이 matchlist라는 새로 생성된 파일 속에는 아래처럼 /root/minconda3라는 string을 포함한 text file들의 목록이 들어 있습니다.
u0017496@sys-87926:~$ tail matchlist.txt
/usr/local/miniconda3/bin/idle3.6
/usr/local/miniconda3/bin/route53
/usr/local/miniconda3/bin/python3.6m-config
/usr/local/miniconda3/bin/cq
/usr/local/miniconda3/bin/conda
/usr/local/miniconda3/bin/instance_events
/usr/local/miniconda3/bin/freetype-config
/usr/local/miniconda3/bin/conda-env
/usr/local/miniconda3/bin/launch_instance
/usr/local/miniconda3/include/openssl/opensslconf.h
이제 이 파일들의 *.org(원본 보관용)과 *.new(임시작업용)을 복사해두고
u0017496@sys-87926:~$ for i in `cat matchlist.txt`
> do
> cp $i ${i}.org
> cp $i ${i}.new
> done
아래와 같이 sed 명령으로 "root/miniconda3"를 "usr/local/miniconda3"로 교체하여 임시작업용 *.new에 넣습니다. 그러고 난 뒤 원본을 이 *.new로 바꿔줍니다.
u0017496@sys-87926:~$ for i in `cat matchlist.txt`
> do
> sed 's/root\/miniconda3/usr\/local\/miniconda3/g' $i > ${i}.new
> cp ${i}.new $i
> done
임시작업용 파일들인 *.new는 나중에 삭제하셔도 됩니다.
보시면 아래와 같이 /root/miniconda가 /usr/local/miniconda3로 바뀐 것을 보실 수 있습니다.
u0017496@sys-87926:~$ grep usr\/local /usr/local/miniconda3/bin/pyvenv-3.6
#!/usr/local/miniconda3/bin/python
이제 conda 명령이 제대로 수행되는지 테스트 해보겠습니다. 보시다시피 잘 됩니다.
u0017496@sys-87926:~$ which conda
/usr/local/miniconda3/bin/conda
u0017496@sys-87926:~$ conda list
# packages in environment at /usr/local/miniconda3:
#
asn1crypto 0.22.0 py36_0
bazel 0.4.5 0
boto 2.47.0 <pip>
bz2file 0.98 <pip>
cffi 1.10.0 py36_0
conda 4.3.18 py36_0
conda-env 2.6.0 0
cryptography 1.8.1 py36_0
cudatoolkit 8.0 0
cudnn 6.0.21 0
cycler 0.10.0 py36_0
freetype 2.5.5 2
gensim 2.0.0 <pip>
idna 2.5 py36_0
Keras 2.0.4 <pip>
konlpy 0.4.4 <pip>
libffi 3.2.1 1
libpng 1.6.27 0
libprotobuf 3.2.0 0
matplotlib 2.0.2 np112py36_0
numpy 1.13.0 <pip>
numpy 1.12.1 py36_0
openblas 0.2.19 0
openssl 1.0.2k 2
packaging 16.8 py36_0
pip 9.0.1 py36_1
protobuf 3.2.0 py36_0
pyasn1 0.2.3 py36_0
pycosat 0.6.2 py36_0
pycparser 2.17 py36_0
pyopenssl 17.0.0 py36_0
pyparsing 2.1.4 py36_0
python 3.6.1 2
python-dateutil 2.6.0 py36_0
pytz 2017.2 py36_0
PyYAML 3.12 <pip>
requests 2.14.2 py36_0
ruamel_yaml 0.11.14 py36_1
scikit-learn 0.18.1 np112py36_1
scipy 0.19.0 <pip>
scipy 0.19.0 np112py36_0
setuptools 27.2.0 py36_0
six 1.10.0 py36_0
smart-open 1.5.3 <pip>
sqlite 3.13.0 0
tensorflow 1.1.0 np112py36_0
tensorflow-gpu 1.1.0 np112py36_0
Theano 0.9.0 <pip>
werkzeug 0.12.2 py36_0
wheel 0.29.0 py36_0
xz 5.2.2 1
yaml 0.1.6 0
zlib 1.2.8 3
새로 package를 install하는 것도 잘 됩니다.
u0017496@sys-87926:~$ conda install flask
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /usr/local/miniconda3:
The following NEW packages will be INSTALLED:
click: 6.7-py36_0
flask: 0.12.2-py36_0
itsdangerous: 0.24-py36_0
jinja2: 2.9.6-py36_0
markupsafe: 0.23-py36_2
Proceed ([y]/n)? y
click-6.7-py36 100% |######################################| Time: 0:00:00 7.96 MB/s
itsdangerous-0 100% |######################################| Time: 0:00:00 13.50 MB/s
markupsafe-0.2 100% |######################################| Time: 0:00:00 19.45 MB/s
jinja2-2.9.6-p 100% |######################################| Time: 0:00:00 8.85 MB/s
flask-0.12.2-p 100% |######################################| Time: 0:00:00 8.55 MB/s
뿐만 아니라, 새로 인터넷에서 download 받아 설치하는 것도 잘 됩니다. (물론 이건 인터넷이 access되는 환경에서만 가능한 테스트입니다.)
u0017496@sys-87926:~$ pip install Jpype1
Collecting Jpype1
Downloading JPype1-0.6.2.tar.gz (147kB)
100% |████████████████████████████████| 153kB 2.8MB/s
Building wheels for collected packages: Jpype1
Running setup.py bdist_wheel for Jpype1 ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/8e/f3/e6/a1250b8e8d2bd105f4dd21b1dc801dbcf5d815592443bfe741
Successfully built Jpype1
Installing collected packages: Jpype1
Successfully installed Jpype1-0.6.2
KoNLPy를 python에서 import 해보면 잘 되는 것을 보실 수 있습니다. (이걸 테스트하시려면 위와 같이 Jpype1가 설치되어 있어야 합니다.)
u0017496@sys-87926:~$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el
u0017496@sys-87926:~$ python
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 15:31:35)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from konlpy.tag import *
>>> Kkma()
<konlpy.tag._kkma.Kkma object at 0x3fff97589b38>
>>>
이제 다른 user를 만들어서 이 /usr/local/miniconda를 쓸 수 있는지 보겠습니다.
root@sys-87926:/home/u0017496# adduser newuser
새로 만들어진 newuser로 login 한 뒤, PATH를 맞춰줍니다.
newuser@sys-87926:~$ vi .bashrc
export PATH=/usr/local/miniconda3/bin:$PATH
export PYTHONPATH=/usr/local/miniconda3/lib/python3.6/site-packages
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el
newuser@sys-87926:~$ . ./.bashrc
보시다시피 다 잘 됩니다.
newuser@sys-87926:~$ conda list
# packages in environment at /usr/local/miniconda3:
#
asn1crypto 0.22.0 py36_0
bazel 0.4.5 0
boto 2.47.0 <pip>
bz2file 0.98 <pip>
...
yaml 0.1.6 0
zlib 1.2.8 3
newuser@sys-87926:~$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el
newuser@sys-87926:~$ python
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 15:31:35)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from konlpy.tag import *
>>> Kkma()
<konlpy.tag._kkma.Kkma object at 0x3fffa4e49780>
>>>
/usr/local/miniconda3의 permission이 아래처럼 755로 되어 있으므로, 아래처럼 newuser라는 새로운 사용자는 conda uninstall이나 install 같은 것은 할 수 없습니다. 만약 다른 사용자들도 자유롭게 install/uninstall을 하기를 원하신다면 그에 맞춰 permission을 조정해주셔야 합니다. 그러나 되도록 그러지 않는 편이 낫겠지요.
u0017496@sys-87926:~$ ls -la /usr/local/miniconda3
total 76
drwxr-xr-x 11 u0017496 u0017496 4096 Jun 23 02:55 .
drwxr-xr-x 13 root root 4096 Jun 22 22:35 ..
drwxr-xr-x 2 u0017496 u0017496 4096 Jun 23 02:20 bin
drwxr-xr-x 2 u0017496 u0017496 12288 Jun 23 02:55 conda-meta
drwxr-xr-x 3 u0017496 u0017496 4096 Jun 23 02:21 envs
drwxr-xr-x 3 u0017496 u0017496 4096 Jun 22 22:43 etc
drwxr-xr-x 8 u0017496 u0017496 4096 Jun 22 22:40 include
drwxr-xr-x 6 u0017496 u0017496 4096 Jun 23 02:12 lib
-rw-r--r-- 1 u0017496 u0017496 3699 Jun 22 22:35 LICENSE.txt
drwxr-xr-x 238 u0017496 u0017496 24576 Jun 23 02:23 pkgs
drwxr-xr-x 4 u0017496 u0017496 4096 Jun 22 22:43 share
drwxr-xr-x 3 u0017496 u0017496 4096 Jun 22 22:43 ssl
newuser@sys-87926:~$ conda uninstall matplotlib
CondaIOError: Missing write permissions in: /usr/local/miniconda3
#
# You don't appear to have the necessary permissions to remove packages
# into the install area '/usr/local/miniconda3'.
# However you can clone this environment into your home directory and
# then make changes to it.
# This may be done using the command:
#
# $ conda create -n my_root --clone="/usr/local/miniconda3"
** 이 작업 및 test 결과는 아래의 Continuum github에도 comment로 올렸습니다.
https://github.com/ContinuumIO/anaconda-issues/issues/270
2017년 6월 23일 금요일
2017년 5월 15일 월요일
Minsky 서버에 Continuum 아나콘다(Anaconda) 설치하기 + Tensorflow로 inception v3 training 해보기
Continuum에서 내놓은 아나콘다(Anaconda)는 여태까지는 x86용으로만 존재했으나, 최근 ARM processor와 POWER8 processor를 위한 min-conda를 내놓았습니다. 아래 site에 해당 package를 download 받기 위한 link와 설치 방법이 정리되어 있습니다.
https://www.continuum.io/content/conda-support-raspberry-pi-2-and-power8-le
불행히도 ppc64le를 위한 link는 잘못 지정되어 있어 '404 Not Found'가 나옵니다만, 실제로는 link만 잘못된 것이고 아래와 같이 file은 실제로 존재합니다. 위의 것은 python version2 용이고, 아래 것은 python 3용입니다.
https://repo.continuum.io/miniconda/Miniconda2-4.3.14-Linux-ppc64le.sh
https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
여기서는 python 3용을 설치해보겠습니다.
u0017496@sys-87250:~$ wget https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
--2017-05-14 21:54:26-- https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
Resolving repo.continuum.io (repo.continuum.io)... 104.16.19.10, 104.16.18.10, 2400:cb00:2048:1::6810:120a, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.16.19.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34765794 (33M) [application/x-sh]
Saving to: ‘Miniconda3-4.3.14-Linux-ppc64le.sh’
Miniconda3-4.3.14-Linux-pp 100%[=====================================>] 33.15M 10.2MB/s in 3.3s
2017-05-17 22:05:18 (10.0 MB/s) - ‘Miniconda3-4.3.14-Linux-ppc64le.sh’ saved [34765794/34765794]
u0017496@sys-87250:~$ chmod a+x Miniconda3-4.3.14-Linux-ppc64le.sh
이제 이 shell script를 수행하면 license 동의 등에 답을 해야 하며, 그 외에도 여러가지 입력 값을 넣어야 합니다. 대부분 그냥 enter를 누르시면 됩니다.
u0017496@sys-87250:~$ ./Miniconda3-4.3.14-Linux-ppc64le.sh
Welcome to Miniconda3 4.3.14 (by Continuum Analytics, Inc.)
In order to continue the installation process, please review the license
agreement.
Please, press ENTER to continue
>>>
(중략)
[/home/u0017496/miniconda3] >>>
PREFIX=/home/u0017496/miniconda3
installing: python-3.6.0-0 ...
installing: cffi-1.9.1-py36_0 ...
installing: conda-env-2.6.0-0 ...
installing: cryptography-1.7.1-py36_0 ...
installing: idna-2.2-py36_0 ...
installing: libffi-3.2.1-1 ...
installing: openssl-1.0.2k-1 ...
installing: pyasn1-0.2.3-py36_0 ...
installing: pycosat-0.6.2-py36_0 ...
installing: pycparser-2.17-py36_0 ...
installing: pyopenssl-16.2.0-py36_0 ...
installing: requests-2.13.0-py36_0 ...
installing: ruamel_yaml-0.11.14-py36_1 ...
installing: setuptools-27.2.0-py36_0 ...
installing: six-1.10.0-py36_0 ...
installing: sqlite-3.13.0-0 ...
installing: xz-5.2.2-1 ...
installing: yaml-0.1.6-0 ...
installing: zlib-1.2.8-3 ...
installing: conda-4.3.14-py36_0 ...
installing: pip-9.0.1-py36_1 ...
installing: wheel-0.29.0-py36_0 ...
Python 3.6.0 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
Do you wish the installer to prepend the Miniconda3 install location
to PATH in your /home/u0017496/.bashrc ? [yes|no]
[no] >>> yes
Prepending PATH=/home/u0017496/miniconda3/bin to PATH in /home/u0017496/.bashrc
A backup will be made to: /home/u0017496/.bashrc-miniconda3.bak
For this change to become active, you have to open a new terminal.
Thank you for installing Miniconda2!
Share your notebooks and packages on Anaconda Cloud!
Sign up for free: https://anaconda.org
이제 mini-conda를 설치했으니 conda 명령을 쓸 수 있어야 합니다. 그러나 보시다시피 conda가 없습니다.
u0017496@sys-87250:~$ which conda
이는 mini-conda 설치시 ~/.bashrc에 conda의 PATH 정보가 자동으로 들어가긴 했지만 .bashrc가 수행되지 않았기 때문에 그런 것입니다. 수행하시면 conda의 PATH가 잡혀 있는 것을 보실 수 있습니다.
u0017496@sys-87250:~$ . ~/.bashrc
u0017496@sys-87250:~$ which conda
/home/u0017496/miniconda3/bin/conda
이제 다음과 같이 ananconda에 포함된 python library들을 보실 수 있습니다.
u0017496@sys-87250:~$ conda list
# packages in environment at /home/u0017496/miniconda3:
#
cffi 1.9.1 py36_0
conda 4.3.14 py36_0
conda-env 2.6.0 0
cryptography 1.7.1 py36_0
idna 2.2 py36_0
libffi 3.2.1 1
openssl 1.0.2k 1
pip 9.0.1 py36_1
pyasn1 0.2.3 py36_0
pycosat 0.6.2 py36_0
pycparser 2.17 py36_0
pyopenssl 16.2.0 py36_0
python 3.6.0 0
requests 2.13.0 py36_0
ruamel_yaml 0.11.14 py36_1
setuptools 27.2.0 py36_0
six 1.10.0 py36_0
sqlite 3.13.0 0
wheel 0.29.0 py36_0
xz 5.2.2 1
yaml 0.1.6 0
zlib 1.2.8 3
현재까지 Continuum에서 빌드해놓은 package들을 모조리 다 설치하는 명령을 다음에 정리했습니다.
u0017496@sys-87250:~$ for i in `conda list | awk '{print $1}' | grep -v \#`
> do
> conda install $i
> done
이제 위에서 설치한 패키지 중 pip가 제대로 설치되었는지 conda search로 확인해보겠습니다. 아래와 같이 * 표시가 된 것이 설치된 것입니다.
u0017496@sys-87250:~$ conda search pip
Fetching package metadata .........
pip 7.1.0 py27_0 defaults
7.1.0 py34_0 defaults
7.1.0 py27_1 defaults
7.1.0 py34_1 defaults
7.1.2 py27_0 defaults
7.1.2 py34_0 defaults
8.1.0 py27_0 defaults
8.1.0 py34_0 defaults
8.1.0 py35_0 defaults
8.1.2 py27_0 defaults
8.1.2 py34_0 defaults
8.1.2 py35_0 defaults
9.0.0 py27_0 defaults
9.0.0 py34_0 defaults
9.0.0 py35_0 defaults
9.0.1 py27_1 defaults
9.0.1 py35_1 defaults
* 9.0.1 py36_1 defaults
이렇게 conda에서 제공하는 pip로 keras 2.0.4를 설치해보겠습니다.
u0017496@sys-87250:~/miniconda3/lib$ pip install keras==2.0.4
Collecting keras==2.0.4
Downloading Keras-2.0.4.tar.gz (199kB)
100% |████████████████████████████████| 204kB 3.1MB/s
Collecting theano (from keras==2.0.4)
Downloading Theano-0.9.0.tar.gz (3.1MB)
100% |████████████████████████████████| 3.1MB 310kB/s
Collecting pyyaml (from keras==2.0.4)
Downloading PyYAML-3.12.tar.gz (253kB)
100% |████████████████████████████████| 256kB 3.6MB/s
Requirement already satisfied: six in ./python3.6/site-packages (from keras==2.0.4)
Requirement already satisfied: numpy>=1.9.1 in ./python3.6/site-packages (from theano->keras==2.0.4)
Requirement already satisfied: scipy>=0.14 in ./python3.6/site-packages (from theano->keras==2.0.4)
Building wheels for collected packages: keras, theano, pyyaml
Running setup.py bdist_wheel for keras ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/48/82/42/f06a8c03a8f95ada523a81ba723e89f059693e6ad868d09727
Running setup.py bdist_wheel for theano ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/d5/5b/93/433299b86e3e9b25f0f600e4e4ebf18e38eb7534ea518eba13
Running setup.py bdist_wheel for pyyaml ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
Successfully built keras theano pyyaml
Installing collected packages: theano, pyyaml, keras
Successfully installed keras-2.0.4 pyyaml-3.12 theano-0.9.0
또 gensim 2.0.0과 KoNLPy를 설치해보겠습니다.
u0017496@sys-87250:~$ pip install gensim==2.0.0
Collecting gensim==2.0.0
Downloading gensim-2.0.0.tar.gz (14.1MB)
100% |████████████████████████████████| 14.2MB 88kB/s
Requirement already satisfied: numpy>=1.3 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: scipy>=0.7.0 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: six>=1.5.0 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: smart_open>=1.2.1 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: boto>=2.32 in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Requirement already satisfied: bz2file in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Requirement already satisfied: requests in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Building wheels for collected packages: gensim
Running setup.py bdist_wheel for gensim ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/e9/5f/e7/4ff23a3fe4b181b44f37eed5602f179c1cc92a0a34f337e745
Successfully built gensim
Installing collected packages: gensim
Found existing installation: gensim 1.0.1
Uninstalling gensim-1.0.1:
Successfully uninstalled gensim-1.0.1
Successfully installed gensim-2.0.0
이제 conda 명령을 통해 추가로 numpy와 matplotlib, scipy와 scikit-learn를 설치해보겠습니다. matplotlib의 prerequisite이 numpy이고, scikit-learn의 prerequisite이 scipy라서 그것들은 자동으로 설치되니까, 실제로는 conda 명령은 두번만 쓰면 됩니다.
u0017496@sys-87250:~$ conda install matplotlib
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
cycler: 0.10.0-py36_0
freetype: 2.5.5-2
libpng: 1.6.27-0
matplotlib: 2.0.2-np112py36_0
numpy: 1.12.1-py36_0
openblas: 0.2.19-0
python-dateutil: 2.6.0-py36_0
pytz: 2017.2-py36_0
Proceed ([y]/n)? y
openblas-0.2.1 100% |###########################################################| Time: 0:00:00 10.21 MB/s
libpng-1.6.27- 100% |###########################################################| Time: 0:00:00 12.75 MB/s
freetype-2.5.5 100% |###########################################################| Time: 0:00:00 10.53 MB/s
numpy-1.12.1-p 100% |###########################################################| Time: 0:00:00 15.12 MB/s
pytz-2017.2-py 100% |###########################################################| Time: 0:00:00 13.25 MB/s
cycler-0.10.0- 100% |###########################################################| Time: 0:00:00 15.61 MB/s
python-dateuti 100% |###########################################################| Time: 0:00:00 6.43 MB/s
matplotlib-2.0 100% |###########################################################| Time: 0:00:00 14.62 MB/s
u0017496@sys-87250:~$ conda install scikit-learn
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
scikit-learn: 0.18.1-np112py36_1
scipy: 0.19.0-np112py36_0
Proceed ([y]/n)? y
scipy-0.19.0-n 100% |###########################################################| Time: 0:00:02 14.75 MB/s
scikit-learn-0 100% |###########################################################| Time: 0:00:00 15.56 MB/s
이렇게 설치된 것들은 아래와 같이 /home/u0017496/miniconda3/lib/python3.6/site-packages 에 들어갑니다.
u0017496@sys-87250:~$ ls /home/u0017496/miniconda3/lib/python3.6/site-packages/
asn1crypto mpl_toolkits python_dateutil-2.6.0-py3.6.egg-info
asn1crypto-0.22.0-py3.6.egg-info numpy-1.12.1.dist-info pytz
cffi OpenSSL pytz-2017.2-py3.6.egg-info
cffi-1.10.0-py3.6.egg-info packaging README.txt
_cffi_backend.so packaging-16.8-py3.6.egg-info requests
conda pip requests-2.14.2-py3.6.egg-info
conda-4.3.18-py3.6.egg-info pip-9.0.1-py3.6.egg-info ruamel_yaml
conda_env pyasn1 scikit_learn-0.18.1-py3.6.egg-info
cryptography pyasn1-0.2.3-py3.6.egg-info scipy
cryptography-1.8.1-py3.6.egg-info __pycache__ scipy-0.19.0-py3.6.egg-info
cycler-0.10.0-py3.6.egg-info pycosat-0.6.2-py3.6.egg-info setuptools-27.2.0-py3.6.egg
cycler.py pycosat.cpython-36m-powerpc64le-linux-gnu.so setuptools.pth
dateutil pycparser six-1.10.0-py3.6.egg-info
easy-install.pth pycparser-2.17-py3.6.egg-info six.py
idna pylab.py sklearn
idna-2.5-py3.6.egg-info pyOpenSSL-17.0.0-py3.6.egg-info test_pycosat.py
matplotlib pyparsing-2.1.4-py3.6.egg-info wheel
matplotlib-2.0.2-py3.6.egg-info pyparsing.py wheel-0.29.0-py3.6.egg-info
따라서 이것들을 사용하기 위해서는 PYTHONPATH는 다음과 같이 설정하시면 됩니다.
u0017496@sys-87250:~$ export PYTHONPATH=/home/u0017496/miniconda3/lib/python3.6/site-packages:$PYTHONPATH
이제 여기에 (PowerAI에 포함된 tensorflow 말고) conda로 bazel, tensorflow 및 tensorflow-gpu도 설치해보겠습니다.
u0017496@sys-87250:~$ conda install bazel
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
bazel: 0.4.5-0
Proceed ([y]/n)? y
bazel-0.4.5-0. 100% |#############################################| Time: 0:00:09 13.37 MB/s
u0017496@sys-87250:~$ conda install tensorflow
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
libprotobuf: 3.2.0-0
protobuf: 3.2.0-py36_0
tensorflow: 1.1.0-np112py36_0
werkzeug: 0.12.2-py36_0
Proceed ([y]/n)? y
libprotobuf-3. 100% |#############################################| Time: 0:00:00 13.84 MB/s
werkzeug-0.12. 100% |#############################################| Time: 0:00:00 18.67 MB/s
protobuf-3.2.0 100% |#############################################| Time: 0:00:00 10.39 MB/s
tensorflow-1.1 100% |#############################################| Time: 0:00:01 15.16 MB/s
u0017496@sys-87250:~$ conda install tensorflow-gpu
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
cudatoolkit: 8.0-0
cudnn: 6.0.21-0
tensorflow-gpu: 1.1.0-np112py36_0
Proceed ([y]/n)? y
cudatoolkit-8. 100% |#############################################| Time: 0:00:29 11.24 MB/s
cudnn-6.0.21-0 100% |#############################################| Time: 0:00:11 15.97 MB/s
tensorflow-gpu 100% |#############################################| Time: 0:00:06 14.27 MB/s
conda list 명령으로 보면 다음과 같은 것들이 설치된 것을 보실 수 있습니다.
u0017496@sys-87250:~$ conda list
# packages in environment at /home/u0017496/miniconda3:
#
asn1crypto 0.22.0 py36_0
bazel 0.4.5 0
boto 2.46.1 py36_0
bz2file 0.98 py36_0
cffi 1.10.0 py36_0
conda 4.3.18 py36_0
conda-env 2.6.0 0
cryptography 1.8.1 py36_0
cudatoolkit 8.0 0
cudnn 6.0.21 0
cycler 0.10.0 py36_0
freetype 2.5.5 2
gensim 1.0.1 np112py36_0
gensim 2.0.0 <pip>
idna 2.5 py36_0
Keras 2.0.4 <pip>
konlpy 0.4.4 <pip>
libffi 3.2.1 1
libpng 1.6.27 0
libprotobuf 3.2.0 0
matplotlib 2.0.2 np112py36_0
numpy 1.12.1 <pip>
numpy 1.12.1 py36_0
openblas 0.2.19 0
openssl 1.0.2k 2
packaging 16.8 py36_0
pip 9.0.1 py36_1
protobuf 3.2.0 py36_0
pyasn1 0.2.3 py36_0
pycosat 0.6.2 py36_0
pycparser 2.17 py36_0
pyopenssl 17.0.0 py36_0
pyparsing 2.1.4 py36_0
python 3.6.1 2
python-dateutil 2.6.0 py36_0
pytz 2017.2 py36_0
PyYAML 3.12 <pip>
requests 2.14.2 py36_0
ruamel_yaml 0.11.14 py36_1
scikit-learn 0.18.1 np112py36_1
scipy 0.19.0 np112py36_0
setuptools 27.2.0 py36_0
six 1.10.0 py36_0
smart_open 1.5.2 py36_0
sqlite 3.13.0 0
tensorflow 1.1.0 np112py36_0
tensorflow-gpu 1.1.0 np112py36_0
Theano 0.9.0 <pip>
werkzeug 0.12.2 py36_0
wheel 0.29.0 py36_0
xz 5.2.2 1
yaml 0.1.6 0
zlib 1.2.8 3
설치하는 김에, 이렇게 conda로 설치한 tensorflow를 이용하여 inception v3 model을 training 해보겠습니다. 다음 순서대로 따라 하시면 됩니다.
u0017496@sys-87250:~/inception$ pwd
/home/u0017496/inception
u0017496@sys-87250:~/inception$ export INCEPTION_DIR=/home/u0017496/inception
u0017496@sys-87250:~/inception$ curl -O http://download.tensorflow.org/models/image/imagenet/inception-v3-2016-03-01.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 380M 100 380M 0 0 5918k 0 0:01:05 0:01:05 --:--:-- 4233k
u0017496@sys-87250:~/inception$ tar -xvf inception-v3-2016-03-01.tar.gz
inception-v3/
inception-v3/checkpoint
inception-v3/README.txt
inception-v3/model.ckpt-157585
u0017496@sys-87250:~/inception$ git clone https://github.com/tensorflow/models.git
Cloning into 'models'...
remote: Counting objects: 4703, done.
remote: Compressing objects: 100% (43/43), done.
remote: Total 4703 (delta 17), reused 31 (delta 11), pack-reused 4649
Receiving objects: 100% (4703/4703), 153.34 MiB | 5.62 MiB/s, done.
Resolving deltas: 100% (2374/2374), done.
Checking connectivity... done.
u0017496@sys-87250:~/inception/models/inception$ export FLOWERS_DIR=/home/u0017496/inception/models/inception
u0017496@sys-87250:~/inception/models/inception$ mkdir -p $FLOWERS_DIR/data
u0017496@sys-87250:~/inception/models/inception$ which bazel
/home/u0017496/miniconda3/bin/bazel
u0017496@sys-87250:~/inception/models/inception$ bazel build inception/download_and_preprocess_flowers
Extracting Bazel installation...
....................
INFO: Found 1 target...
Target //inception:download_and_preprocess_flowers up-to-date:
bazel-bin/inception/download_and_preprocess_flowers
INFO: Elapsed time: 6.943s, Critical Path: 0.05s
u0017496@sys-87250:~/inception/models/inception$ export TEST_TMPDIR=/home/u0017496/.cache
u0017496@sys-87250:~/inception/models/inception$ bazel build inception/download_and_preprocess_flowers
INFO: $TEST_TMPDIR defined: output root default is '/home/u0017496/.cache'.
Extracting Bazel installation...
.............
INFO: Found 1 target...
Target //inception:download_and_preprocess_flowers up-to-date:
bazel-bin/inception/download_and_preprocess_flowers
INFO: Elapsed time: 4.867s, Critical Path: 0.03s
u0017496@sys-87250:~/inception/models/inception$ bazel-bin/inception/download_and_preprocess_flowers $FLOWERS_DIR/data
Downloading flower data set.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 218M 100 218M 0 0 9372k 0 0:00:23 0:00:23 --:--:-- 10.1M
(중략)
Found 3170 JPEG files across 5 labels inside /home/u0017496/inception/models/inception/data/raw-data/train.
Launching 2 threads for spacings: [[0, 1585], [1585, 3170]]
2017-05-19 05:33:44.191446 [thread 0]: Processed 1000 of 1585 images in thread batch.
2017-05-19 05:33:44.213856 [thread 1]: Processed 1000 of 1585 images in thread batch.
2017-05-19 05:33:54.902070 [thread 1]: Wrote 1585 images to /home/u0017496/inception/models/inception/data/train-00001-of-00002
2017-05-19 05:33:54.902172 [thread 1]: Wrote 1585 images to 1585 shards.
2017-05-19 05:33:54.911283 [thread 0]: Wrote 1585 images to /home/u0017496/inception/models/inception/data/train-00000-of-00002
2017-05-19 05:33:54.911360 [thread 0]: Wrote 1585 images to 1585 shards.
2017-05-19 05:33:55.171141: Finished writing all 3170 images in data set.
아래에서 보시다시피 이 inception v3는 꽃 사진을 분류하는 neural network입니다.
u0017496@sys-87250:~/inception/models/inception$ du -sm data/raw-data/train/*
29 data/raw-data/train/daisy
44 data/raw-data/train/dandelion
1 data/raw-data/train/LICENSE.txt
33 data/raw-data/train/roses
47 data/raw-data/train/sunflowers
48 data/raw-data/train/tulips
u0017496@sys-87250:~/inception/models/inception$ bazel build inception/flowers_train
INFO: $TEST_TMPDIR defined: output root default is '/home/u0017496/.cache'.
............................
INFO: Found 1 target...
Target //inception:flowers_train up-to-date:
bazel-bin/inception/flowers_train
INFO: Elapsed time: 6.502s, Critical Path: 0.03s
이제 비로소 inception v3의 training 준비가 끝났습니다. 이제 다음 명령으로 training을 시작합니다.
u0017496@sys-87250:~/inception/models/inception$ time bazel-bin/inception/flowers_train --train_dir=$FLOWERS_DIR/train --data_dir=$FLOWERS_DIR/data --pretrained_model_checkpoint_path=$INCEPTION_DIR/inception-v3/model.ckpt-157585 --fine_tune=True --initial_learning_rate=0.001 -input_queue_memory_factor=1 --max_steps=50 --num_gpus 1 --batch_size=32
NVIDIA: no NVIDIA devices found
2017-05-19 05:41:03.740213: E tensorflow/stream_executor/cuda/cuda_driver.cc:405] failed call to cuInit: CUDA_ERROR_UNKNOWN
2017-05-19 05:41:03.740670: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:145] kernel driver does not appear to be running on this host (sys-87250): /proc/driver/nvidia/version does not exist
2017-05-19 05:41:51.947244: Pre-trained model restored from /home/u0017496/inception/inception-v3/model.ckpt-157585
2017-05-19 05:47:22.023602: step 0, loss = 2.79 (0.2 examples/sec; 182.713 sec/batch)
2017-05-19 06:05:58.942671: step 10, loss = 2.53 (0.4 examples/sec; 78.882 sec/batch)
2017-05-19 06:19:26.875533: step 20, loss = 2.40 (0.4 examples/sec; 82.410 sec/batch)
2017-05-19 06:33:10.333275: step 30, loss = 2.20 (0.4 examples/sec; 77.844 sec/batch)
2017-05-19 06:48:27.688993: step 40, loss = 2.24 (0.3 examples/sec; 96.148 sec/batch)
real 84m30.882s
user 135m20.864s
sys 2m30.832s
이제 와서 고백하지만 제가 설치 demo를 보여드린 이 서버는 사실 GPU가 달려 있지 않은 POWER8 서버입니다. GPU가 없는 경우 CPU를 이용하게 되는데, 그런 경우 이 training의 완료는 보시다시피 매우, 매우 오래 걸립니다. 저 output을 보면 초당 example 0.4개 처리로 나옵니다만, P100을 이용하는 경우 (GPU 개수 및 batch size에 따라) 초당 50개~200개 단위로 처리가 됩니다.
아래는 전에 PowerAI를 설치한 Minsky 서버에서 수행했던 inception v3의 결과 log 일부입니다.
2017-05-16 03:48:46.352210: Pre-trained model restored from /gpfs/gpfs_gl4_16mb/b7p088za/inception-v3/model.ckpt-157585
2017-05-16 03:52:44.322381: step 0, loss = 2.72 (17.6 examples/sec; 21.830 sec/batch)
2017-05-16 03:55:29.550791: step 10, loss = 2.57 (213.6 examples/sec; 1.797 sec/batch)
2017-05-16 03:55:47.619990: step 20, loss = 2.35 (212.1 examples/sec; 1.810 sec/batch)
2017-05-16 03:56:05.953991: step 30, loss = 2.17 (206.6 examples/sec; 1.859 sec/batch)
2017-05-16 03:56:24.306742: step 40, loss = 1.98 (209.4 examples/sec; 1.834 sec/batch)
2017-05-16 03:56:42.490063: step 50, loss = 1.92 (217.8 examples/sec; 1.763 sec/batch)
2017-05-16 03:57:00.444537: step 60, loss = 1.67 (216.6 examples/sec; 1.773 sec/batch)
2017-05-16 03:57:18.366941: step 70, loss = 1.58 (212.7 examples/sec; 1.806 sec/batch)
2017-05-16 03:57:36.467837: step 80, loss = 1.55 (213.6 examples/sec; 1.798 sec/batch)
https://www.continuum.io/content/conda-support-raspberry-pi-2-and-power8-le
불행히도 ppc64le를 위한 link는 잘못 지정되어 있어 '404 Not Found'가 나옵니다만, 실제로는 link만 잘못된 것이고 아래와 같이 file은 실제로 존재합니다. 위의 것은 python version2 용이고, 아래 것은 python 3용입니다.
https://repo.continuum.io/miniconda/Miniconda2-4.3.14-Linux-ppc64le.sh
https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
여기서는 python 3용을 설치해보겠습니다.
u0017496@sys-87250:~$ wget https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
--2017-05-14 21:54:26-- https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-ppc64le.sh
Resolving repo.continuum.io (repo.continuum.io)... 104.16.19.10, 104.16.18.10, 2400:cb00:2048:1::6810:120a, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.16.19.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34765794 (33M) [application/x-sh]
Saving to: ‘Miniconda3-4.3.14-Linux-ppc64le.sh’
Miniconda3-4.3.14-Linux-pp 100%[=====================================>] 33.15M 10.2MB/s in 3.3s
2017-05-17 22:05:18 (10.0 MB/s) - ‘Miniconda3-4.3.14-Linux-ppc64le.sh’ saved [34765794/34765794]
u0017496@sys-87250:~$ chmod a+x Miniconda3-4.3.14-Linux-ppc64le.sh
이제 이 shell script를 수행하면 license 동의 등에 답을 해야 하며, 그 외에도 여러가지 입력 값을 넣어야 합니다. 대부분 그냥 enter를 누르시면 됩니다.
u0017496@sys-87250:~$ ./Miniconda3-4.3.14-Linux-ppc64le.sh
Welcome to Miniconda3 4.3.14 (by Continuum Analytics, Inc.)
In order to continue the installation process, please review the license
agreement.
Please, press ENTER to continue
>>>
(중략)
[/home/u0017496/miniconda3] >>>
PREFIX=/home/u0017496/miniconda3
installing: python-3.6.0-0 ...
installing: cffi-1.9.1-py36_0 ...
installing: conda-env-2.6.0-0 ...
installing: cryptography-1.7.1-py36_0 ...
installing: idna-2.2-py36_0 ...
installing: libffi-3.2.1-1 ...
installing: openssl-1.0.2k-1 ...
installing: pyasn1-0.2.3-py36_0 ...
installing: pycosat-0.6.2-py36_0 ...
installing: pycparser-2.17-py36_0 ...
installing: pyopenssl-16.2.0-py36_0 ...
installing: requests-2.13.0-py36_0 ...
installing: ruamel_yaml-0.11.14-py36_1 ...
installing: setuptools-27.2.0-py36_0 ...
installing: six-1.10.0-py36_0 ...
installing: sqlite-3.13.0-0 ...
installing: xz-5.2.2-1 ...
installing: yaml-0.1.6-0 ...
installing: zlib-1.2.8-3 ...
installing: conda-4.3.14-py36_0 ...
installing: pip-9.0.1-py36_1 ...
installing: wheel-0.29.0-py36_0 ...
Python 3.6.0 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
Do you wish the installer to prepend the Miniconda3 install location
to PATH in your /home/u0017496/.bashrc ? [yes|no]
Prepending PATH=/home/u0017496/miniconda3/bin to PATH in /home/u0017496/.bashrc
A backup will be made to: /home/u0017496/.bashrc-miniconda3.bak
For this change to become active, you have to open a new terminal.
Thank you for installing Miniconda2!
Share your notebooks and packages on Anaconda Cloud!
Sign up for free: https://anaconda.org
이제 mini-conda를 설치했으니 conda 명령을 쓸 수 있어야 합니다. 그러나 보시다시피 conda가 없습니다.
u0017496@sys-87250:~$ which conda
이는 mini-conda 설치시 ~/.bashrc에 conda의 PATH 정보가 자동으로 들어가긴 했지만 .bashrc가 수행되지 않았기 때문에 그런 것입니다. 수행하시면 conda의 PATH가 잡혀 있는 것을 보실 수 있습니다.
u0017496@sys-87250:~$ . ~/.bashrc
u0017496@sys-87250:~$ which conda
/home/u0017496/miniconda3/bin/conda
이제 다음과 같이 ananconda에 포함된 python library들을 보실 수 있습니다.
u0017496@sys-87250:~$ conda list
# packages in environment at /home/u0017496/miniconda3:
#
cffi 1.9.1 py36_0
conda 4.3.14 py36_0
conda-env 2.6.0 0
cryptography 1.7.1 py36_0
idna 2.2 py36_0
libffi 3.2.1 1
openssl 1.0.2k 1
pip 9.0.1 py36_1
pyasn1 0.2.3 py36_0
pycosat 0.6.2 py36_0
pycparser 2.17 py36_0
pyopenssl 16.2.0 py36_0
python 3.6.0 0
requests 2.13.0 py36_0
ruamel_yaml 0.11.14 py36_1
setuptools 27.2.0 py36_0
six 1.10.0 py36_0
sqlite 3.13.0 0
wheel 0.29.0 py36_0
xz 5.2.2 1
yaml 0.1.6 0
zlib 1.2.8 3
현재까지 Continuum에서 빌드해놓은 package들을 모조리 다 설치하는 명령을 다음에 정리했습니다.
u0017496@sys-87250:~$ for i in `conda list | awk '{print $1}' | grep -v \#`
> do
> conda install $i
> done
(중략)
이제 위에서 설치한 패키지 중 pip가 제대로 설치되었는지 conda search로 확인해보겠습니다. 아래와 같이 * 표시가 된 것이 설치된 것입니다.
u0017496@sys-87250:~$ conda search pip
Fetching package metadata .........
pip 7.1.0 py27_0 defaults
7.1.0 py34_0 defaults
7.1.0 py27_1 defaults
7.1.0 py34_1 defaults
7.1.2 py27_0 defaults
7.1.2 py34_0 defaults
8.1.0 py27_0 defaults
8.1.0 py34_0 defaults
8.1.0 py35_0 defaults
8.1.2 py27_0 defaults
8.1.2 py34_0 defaults
8.1.2 py35_0 defaults
9.0.0 py27_0 defaults
9.0.0 py34_0 defaults
9.0.0 py35_0 defaults
9.0.1 py27_1 defaults
9.0.1 py35_1 defaults
* 9.0.1 py36_1 defaults
u0017496@sys-87250:~$ which pip
/home/u0017496/miniconda3/bin/pip
u0017496@sys-87250:~$ pip --version
pip 9.0.1 from /home/u0017496/miniconda3/lib/python3.6/site-packages (python 3.6)
이렇게 conda에서 제공하는 pip로 keras 2.0.4를 설치해보겠습니다.
u0017496@sys-87250:~/miniconda3/lib$ pip install keras==2.0.4
Collecting keras==2.0.4
Downloading Keras-2.0.4.tar.gz (199kB)
100% |████████████████████████████████| 204kB 3.1MB/s
Collecting theano (from keras==2.0.4)
Downloading Theano-0.9.0.tar.gz (3.1MB)
100% |████████████████████████████████| 3.1MB 310kB/s
Collecting pyyaml (from keras==2.0.4)
Downloading PyYAML-3.12.tar.gz (253kB)
100% |████████████████████████████████| 256kB 3.6MB/s
Requirement already satisfied: six in ./python3.6/site-packages (from keras==2.0.4)
Requirement already satisfied: numpy>=1.9.1 in ./python3.6/site-packages (from theano->keras==2.0.4)
Requirement already satisfied: scipy>=0.14 in ./python3.6/site-packages (from theano->keras==2.0.4)
Building wheels for collected packages: keras, theano, pyyaml
Running setup.py bdist_wheel for keras ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/48/82/42/f06a8c03a8f95ada523a81ba723e89f059693e6ad868d09727
Running setup.py bdist_wheel for theano ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/d5/5b/93/433299b86e3e9b25f0f600e4e4ebf18e38eb7534ea518eba13
Running setup.py bdist_wheel for pyyaml ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
Successfully built keras theano pyyaml
Installing collected packages: theano, pyyaml, keras
Successfully installed keras-2.0.4 pyyaml-3.12 theano-0.9.0
또 gensim 2.0.0과 KoNLPy를 설치해보겠습니다.
u0017496@sys-87250:~$ pip install gensim==2.0.0
Collecting gensim==2.0.0
Downloading gensim-2.0.0.tar.gz (14.1MB)
100% |████████████████████████████████| 14.2MB 88kB/s
Requirement already satisfied: numpy>=1.3 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: scipy>=0.7.0 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: six>=1.5.0 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: smart_open>=1.2.1 in ./miniconda3/lib/python3.6/site-packages (from gensim==2.0.0)
Requirement already satisfied: boto>=2.32 in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Requirement already satisfied: bz2file in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Requirement already satisfied: requests in ./miniconda3/lib/python3.6/site-packages (from smart_open>=1.2.1->gensim==2.0.0)
Building wheels for collected packages: gensim
Running setup.py bdist_wheel for gensim ... done
Stored in directory: /home/u0017496/.cache/pip/wheels/e9/5f/e7/4ff23a3fe4b181b44f37eed5602f179c1cc92a0a34f337e745
Successfully built gensim
Installing collected packages: gensim
Found existing installation: gensim 1.0.1
Uninstalling gensim-1.0.1:
Successfully uninstalled gensim-1.0.1
Successfully installed gensim-2.0.0
u0017496@sys-87250:~$ pip install konlpy
Collecting konlpy
Downloading konlpy-0.4.4-py2.py3-none-any.whl (22.5MB)
100% |████████████████████████████████| 22.5MB 57kB/s
Installing collected packages: konlpy
Successfully installed konlpy-0.4.4
이제 conda 명령을 통해 추가로 numpy와 matplotlib, scipy와 scikit-learn를 설치해보겠습니다. matplotlib의 prerequisite이 numpy이고, scikit-learn의 prerequisite이 scipy라서 그것들은 자동으로 설치되니까, 실제로는 conda 명령은 두번만 쓰면 됩니다.
u0017496@sys-87250:~$ conda install matplotlib
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
cycler: 0.10.0-py36_0
freetype: 2.5.5-2
libpng: 1.6.27-0
matplotlib: 2.0.2-np112py36_0
numpy: 1.12.1-py36_0
openblas: 0.2.19-0
python-dateutil: 2.6.0-py36_0
pytz: 2017.2-py36_0
Proceed ([y]/n)? y
openblas-0.2.1 100% |###########################################################| Time: 0:00:00 10.21 MB/s
libpng-1.6.27- 100% |###########################################################| Time: 0:00:00 12.75 MB/s
freetype-2.5.5 100% |###########################################################| Time: 0:00:00 10.53 MB/s
numpy-1.12.1-p 100% |###########################################################| Time: 0:00:00 15.12 MB/s
pytz-2017.2-py 100% |###########################################################| Time: 0:00:00 13.25 MB/s
cycler-0.10.0- 100% |###########################################################| Time: 0:00:00 15.61 MB/s
python-dateuti 100% |###########################################################| Time: 0:00:00 6.43 MB/s
matplotlib-2.0 100% |###########################################################| Time: 0:00:00 14.62 MB/s
u0017496@sys-87250:~$ conda install scikit-learn
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
scikit-learn: 0.18.1-np112py36_1
scipy: 0.19.0-np112py36_0
Proceed ([y]/n)? y
scipy-0.19.0-n 100% |###########################################################| Time: 0:00:02 14.75 MB/s
scikit-learn-0 100% |###########################################################| Time: 0:00:00 15.56 MB/s
이렇게 설치된 것들은 아래와 같이 /home/u0017496/miniconda3/lib/python3.6/site-packages 에 들어갑니다.
u0017496@sys-87250:~$ ls /home/u0017496/miniconda3/lib/python3.6/site-packages/
asn1crypto mpl_toolkits python_dateutil-2.6.0-py3.6.egg-info
asn1crypto-0.22.0-py3.6.egg-info numpy-1.12.1.dist-info pytz
cffi OpenSSL pytz-2017.2-py3.6.egg-info
cffi-1.10.0-py3.6.egg-info packaging README.txt
_cffi_backend.so packaging-16.8-py3.6.egg-info requests
conda pip requests-2.14.2-py3.6.egg-info
conda-4.3.18-py3.6.egg-info pip-9.0.1-py3.6.egg-info ruamel_yaml
conda_env pyasn1 scikit_learn-0.18.1-py3.6.egg-info
cryptography pyasn1-0.2.3-py3.6.egg-info scipy
cryptography-1.8.1-py3.6.egg-info __pycache__ scipy-0.19.0-py3.6.egg-info
cycler-0.10.0-py3.6.egg-info pycosat-0.6.2-py3.6.egg-info setuptools-27.2.0-py3.6.egg
cycler.py pycosat.cpython-36m-powerpc64le-linux-gnu.so setuptools.pth
dateutil pycparser six-1.10.0-py3.6.egg-info
easy-install.pth pycparser-2.17-py3.6.egg-info six.py
idna pylab.py sklearn
idna-2.5-py3.6.egg-info pyOpenSSL-17.0.0-py3.6.egg-info test_pycosat.py
matplotlib pyparsing-2.1.4-py3.6.egg-info wheel
matplotlib-2.0.2-py3.6.egg-info pyparsing.py wheel-0.29.0-py3.6.egg-info
따라서 이것들을 사용하기 위해서는 PYTHONPATH는 다음과 같이 설정하시면 됩니다.
u0017496@sys-87250:~$ export PYTHONPATH=/home/u0017496/miniconda3/lib/python3.6/site-packages:$PYTHONPATH
이제 여기에 (PowerAI에 포함된 tensorflow 말고) conda로 bazel, tensorflow 및 tensorflow-gpu도 설치해보겠습니다.
u0017496@sys-87250:~$ conda install bazel
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
bazel: 0.4.5-0
Proceed ([y]/n)? y
bazel-0.4.5-0. 100% |#############################################| Time: 0:00:09 13.37 MB/s
u0017496@sys-87250:~$ conda install tensorflow
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
libprotobuf: 3.2.0-0
protobuf: 3.2.0-py36_0
tensorflow: 1.1.0-np112py36_0
werkzeug: 0.12.2-py36_0
Proceed ([y]/n)? y
libprotobuf-3. 100% |#############################################| Time: 0:00:00 13.84 MB/s
werkzeug-0.12. 100% |#############################################| Time: 0:00:00 18.67 MB/s
protobuf-3.2.0 100% |#############################################| Time: 0:00:00 10.39 MB/s
tensorflow-1.1 100% |#############################################| Time: 0:00:01 15.16 MB/s
u0017496@sys-87250:~$ conda install tensorflow-gpu
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda3:
The following NEW packages will be INSTALLED:
cudatoolkit: 8.0-0
cudnn: 6.0.21-0
tensorflow-gpu: 1.1.0-np112py36_0
Proceed ([y]/n)? y
cudatoolkit-8. 100% |#############################################| Time: 0:00:29 11.24 MB/s
cudnn-6.0.21-0 100% |#############################################| Time: 0:00:11 15.97 MB/s
tensorflow-gpu 100% |#############################################| Time: 0:00:06 14.27 MB/s
conda list 명령으로 보면 다음과 같은 것들이 설치된 것을 보실 수 있습니다.
u0017496@sys-87250:~$ conda list
# packages in environment at /home/u0017496/miniconda3:
#
asn1crypto 0.22.0 py36_0
bazel 0.4.5 0
boto 2.46.1 py36_0
bz2file 0.98 py36_0
cffi 1.10.0 py36_0
conda 4.3.18 py36_0
conda-env 2.6.0 0
cryptography 1.8.1 py36_0
cudatoolkit 8.0 0
cudnn 6.0.21 0
cycler 0.10.0 py36_0
freetype 2.5.5 2
gensim 1.0.1 np112py36_0
gensim 2.0.0 <pip>
idna 2.5 py36_0
Keras 2.0.4 <pip>
konlpy 0.4.4 <pip>
libffi 3.2.1 1
libpng 1.6.27 0
libprotobuf 3.2.0 0
matplotlib 2.0.2 np112py36_0
numpy 1.12.1 <pip>
numpy 1.12.1 py36_0
openblas 0.2.19 0
openssl 1.0.2k 2
packaging 16.8 py36_0
pip 9.0.1 py36_1
protobuf 3.2.0 py36_0
pyasn1 0.2.3 py36_0
pycosat 0.6.2 py36_0
pycparser 2.17 py36_0
pyopenssl 17.0.0 py36_0
pyparsing 2.1.4 py36_0
python 3.6.1 2
python-dateutil 2.6.0 py36_0
pytz 2017.2 py36_0
PyYAML 3.12 <pip>
requests 2.14.2 py36_0
ruamel_yaml 0.11.14 py36_1
scikit-learn 0.18.1 np112py36_1
scipy 0.19.0 np112py36_0
setuptools 27.2.0 py36_0
six 1.10.0 py36_0
smart_open 1.5.2 py36_0
sqlite 3.13.0 0
tensorflow 1.1.0 np112py36_0
tensorflow-gpu 1.1.0 np112py36_0
Theano 0.9.0 <pip>
werkzeug 0.12.2 py36_0
wheel 0.29.0 py36_0
xz 5.2.2 1
yaml 0.1.6 0
zlib 1.2.8 3
설치하는 김에, 이렇게 conda로 설치한 tensorflow를 이용하여 inception v3 model을 training 해보겠습니다. 다음 순서대로 따라 하시면 됩니다.
u0017496@sys-87250:~/inception$ pwd
/home/u0017496/inception
u0017496@sys-87250:~/inception$ export INCEPTION_DIR=/home/u0017496/inception
u0017496@sys-87250:~/inception$ curl -O http://download.tensorflow.org/models/image/imagenet/inception-v3-2016-03-01.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 380M 100 380M 0 0 5918k 0 0:01:05 0:01:05 --:--:-- 4233k
u0017496@sys-87250:~/inception$ tar -xvf inception-v3-2016-03-01.tar.gz
inception-v3/
inception-v3/checkpoint
inception-v3/README.txt
inception-v3/model.ckpt-157585
u0017496@sys-87250:~/inception$ git clone https://github.com/tensorflow/models.git
Cloning into 'models'...
remote: Counting objects: 4703, done.
remote: Compressing objects: 100% (43/43), done.
remote: Total 4703 (delta 17), reused 31 (delta 11), pack-reused 4649
Receiving objects: 100% (4703/4703), 153.34 MiB | 5.62 MiB/s, done.
Resolving deltas: 100% (2374/2374), done.
Checking connectivity... done.
u0017496@sys-87250:~/inception/models/inception$ export FLOWERS_DIR=/home/u0017496/inception/models/inception
u0017496@sys-87250:~/inception/models/inception$ mkdir -p $FLOWERS_DIR/data
u0017496@sys-87250:~/inception/models/inception$ which bazel
/home/u0017496/miniconda3/bin/bazel
u0017496@sys-87250:~/inception/models/inception$ bazel build inception/download_and_preprocess_flowers
Extracting Bazel installation...
....................
INFO: Found 1 target...
Target //inception:download_and_preprocess_flowers up-to-date:
bazel-bin/inception/download_and_preprocess_flowers
INFO: Elapsed time: 6.943s, Critical Path: 0.05s
u0017496@sys-87250:~/inception/models/inception$ export TEST_TMPDIR=/home/u0017496/.cache
u0017496@sys-87250:~/inception/models/inception$ bazel build inception/download_and_preprocess_flowers
INFO: $TEST_TMPDIR defined: output root default is '/home/u0017496/.cache'.
Extracting Bazel installation...
.............
INFO: Found 1 target...
Target //inception:download_and_preprocess_flowers up-to-date:
bazel-bin/inception/download_and_preprocess_flowers
INFO: Elapsed time: 4.867s, Critical Path: 0.03s
u0017496@sys-87250:~/inception/models/inception$ bazel-bin/inception/download_and_preprocess_flowers $FLOWERS_DIR/data
Downloading flower data set.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 218M 100 218M 0 0 9372k 0 0:00:23 0:00:23 --:--:-- 10.1M
(중략)
Found 3170 JPEG files across 5 labels inside /home/u0017496/inception/models/inception/data/raw-data/train.
Launching 2 threads for spacings: [[0, 1585], [1585, 3170]]
2017-05-19 05:33:44.191446 [thread 0]: Processed 1000 of 1585 images in thread batch.
2017-05-19 05:33:44.213856 [thread 1]: Processed 1000 of 1585 images in thread batch.
2017-05-19 05:33:54.902070 [thread 1]: Wrote 1585 images to /home/u0017496/inception/models/inception/data/train-00001-of-00002
2017-05-19 05:33:54.902172 [thread 1]: Wrote 1585 images to 1585 shards.
2017-05-19 05:33:54.911283 [thread 0]: Wrote 1585 images to /home/u0017496/inception/models/inception/data/train-00000-of-00002
2017-05-19 05:33:54.911360 [thread 0]: Wrote 1585 images to 1585 shards.
2017-05-19 05:33:55.171141: Finished writing all 3170 images in data set.
아래에서 보시다시피 이 inception v3는 꽃 사진을 분류하는 neural network입니다.
u0017496@sys-87250:~/inception/models/inception$ du -sm data/raw-data/train/*
29 data/raw-data/train/daisy
44 data/raw-data/train/dandelion
1 data/raw-data/train/LICENSE.txt
33 data/raw-data/train/roses
47 data/raw-data/train/sunflowers
48 data/raw-data/train/tulips
u0017496@sys-87250:~/inception/models/inception$ bazel build inception/flowers_train
INFO: $TEST_TMPDIR defined: output root default is '/home/u0017496/.cache'.
............................
INFO: Found 1 target...
Target //inception:flowers_train up-to-date:
bazel-bin/inception/flowers_train
INFO: Elapsed time: 6.502s, Critical Path: 0.03s
이제 비로소 inception v3의 training 준비가 끝났습니다. 이제 다음 명령으로 training을 시작합니다.
u0017496@sys-87250:~/inception/models/inception$ time bazel-bin/inception/flowers_train --train_dir=$FLOWERS_DIR/train --data_dir=$FLOWERS_DIR/data --pretrained_model_checkpoint_path=$INCEPTION_DIR/inception-v3/model.ckpt-157585 --fine_tune=True --initial_learning_rate=0.001 -input_queue_memory_factor=1 --max_steps=50 --num_gpus 1 --batch_size=32
NVIDIA: no NVIDIA devices found
2017-05-19 05:41:03.740213: E tensorflow/stream_executor/cuda/cuda_driver.cc:405] failed call to cuInit: CUDA_ERROR_UNKNOWN
2017-05-19 05:41:03.740670: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:145] kernel driver does not appear to be running on this host (sys-87250): /proc/driver/nvidia/version does not exist
2017-05-19 05:41:51.947244: Pre-trained model restored from /home/u0017496/inception/inception-v3/model.ckpt-157585
2017-05-19 05:47:22.023602: step 0, loss = 2.79 (0.2 examples/sec; 182.713 sec/batch)
2017-05-19 06:05:58.942671: step 10, loss = 2.53 (0.4 examples/sec; 78.882 sec/batch)
2017-05-19 06:19:26.875533: step 20, loss = 2.40 (0.4 examples/sec; 82.410 sec/batch)
2017-05-19 06:33:10.333275: step 30, loss = 2.20 (0.4 examples/sec; 77.844 sec/batch)
2017-05-19 06:48:27.688993: step 40, loss = 2.24 (0.3 examples/sec; 96.148 sec/batch)
real 84m30.882s
user 135m20.864s
sys 2m30.832s
이제 와서 고백하지만 제가 설치 demo를 보여드린 이 서버는 사실 GPU가 달려 있지 않은 POWER8 서버입니다. GPU가 없는 경우 CPU를 이용하게 되는데, 그런 경우 이 training의 완료는 보시다시피 매우, 매우 오래 걸립니다. 저 output을 보면 초당 example 0.4개 처리로 나옵니다만, P100을 이용하는 경우 (GPU 개수 및 batch size에 따라) 초당 50개~200개 단위로 처리가 됩니다.
아래는 전에 PowerAI를 설치한 Minsky 서버에서 수행했던 inception v3의 결과 log 일부입니다.
2017-05-16 03:48:46.352210: Pre-trained model restored from /gpfs/gpfs_gl4_16mb/b7p088za/inception-v3/model.ckpt-157585
2017-05-16 03:52:44.322381: step 0, loss = 2.72 (17.6 examples/sec; 21.830 sec/batch)
2017-05-16 03:55:29.550791: step 10, loss = 2.57 (213.6 examples/sec; 1.797 sec/batch)
2017-05-16 03:55:47.619990: step 20, loss = 2.35 (212.1 examples/sec; 1.810 sec/batch)
2017-05-16 03:56:05.953991: step 30, loss = 2.17 (206.6 examples/sec; 1.859 sec/batch)
2017-05-16 03:56:24.306742: step 40, loss = 1.98 (209.4 examples/sec; 1.834 sec/batch)
2017-05-16 03:56:42.490063: step 50, loss = 1.92 (217.8 examples/sec; 1.763 sec/batch)
2017-05-16 03:57:00.444537: step 60, loss = 1.67 (216.6 examples/sec; 1.773 sec/batch)
2017-05-16 03:57:18.366941: step 70, loss = 1.58 (212.7 examples/sec; 1.806 sec/batch)
2017-05-16 03:57:36.467837: step 80, loss = 1.55 (213.6 examples/sec; 1.798 sec/batch)
피드 구독하기:
글 (Atom)