Minsky의 아키텍처인 ppc64le(IBM POWER8)에서도 사용 가능한 anti-virus SW가 있습니다. CLAM Anti-Virus (clamav)입니다.
CLAMAV는 open source 기반의 anti-virus SW로서, 다음이 홈페이지로 되어 있고, source를 download 받을 수도 있습니다.
http://www.clamav.net/
ppc64le에서 빌드하는 방법도 매우 간단하여, 그냥 ./configure && make && sudo make install 만 해주시면 됩니다.
그러나 deep learning에서 주로 사용하는 Ubuntu에는 아예 OS의 표준 apt repository에 포함되어 있어 손쉽게 설치 및 사용이 가능합니다.
설치는 다음과 같이 apt-get install 명령으로 하시면 됩니다.
u0017649@sys-89983:~$ sudo apt-get install clamav clamav-daemon clamav-freshclam clamav-base libclamav-dev clamav-testfiles
clamav-daemon은 다음과 같이 start 하시면 됩니다.
u0017649@sys-89983:~$ sudo systemctl start clamav-daemon.service
u0017649@sys-89983:~$ sudo systemctl status clamav-daemon.service
● clamav-daemon.service - Clam AntiVirus userspace daemon
Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2017-12-17 21:07:57 EST; 4s ago
Docs: man:clamd(8)
man:clamd.conf(5)
http://www.clamav.net/lang/en/doc/
Main PID: 9462 (clamd)
Tasks: 1
Memory: 234.0M
CPU: 4.121s
CGroup: /system.slice/clamav-daemon.service
└─9462 /usr/sbin/clamd --foreground=true
/home/u0017649/hpcc-1.5.0 라는 directory 내용을 scan하여, 혹시 virus에 감염된 파일이 있을 경우 경고(bell)를 울려주는 명령은 다음과 같이 하시면 됩니다.
u0017649@sys-89983:~$ clamscan -r --bell -i /home/u0017649/hpcc-1.5.0
----------- SCAN SUMMARY -----------
Known viruses: 6366898
Engine version: 0.99.2
Scanned directories: 34
Scanned files: 737
Infected files: 0
Data scanned: 9.64 MB
Data read: 6.11 MB (ratio 1.58:1)
Time: 18.255 sec (0 m 18 s)
만약 virus에 감염된 파일이 있을 경우 자동으로 제거까지 하기를 원한다면 다음과 같이 --remove 옵션을 사용하시면 됩니다. 다만, ppc64le 아키텍처에서 virus 감염 파일을 구하는 것은 정말 어려울 것이므로, 위에 언급된 clamav 홈페이지에서 clamav source code를 download 받아서 그 source를 scan해보겠습니다. 그 속에는 test용으로 들어있는 파일들이 있는 모양이더라구요.
u0017649@sys-89983:~$ tar -zxf clamav-0.99.2.tar.gz
u0017649@sys-89983:~$ clamscan -r --remove /home/u0017649/clamav-0.99.2 > clamscan.out
다음과 같이 3개 파일이 감염되었다고 제거된 것을 보실 수 있습니다.
u0017649@sys-89983:~$ grep -i removed clamscan.out
/home/u0017649/clamav-0.99.2/test/.split/split.clam_IScab_int.exeaa: Removed.
/home/u0017649/clamav-0.99.2/test/.split/split.clam.isoaa: Removed.
/home/u0017649/clamav-0.99.2/test/.split/split.clam_IScab_ext.exeaa: Removed.
/home/u0017649/clamav-0.99.2/test/.split/split.clamjol.isoaa: Removed.
u0017649@sys-89983:~$ tail clamscan.out
----------- SCAN SUMMARY -----------
Known viruses: 6366898
Engine version: 0.99.2
Scanned directories: 227
Scanned files: 3231
Infected files: 4
Data scanned: 93.26 MB
Data read: 50.67 MB (ratio 1.84:1)
Time: 28.488 sec (0 m 28 s)
Anti-virus SW는 virus 목록 등이 계속 업데이트 되는 것이 중요하지요. 그런 일을 해주는 것이 freshclam 입니다. 이건 설치되면 자동으로 수행되는데, 그 log는 다음과 같이 확인하실 수 있습니다.
u0017649@sys-89983:~$ sudo tail -f /var/log/clamav/freshclam.log
Sun Dec 17 20:57:04 2017 -> ClamAV update process started at Sun Dec 17 20:57:04 2017
Sun Dec 17 20:58:02 2017 -> Downloading main.cvd [100%]
Sun Dec 17 20:58:13 2017 -> main.cvd updated (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
Sun Dec 17 20:58:35 2017 -> Downloading daily.cvd [100%]
Sun Dec 17 20:58:39 2017 -> daily.cvd updated (version: 24138, sigs: 1806393, f-level: 63, builder: neo)
Sun Dec 17 20:58:40 2017 -> Downloading bytecode.cvd [100%]
Sun Dec 17 20:58:40 2017 -> bytecode.cvd updated (version: 319, sigs: 75, f-level: 63, builder: neo)
Sun Dec 17 20:58:44 2017 -> Database updated (6372717 signatures) from db.local.clamav.net (IP: 157.131.0.17)
Sun Dec 17 20:58:44 2017 -> WARNING: Clamd was NOT notified: Can't connect to clamd through /var/run/clamav/clamd.ctl: No such file or directory
Sun Dec 17 20:58:44 2017 -> --------------------------------------
위의 log를 보면 clamd.ctl 파일이 없어서 clamd에 대한 notification이 제대로 되지 않은 것을 보실 수 있습니다. 저 file은 clamav-daemon을 처음 살릴 때 자동 생성되는데, 제가 위에서 'systemctl start clamav-daemon.service' 명령으로 clamav-daemon을 살리기 전에 freshclam이 구동되는 바람에 벌어진 일 같습니다. 이제 clamav-daemon을 제가 살려 놓았으므로, 다음과 같이 freshclam을 죽였다가 살리면 해결됩니다.
u0017649@sys-89983:~$ ps -ef | grep freshclam
clamav 8894 1 1 20:57 ? 00:00:10 /usr/bin/freshclam -d --foreground=true
u0017649 9473 31958 0 21:08 pts/0 00:00:00 grep --color=auto freshclam
u0017649@sys-89983:~$ sudo kill -9 8894
u0017649@sys-89983:~$ sudo /usr/bin/freshclam -d --foreground=false
위에서는 freshcalm을 background daemon으로 살렸습니다. 다시 log를 보시지요.
u0017649@sys-89983:~$ sudo tail -f /var/log/clamav/freshclam.log
Sun Dec 17 20:58:44 2017 -> Database updated (6372717 signatures) from db.local.clamav.net (IP: 157.131.0.17)
Sun Dec 17 20:58:44 2017 -> WARNING: Clamd was NOT notified: Can't connect to clamd through /var/run/clamav/clamd.ctl: No such file or directory
Sun Dec 17 20:58:44 2017 -> --------------------------------------
Sun Dec 17 21:09:37 2017 -> --------------------------------------
Sun Dec 17 21:09:37 2017 -> freshclam daemon 0.99.2 (OS: linux-gnu, ARCH: ppc, CPU: powerpc64le)
Sun Dec 17 21:09:37 2017 -> ClamAV update process started at Sun Dec 17 21:09:37 2017
Sun Dec 17 21:09:37 2017 -> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
Sun Dec 17 21:09:37 2017 -> daily.cvd is up to date (version: 24138, sigs: 1806393, f-level: 63, builder: neo)
Sun Dec 17 21:09:37 2017 -> bytecode.cvd is up to date (version: 319, sigs: 75, f-level: 63, builder: neo)
Sun Dec 17 21:09:37 2017 -> --------------------------------------
이제 error 없이 잘 update된 것을 보실 수 있습니다.
clamconf 명령은 clamav 관련 각종 config 파일을 점검해주는 명령입니다. 그 output은 아래와 같습니다.
u0017649@sys-89983:~$ clamconf
Checking configuration files in /etc/clamav
Config file: clamd.conf
-----------------------
LogFile = "/var/log/clamav/clamav.log"
StatsHostID = "auto"
StatsEnabled disabled
StatsPEDisabled = "yes"
StatsTimeout = "10"
LogFileUnlock disabled
LogFileMaxSize = "4294967295"
LogTime = "yes"
LogClean disabled
LogSyslog disabled
LogFacility = "LOG_LOCAL6"
LogVerbose disabled
LogRotate = "yes"
ExtendedDetectionInfo = "yes"
PidFile disabled
TemporaryDirectory disabled
DatabaseDirectory = "/var/lib/clamav"
OfficialDatabaseOnly disabled
LocalSocket = "/var/run/clamav/clamd.ctl"
LocalSocketGroup = "clamav"
LocalSocketMode = "666"
FixStaleSocket = "yes"
TCPSocket disabled
TCPAddr disabled
MaxConnectionQueueLength = "15"
StreamMaxLength = "26214400"
StreamMinPort = "1024"
StreamMaxPort = "2048"
MaxThreads = "12"
ReadTimeout = "180"
CommandReadTimeout = "5"
SendBufTimeout = "200"
MaxQueue = "100"
IdleTimeout = "30"
ExcludePath disabled
MaxDirectoryRecursion = "15"
FollowDirectorySymlinks disabled
FollowFileSymlinks disabled
CrossFilesystems = "yes"
SelfCheck = "3600"
DisableCache disabled
VirusEvent disabled
ExitOnOOM disabled
AllowAllMatchScan = "yes"
Foreground disabled
Debug disabled
LeaveTemporaryFiles disabled
User = "clamav"
AllowSupplementaryGroups disabled
Bytecode = "yes"
BytecodeSecurity = "TrustSigned"
BytecodeTimeout = "60000"
BytecodeUnsigned disabled
BytecodeMode = "Auto"
DetectPUA disabled
ExcludePUA disabled
IncludePUA disabled
AlgorithmicDetection = "yes"
ScanPE = "yes"
ScanELF = "yes"
DetectBrokenExecutables disabled
ScanMail = "yes"
ScanPartialMessages disabled
PhishingSignatures = "yes"
PhishingScanURLs = "yes"
PhishingAlwaysBlockCloak disabled
PhishingAlwaysBlockSSLMismatch disabled
PartitionIntersection disabled
HeuristicScanPrecedence disabled
StructuredDataDetection disabled
StructuredMinCreditCardCount = "3"
StructuredMinSSNCount = "3"
StructuredSSNFormatNormal = "yes"
StructuredSSNFormatStripped disabled
ScanHTML = "yes"
ScanOLE2 = "yes"
OLE2BlockMacros disabled
ScanPDF = "yes"
ScanSWF = "yes"
ScanXMLDOCS = "yes"
ScanHWP3 = "yes"
ScanArchive = "yes"
ArchiveBlockEncrypted disabled
ForceToDisk disabled
MaxScanSize = "104857600"
MaxFileSize = "26214400"
MaxRecursion = "16"
MaxFiles = "10000"
MaxEmbeddedPE = "10485760"
MaxHTMLNormalize = "10485760"
MaxHTMLNoTags = "2097152"
MaxScriptNormalize = "5242880"
MaxZipTypeRcg = "1048576"
MaxPartitions = "50"
MaxIconsPE = "100"
MaxRecHWP3 = "16"
PCREMatchLimit = "10000"
PCRERecMatchLimit = "5000"
PCREMaxFileSize = "26214400"
ScanOnAccess disabled
OnAccessMountPath disabled
OnAccessIncludePath disabled
OnAccessExcludePath disabled
OnAccessExcludeUID disabled
OnAccessMaxFileSize = "5242880"
OnAccessDisableDDD disabled
OnAccessPrevention disabled
OnAccessExtraScanning disabled
DevACOnly disabled
DevACDepth disabled
DevPerformance disabled
DevLiblog disabled
DisableCertCheck disabled
Config file: freshclam.conf
---------------------------
StatsHostID disabled
StatsEnabled disabled
StatsTimeout disabled
LogFileMaxSize = "4294967295"
LogTime = "yes"
LogSyslog disabled
LogFacility = "LOG_LOCAL6"
LogVerbose disabled
LogRotate = "yes"
PidFile disabled
DatabaseDirectory = "/var/lib/clamav"
Foreground disabled
Debug disabled
AllowSupplementaryGroups disabled
UpdateLogFile = "/var/log/clamav/freshclam.log"
DatabaseOwner = "clamav"
Checks = "24"
DNSDatabaseInfo = "current.cvd.clamav.net"
DatabaseMirror = "db.local.clamav.net", "database.clamav.net"
PrivateMirror disabled
MaxAttempts = "5"
ScriptedUpdates = "yes"
TestDatabases = "yes"
CompressLocalDatabase disabled
ExtraDatabase disabled
DatabaseCustomURL disabled
HTTPProxyServer disabled
HTTPProxyPort disabled
HTTPProxyUsername disabled
HTTPProxyPassword disabled
HTTPUserAgent disabled
NotifyClamd = "/etc/clamav/clamd.conf"
OnUpdateExecute disabled
OnErrorExecute disabled
OnOutdatedExecute disabled
LocalIPAddress disabled
ConnectTimeout = "30"
ReceiveTimeout = "30"
SubmitDetectionStats disabled
DetectionStatsCountry disabled
DetectionStatsHostID disabled
SafeBrowsing disabled
Bytecode = "yes"
clamav-milter.conf not found
Software settings
-----------------
Version: 0.99.2
Optional features supported: MEMPOOL IPv6 FRESHCLAM_DNS_FIX AUTOIT_EA06 BZIP2 LIBXML2 PCRE ICONV JSON
Database information
--------------------
Database directory: /var/lib/clamav
bytecode.cvd: version 319, sigs: 75, built on Wed Dec 6 21:17:11 2017
main.cvd: version 58, sigs: 4566249, built on Wed Jun 7 17:38:10 2017
daily.cvd: version 24138, sigs: 1806393, built on Sun Dec 17 16:10:39 2017
Total number of signatures: 6372717
Platform information
--------------------
uname: Linux 4.4.0-103-generic #126-Ubuntu SMP Mon Dec 4 16:22:09 UTC 2017 ppc64le
OS: linux-gnu, ARCH: ppc, CPU: powerpc64le
Full OS version: Ubuntu 16.04.2 LTS
zlib version: 1.2.8 (1.2.8), compile flags: a9
platform id: 0x0a3152520800000000050400
Build information
-----------------
GNU C: 5.4.0 20160609 (5.4.0)
CPPFLAGS: -Wdate-time -D_FORTIFY_SOURCE=2
CFLAGS: -g -O3 -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64 -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE
CXXFLAGS:
LDFLAGS: -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed
Configure: '--build=powerpc64le-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libexecdir=/usr/lib/clamav' '--disable-maintainer-mode' '--disable-dependency-tracking' 'CFLAGS=-g -O3 -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O3 -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed' '--with-dbdir=/var/lib/clamav' '--sysconfdir=/etc/clamav' '--disable-clamav' '--disable-unrar' '--enable-milter' '--enable-dns-fix' '--with-libjson' '--with-gnu-ld' '--with-systemdsystemunitdir=/lib/systemd/system' 'build_alias=powerpc64le-linux-gnu'
sizeof(void*) = 8
Engine flevel: 82, dconf: 82
2017년 12월 18일 월요일
2017년 10월 16일 월요일
Minsky 서버에서 Tensorflow r1.3을 source로부터 build하기
2017년 10월 현재 최신 PowerAI 버전은 4.0으로서, 이 속에 포함된 tensorflow는 버전 r1.1입니다.
root@ubuntu:/tmp/kkk# dpkg -l | grep mldl
ii mldl-repo-local 4.0.0 ppc64el IBM repository for Deep Learning tools for POWER linux
ii power-mldl 4.0.0 ppc64el Meta-package for Deep Learning frameworks for POWER
root@ubuntu:/tmp/kkk# dpkg -l | grep tensor
ii ddl-tensorflow 0.9.0-4ibm1 ppc64el TensorFlow operator for IBM PowerAI Distributed Deep Learning
ii tensorflow 1.1.0-4ibm1 ppc64el Open source software library for numerical computation
현재 github에 올라온 tensorflow 최신 버전은 r1.4이며, 몇몇 최신 버전을 사용하시는 고객께서는 r1.3을 테스트하시고자 합니다. PowerAI는 대략 3개월 단위로 최신 ML/DL framework 버전을 갱신하니까, 다음 버전의 PowerAI는 아마도 11월 경에 나올 가능성이 많습니다. 그렇다면 tensorflow나 기타 framework의 최신 버전을 쓰려면 그때까지 참고 기다려야 할까요 ?
일단 기다려주십사 하는 것이 IBM의 공식 입장입니다만, 사실 꼭 그러실 필요는 없습니다. 오픈 소스니까요. 그냥 tensorflow r1.3을 직접 소스로부터 빌드하셔서 사용하실 수도 있습니다. 여기서는 그 방법을 살펴보시고, r1.3이 포함된 site-packages.tar도 올려놓겠습니다.
먼저, 여기서 제가 빌드를 위해 사용한 환경은 Ubuntu 16.04.2 LTS ppc64le입니다. CUDA는 8.0.61, libcudnn은 6.0.21, 그리고 gcc는 5.4 입니다.
u0017649@sys-89490:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
u0017649@sys-89490:~$ dpkg -l | grep cuda
ii cuda 8.0.61-1 ppc64el CUDA meta-package
ii cuda-8-0 8.0.61-1 ppc64el CUDA 8.0 meta-package
...
ii libcudnn6 6.0.21-1+cuda8.0 ppc64el cuDNN runtime libraries
ii libcudnn6-dev 6.0.21-1+cuda8.0 ppc64el cuDNN development libraries and headers
u0017649@sys-89490:~$ /usr/bin/gcc --version
gcc (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
여기서는 아래의 tensorflow 공식 홈페이지에 나온 source로부터의 빌드 방법을 거의 그대로 따라가겠습니다.
https://www.tensorflow.org/install/install_sources#ConfigureInstallation
Tensorflow 홈페이지에서는 아래의 libcupti-dev 외에 python 관련 패키지를 잔뜩 설치해야 하라고 나오는데, 실제로는 그냥 anaconda만 설치하고 진행해도 괜찮은 것 같습니다.
u0017649@sys-89490:~$ sudo apt-get install libcupti-dev
아래는 ananconda3 설치 절차입니다.
u0017649@sys-89490:~$ wget wget https://repo.continuum.io/archive/Anaconda3-4.4.0.1-Linux-ppc64le.sh
u0017649@sys-89490:~$ chmod u+x Anaconda*
u0017649@sys-89490:~$ sudo ./Anaconda3-4.4.0.1-Linux-ppc64le.sh
...
[/home/u0017649/anaconda3] >>> /usr/local/anaconda3
u0017649@sys-89490:~$ export PATH="/usr/local/anaconda3/bin:$PATH"
여기서는 주로 u0017649 userid를 사용하므로, 그 userid로 anaconda root directory(/usr/local/anaconda3)를 자유롭게 쓸 수 있도록 그 directory의 ownership을 바꿔줍니다.
u0017649@sys-89490:~$ sudo chown -R u0017649:u0017649 /usr/local/anaconda3
이제 conda 명령을 이용해서 bazel과 numpy를 설치합니다.
u0017649@sys-89490:~$ conda install bazel numpy
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /usr/local/anaconda3:
The following NEW packages will be INSTALLED:
bazel: 0.4.5-0
The following packages will be UPDATED:
anaconda: 4.4.0-np112py36_0 --> 4.4.0-np112py36_1
conda: 4.3.21-py36_0 --> 4.3.27-py36_0
Proceed ([y]/n)? y
bazel-0.4.5-0. 100% |####################################################| Time: 0:00:09 14.58 MB/s
conda-4.3.27-p 100% |####################################################| Time: 0:00:00 10.31 MB/s
anaconda-4.4.0 100% |####################################################| Time: 0:00:00 6.57 MB/s
u0017649@sys-89490:~$ which bazel
/usr/local/anaconda3/bin/bazel
이제 tensorflow의 source를 git clone 명령으로 download 받습니다.
u0017649@sys-89490:~$ git clone --recursive https://github.com/tensorflow/tensorflow.git
Cloning into 'tensorflow'...
remote: Counting objects: 246390, done.
remote: Total 246390 (delta 0), reused 0 (delta 0), pack-reused 246390
Receiving objects: 100% (246390/246390), 125.14 MiB | 6.34 MiB/s, done.
Resolving deltas: 100% (192026/192026), done.
Checking connectivity... done.
우리가 빌드하려는 것은 r1.3으로 그 branch로 switch합니다.
u0017649@sys-89490:~$ cd tensorflow
u0017649@sys-89490:~/tensorflow$ git checkout r1.3
Checking out files: 100% (3934/3934), done.
Branch r1.3 set up to track remote branch r1.3 from origin.
Switched to a new branch 'r1.3'
이제 build 준비가 끝났습니다. Build 절차는 기본적으로, configure --> bazel로 pip package build --> wheel file build --> pip 명령으로 wheel file 설치입니다.
Configure하실 때 대부분은 default를 선택하시되, 몇몇 부분에서는 별도로 지정하셔야 합니다. ppc64le 특성에 맞춰야 하는 부분은 딱 한군데, cuDNN 6 library가 위치한 directory를 지정하는 부분 뿐입니다.
u0017649@sys-89490:~/tensorflow$ ./configure
Extracting Bazel installation...
..................
You have bazel 0.4.5- installed.
Please specify the location of python. [Default is /usr/local/anaconda3/bin/python]:
Found possible Python library paths:
/usr/local/anaconda3/lib/python3.6/site-packages
Please input the desired Python library path to use. Default is [/usr/local/anaconda3/lib/python3.6/site-packages]
Do you wish to build TensorFlow with MKL support? [y/N] n
...
Do you wish to build TensorFlow with CUDA support? [y/N] y
...
Please specify which gcc should be used by nvcc as the host compiler. [Default is /opt/at10.0/bin/gcc]: /usr/bin/gcc
...
Please specify the location where cuDNN 6 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /usr/lib/powerpc64le-linux-gnu
...
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]: "3.7,6.0" # 3.7은 K80, 6.0은 P100을 위한 compute capability입니다.
...
Configure가 끝나면 bazel로 TensorFlow with GPU support를 위한 pip package를 build합니다. 이 과정 중에, 아래 예시한 것과 같이 인터넷으로부터 일부 tar.gz 파일들을 download 받는 부분이 관측됩니다. 즉, 이 과정을 위해서는 인터넷 연결이 필요합니다.
u0017649@sys-89490:~/tensorflow$ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
...
INFO: Downloading http://mirror.bazel.build/github.com/google/boringssl/archive/bbcaa15b0647816b9a\
1a9b9e0d209cd6712f0105.tar.gz: 3,244,032 bytes
...
INFO: Downloading http://mirror.bazel.build/github.com/google/boringssl/archive/bbcaa15b0647816b9a\
1a9b9e0d209cd6712f0105.tar.gz: 4,032,059 bytes
...
INFO: From Compiling tensorflow/python/pywrap_tensorflow_internal.cc:
bazel-out/local_linux-py3-opt/bin/tensorflow/python/pywrap_tensorflow_internal.cc: In function 'PyObject* _wrap_PyRecordReader_New(PyObject*, PyObject*)':
bazel-out/local_linux-py3-opt/bin/tensorflow/python/pywrap_tensorflow_internal.cc:5173:138: warning: 'arg2' may be used uninitialized in this function [-Wmaybe-uninitialized]
result = (tensorflow::io::PyRecordReader *)tensorflow::io::PyRecordReader::New((string const &)*arg1,arg2,(string const &)*arg3,arg4);
^
At global scope:
cc1plus: warning: unrecognized command line option '-Wno-self-assign'
Target //tensorflow/tools/pip_package:build_pip_package up-to-date:
bazel-bin/tensorflow/tools/pip_package/build_pip_package
INFO: Elapsed time: 7106.091s, Critical Path: 4427.34s
이런저런 warning message만 나올 뿐, 잘 끝났습니다. 저는 이 과정을 1-core짜리 cloud 환경에서 돌렸는데, 이렇게 거의 2시간 가까이 걸렸습니다.
이제 여기서 build된 binary를 이용하여 .whl file을 build합니다. 생성될 .whl 파일은 /tmp/tensorflow_pkg directory에 들어가도록 지정합니다.
u0017649@sys-89490:~/tensorflow$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
Fri Oct 13 03:40:03 EDT 2017 : === Using tmpdir: /tmp/tmp.UreS6KLhqt
~/tensorflow/bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles ~/tensorflow
~/tensorflow
/tmp/tmp.UreS6KLhqt ~/tensorflow
Fri Oct 13 03:40:23 EDT 2017 : === Building wheel
warning: no files found matching '*.dll' under directory '*'
warning: no files found matching '*.lib' under directory '*'
warning: no files found matching '*.h' under directory 'tensorflow/include/tensorflow'
warning: no files found matching '*' under directory 'tensorflow/include/Eigen'
warning: no files found matching '*' under directory 'tensorflow/include/external'
warning: no files found matching '*.h' under directory 'tensorflow/include/google'
warning: no files found matching '*' under directory 'tensorflow/include/third_party'
warning: no files found matching '*' under directory 'tensorflow/include/unsupported'
~/tensorflow
Fri Oct 13 03:44:36 EDT 2017 : === Output wheel file is in: /tmp/tensorflow_pkg
u0017649@sys-89490:~$ ls /tmp/tensorflow_pkg
tensorflow-1.3.1-cp36-cp36m-linux_ppc64le.whl
잘 끝났습니다. 이제 이 wheel 파일을 pip 명령으로 설치합니다. 단, 그 전에 PYTHONPATH 환경변수를 확실히 지정할 것을 권고합니다. 여기서도 tensorflow 설치에 필요한 tensorboard나 protobuf 등을 인터넷으로부터 자동으로 download 받는 것 같습니다.
u0017649@sys-89490:~$ export PYTHONPATH=/usr/local/anaconda3/lib/python3.6/site-packages
u0017649@sys-89490:~$ pip install /tmp/tensorflow_pkg/tensorflow-1.3.1-cp36-cp36m-linux_ppc64le.whl
Processing /tmp/tensorflow_pkg/tensorflow-1.3.1-cp36-cp36m-linux_ppc64le.whl
Requirement already satisfied: six>=1.10.0 in /usr/local/anaconda3/lib/python3.6/site-packages (from tensorflow==1.3.1)
Collecting tensorflow-tensorboard<0.2.0,>=0.1.0 (from tensorflow==1.3.1)
Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.python.org', port=443): Read timed out. (read timeout=15)",)': /simple/tensorflow-tensorboard/
Downloading tensorflow_tensorboard-0.1.8-py3-none-any.whl (1.6MB)
100% |????????????????????????????????| 1.6MB 772kB/s
Requirement already satisfied: wheel>=0.26 in /usr/local/anaconda3/lib/python3.6/site-packages (from tensorflow==1.3.1)
Requirement already satisfied: numpy>=1.11.0 in /usr/local/anaconda3/lib/python3.6/site-packages (from tensorflow==1.3.1)
Collecting protobuf>=3.3.0 (from tensorflow==1.3.1)
Downloading protobuf-3.4.0-py2.py3-none-any.whl (375kB)
100% |????????????????????????????????| 378kB 2.5MB/s
Requirement already satisfied: werkzeug>=0.11.10 in /usr/local/anaconda3/lib/python3.6/site-packages (from tensorflow-tensorboard<0.2.0,>=0.1.0->tensorflow==1.3.1)
...
Successfully installed html5lib-0.9999999 markdown-2.6.9 protobuf-3.4.0 tensorflow-1.3.1 tensorflow-tensorboard-0.1.8
자, 이제 설치가 끝났습니다. conda list 명령으로 tensorflow의 설치를 확인합니다.
u0017649@sys-89490:~$ conda list | grep tensor
tensorflow 1.3.1 <pip>
tensorflow-tensorboard 0.1.8 <pip>
이제 tensorflow가 설치된 PYTHONPATH, 즉 /usr/local/anaconda3/lib/python3.6/site-packages를 통째로 tar로 말아서 K80 GPU가 설치된 제2의 서버에 가져가겠습니다. 물론 그 제2의 서버에는 미리 anaconda3가 설치되어 있어야 합니다.
u0017649@sys-89498:/usr/local/anaconda3/lib/python3.6 $ tar -zcf site-packages.tgz site-packages
u0017649@sys-89498:~$ scp /usr/local/anaconda3/lib/python3.6/site-packages.tgz firestone:~/
site-packages.tgz 100% 313MB 19.6MB/s 00:16
이제 위에서 firestone이라고 되어 있는 그 서버로 이동해서 작업합니다. 참고로, 이 firestone이라는 서버는 Ubuntu 16.04.3으로서 빌드 서버의 16.04.2보다 약간 버전이 높습니다만, 별 문제 없이 잘 수행되더군요.
먼저 anaconda3가 설치되어 있는지 확인합니다.
root@ubuntu:~# which conda
/root/anaconda3/bin/conda
여기서는 root user로 /root/anaconda3 위치에 anaconda를 설치해놓았네요. 뭐 별로 좋은 위치는 아닙니다만, 여기서는 이걸 그대로 써보겠습니다. 여기서의 PYTHONPATH, 즉, /root/anaconda3/lib/python3.6/site-packages에 저 위의 서버에서 가져온 site-packages.tgz를 그대로 풀어놓겠습니다. 기존의 site-packages 속에 들어있는 package들은 overwrite 됩니다만, 만약 새로 풀리는 site-packages 밑에 없는 패키지들이 있었다면 그것들은 그대로 보존되니까, 그냥 풀어놓으셔도 됩니다.
root@ubuntu:~# cd /root/anaconda3/lib/python3.6
root@ubuntu:~/anaconda3/lib/python3.6# tar -zxf ~/site-packages.tgz
이제 준비 완료입니다. 먼저 conda list로 tensorflow가 설치되어 있는 것으로 나오는지 확인해봅니다.
root@ubuntu:~/anaconda3/lib/python3.6# conda list | grep tensorflow
tensorflow 1.3.1 <pip>
tensorflow-tensorboard 0.1.8 <pip>
예, 잘 보이네요. 이제 PYTHONPATH를 다시 한번 확실히 선언하고, 시험 삼아 cifar10을 수행해보겠습니다.
root@ubuntu:~# export PYTHONPATH=/root/anaconda3/lib/python3.6/site-packages
root@ubuntu:~# git clone https://github.com/tensorflow/models.git
Cloning into 'models'...
remote: Counting objects: 7644, done.
remote: Total 7644 (delta 0), reused 0 (delta 0), pack-reused 7644
Receiving objects: 100% (7644/7644), 157.90 MiB | 19.09 MiB/s, done.
Resolving deltas: 100% (4139/4139), done.
Checking connectivity... done.
root@ubuntu:~# cd models/tutorials/image/cifar10
root@ubuntu:~/models/tutorials/image/cifar10# ls
BUILD cifar10_input_test.py cifar10_train.py
cifar10_eval.py cifar10_multi_gpu_train.py __init__.py
cifar10_input.py cifar10.py README.md
root@ubuntu:~/models/tutorials/image/cifar10# which python
/root/anaconda3/bin/python
Download 받은 example script 중 cifar10_train.py을 수행하면 training dataset의 download부터 training까지 일사천리로 수행됩니다.
root@ubuntu:~/models/tutorials/image/cifar10# python cifar10_train.py
>> Downloading cifar-10-binary.tar.gz 100.0%
Successfully downloaded cifar-10-binary.tar.gz 170052171 bytes.
/root/anaconda3/lib/python3.6/site-packages/numpy/core/machar.py:127: RuntimeWarning: overflow encountered in add
a = a + a
/root/anaconda3/lib/python3.6/site-packages/numpy/core/machar.py:129: RuntimeWarning: invalid value encountered in subtract
temp1 = temp - a
/root/anaconda3/lib/python3.6/site-packages/numpy/core/machar.py:138: RuntimeWarning: invalid value encountered in subtract
itemp = int_conv(temp-a)
/root/anaconda3/lib/python3.6/site-packages/numpy/core/machar.py:162: RuntimeWarning: overflow encountered in add
a = a + a
/root/anaconda3/lib/python3.6/site-packages/numpy/core/machar.py:164: RuntimeWarning: invalid value encountered in subtract
temp1 = temp - a
/root/anaconda3/lib/python3.6/site-packages/numpy/core/machar.py:171: RuntimeWarning: invalid value encountered in subtract
if any(temp-a != zero):
Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.
...
2017-10-13 22:18:12.974909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 2: N N Y Y
2017-10-13 22:18:12.974915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 3: N N Y Y
2017-10-13 22:18:12.974933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:03:00.0)
2017-10-13 22:18:12.974942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K80, pci bus id: 0000:04:00.0)
2017-10-13 22:18:12.974952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:2) -> (device: 2, name: Tesla K80, pci bus id: 0020:03:00.0)
2017-10-13 22:18:12.974960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:3) -> (device: 3, name: Tesla K80, pci bus id: 0020:04:00.0)
2017-10-13 22:18:22.502001: step 0, loss = 4.67 (52.2 examples/sec; 2.451 sec/batch)
2017-10-13 22:18:22.917980: step 10, loss = 4.62 (3076.9 examples/sec; 0.042 sec/batch)
2017-10-13 22:18:23.240470: step 20, loss = 4.43 (3969.1 examples/sec; 0.032 sec/batch)
2017-10-13 22:18:23.538250: step 30, loss = 4.41 (4298.5 examples/sec; 0.030 sec/batch)
2017-10-13 22:18:23.837076: step 40, loss = 4.29 (4283.4 examples/sec; 0.030 sec/batch)
2017-10-13 22:18:24.134108: step 50, loss = 4.33 (4309.3 examples/sec; 0.030 sec/batch)
2017-10-13 22:18:24.426799: step 60, loss = 4.30 (4373.2 examples/sec; 0.029 sec/batch)
...
2017-10-14 07:09:50.128824: step 999920, loss = 0.10 (4352.2 examples/sec; 0.029 sec/batch)
2017-10-14 07:09:50.428406: step 999930, loss = 0.11 (4272.7 examples/sec; 0.030 sec/batch)
2017-10-14 07:09:50.722993: step 999940, loss = 0.14 (4345.0 examples/sec; 0.029 sec/batch)
2017-10-14 07:09:51.019963: step 999950, loss = 0.11 (4310.2 examples/sec; 0.030 sec/batch)
2017-10-14 07:09:51.317448: step 999960, loss = 0.13 (4302.7 examples/sec; 0.030 sec/batch)
2017-10-14 07:09:51.617180: step 999970, loss = 0.11 (4270.7 examples/sec; 0.030 sec/batch)
2017-10-14 07:09:51.914633: step 999980, loss = 0.14 (4303.0 examples/sec; 0.030 sec/batch)
2017-10-14 07:09:52.213837: step 999990, loss = 0.13 (4278.1 examples/sec; 0.030 sec/batch)
완벽하게 잘 수행되었습니다.
아래의 제 구글 드라이브에 site-packages.tgz를 올려두었습니다. 사이즈는 330MB 정도인 이 파일은 ppc64le 아키텍처의 Ubuntu 16.04, CUDA 8.0, libcuDNN 6.0 환경의 K80 또는 P100을 활용하실 수 있도록 빌드된 것입니다. 필요하시면 마음대로 가져가 쓰셔도 됩니다. 제가 책임은 질 수 없는 패키지라는 것은 양해 바랍니다.
https://drive.google.com/open?id=0B-F0jEb44gqUZGdYb3A2QUNaenM
2017년 9월 15일 금요일
PowerAI 4.0의 DDL을 이용한 caffe와 tensorflow의 병렬처리
PowerAI 4.0에 포함된 DDL(Distributed Deep Learning)의 구체적인 사용법에 대해서 보시겠습니다.
일단 caffe는 IBM 버전 caffe (caffe-ibm)에 DDL 옵션이 통합되어 있으므로 별도 debian 패키지를 설치할 필요가 없습니다. 이 caffe-ibm도 내부적으로는 OpenMPI를 이용하는 것이므로 관련 library들이 설치되기는 해야 합니다만, 이는 caffe-ibm을 설치할 때 함께 자동으로 설치되므로 따로 신경쓰지 않으셔도 됩니다.
nimbix@JARVICENAE-0A0A1835:/data/mnist$ dpkg -l | grep openmpi
ii libopenmpi2-cuda:ppc64el 2.0.1-4ibm1 ppc64el high performance message passing library -- shared library
ii openmpi-bin-cuda 2.0.1-4ibm1 ppc64el high performance message passing library -- binaries
ii openmpi-common-cuda 2.0.1-4ibm1 all high performance message passing library -- common files
ii openmpi-doc-cuda 2.0.1-4ibm1 all high performance message passing library -- man pages
가령 위에서 보는 것과 같이 CUDA-aware OpenMPI를 설치하고나면, mpirun이라는 MPI utility가 설치됩니다. 이 mpirun이라는 것은 여러단계의 soft link가 걸린 orterun이라는 명령어이고, 결국 아래와 같이 openmpi-bin-cuda에서 제공됩니다.
nimbix@JARVICENAE-0A0A1835:/data/mnist$ dpkg -S /usr/bin/orterun
openmpi-bin-cuda: /usr/bin/orterun
IBM 버전 caffe에서의 DDL 사용법은 알고 보면 단순합니다. 다음 4가지만 아시면 됩니다.
1) caffe 명령을 수행할 때 -ddl 옵션을 준다
2) Train/Validation용 dataset은 모든 서버에서 동일한 위치(directory)에 존재해야 한다 (병렬파일시스템 또는 NFS가 편리)
3) 모든 서버는 암호 없이 ssh가 되도록 ssh-keygen과 ssh-copy-id가 되어 있어야 한다
4) 환경변수 등을 다른 서버 노드에도 전달하기 위해서는 mpirun 명령을 사용하는 것이 편하다
다른 것은 다 쉽습니다만 1)번 항목이 조금 어렵게 느껴질 수도 있습니다. 복잡한 부분은 다 빼고, 그냥 쉽게 보면 이렇습니다.
DDL 옵션을 쓴다고 해서 caffe가 여러분이 가진 GPU서버 및 network 환경을 스스로 이해하고 그에 맞게 자동으로 최적화할 수는 없습니다. 따라서 그런 환경, 즉 topology를 caffe에게 여러분이 직접 알려주셔야 합니다. 그게 -ddl 옵션의 mode입니다. 쉽게 예를 들어 설명하면 다음과 같습니다.
$ mpirun -x PATH -x LD_LIBRARY_PATH -n 12 -rf 4x3.rf caffe train -solver /data/mnist/lenet_solver.prototxt -gpu 0 -bvlc -ddl "-mode n:4x3x1 -dev_sync 1"
- mpirun은 여러대의 서버 노드에 동일한 명령을 동일한 환경변수 (-x 옵션)을 써서 수행해주는 병렬환경 명령어입니다.
- 4x3.rf라는 이름의 파일은 rank file입니다. 이 속에 병렬 서버 환경의 toplogy가 들어있습니다. 이걸 어떻게 만드는지는 아래에서 다루겠습니다.
- -n 12라는 것은 MPI client의 총 숫자이며, 쉽게 말해 training에 이용하려는 GPU의 갯수입니다.
- -gpu 0에서, 왜 12개가 아니라 gpu 0이라고 1개로 지정했는지 의아하실 수 있는데, MPI 환경에서는 각각의 GPU가 하나의 learner가 됩니다. 따라서 실제 물리적 서버 1대에 GPU가 몇 장 장착되어있든 상관없이 모두 -gpu 0, 즉 GPU는 1개로 지정한 것입니다.
- "-mode n:4x3x1"에서 n이라는 것은 NCCL (NVIDIA Collective Communications Library, 니클이라고 읽습니다)을 이용하라는 뜻입니다. 4x3x1은 4장의 GPU를 가진 서버 3대가 하나의 rack에 들어있다는 뜻입니다. 사실 어느 rack에 들어있느냐가 중요한 것은 아닌데, 보통 병렬수퍼컴 환경에서는 한대의 rack 안에 장착된 서버끼리는 좀더 고속의 low latency network으로 연결되어있기 때문에 이렇게 rack 표시까지 해주는 것입니다. 만약 4장의 GPU를 가진 서버가 6대씩 장착된 rack이 5대있다면 4x6x5로 표시됩니다.
- dev_sync에서 0은 GPU간 sync를 하지 말라는 것이고, 1은 통신 시작할 때 sync하라는 뜻, 2는 시작할 때와 끝낼 때 각각 sync하라는 뜻입니다.
잠깐, 데이터는 어디에 있는지 어떻게 지정하냐고요 ? 저 위에 지정된 solver 파일, 즉 lenet_solver.prototxt에 neural network이 지정되어 있고, 다시 그 neural network의 prototxt 파일 속에 데이터 위치가 지정되어 있습니다. 아래처럼요.
$ vi lenet_solver.prototxt
#net: "examples/mnist/lenet_train_test.prototxt"
net: "/data/mnist/lenet_train_test.prototxt"
$ vi lenet_train_test.prototxt
...
# source: "examples/mnist/mnist_train_lmdb"
source: "/data/mnist/mnist_train_lmdb"
...
# source: "examples/mnist/mnist_test_lmdb"
source: "/data/mnist/mnist_test_lmdb"
여러 서버 노드들의 GPU마다 수행될 learner들이 어떻게 data를 나누어 가져가느냐고요 ? 가급적이면 서버 노드마다 미리 파티셔닝되어 적절히 분배된 data들을 넣어두는 것이 좋습니다. Data를 N개의 learner들이 읽어갈 때, 각자 순차적으로 파일 이름들이 뒤섞여 들어간 목록으로부터 data를 읽어가는데, 만약 이 data가 물리적으로 미리 파티셔닝하여 노드 별로 분배해놓은 것이 아니라면 그냥 1번 training을 끝낼 때마다 전체 data를 N번 (N epochs) training한 것과 같은 효과를 냅니다. 저 data들의 저장소는 여러 노드에서 동시에 access할 수 있도록 IBM Spectrum Scale (구명칭 GPFS) 같은 병렬 파일시스템으로 하든가, 그게 없다면 성능이 떨어지더라도 NFS 같은 것으로 구성하는 것이 좋습니다.
이제 저 rf 파일, 즉 랭크 파일을 어떻게 만드는지 보시겠습니다. 그냥 손으로, 즉 vi 에디터 같은 것을 이용해서 만드셔도 됩니다만, PowerAI에서 기본 제공되는 rank_gen.py를 이용해서 다음과 같이 만드시는 것이 편합니다.
$ python /opt/DL/ddl/bin/rank_gen.py 4x2x3 sys-89074,sys-89075,sys-89076,sys-89077,sys-89078,sys-89079 > 4x2x3.rf
위에서 콤마(,)로 분리된 이름들이 서버 이름들입니다. 4x2x3이니 4장의 GPU를 가진 서버가 총 6대 있는 것이니, 서버 이름은 반드시 6대를 적으셔야 합니다. 이렇게 만들어진 4x2x3.rf 파일의 내용은 아래와 같습니다. rank_gen.py는 기본적으로 10-core POWER8 chip 2장을 장착한 Minsky 서버를 기준으로 만들기 때문에 아래와 같이 10개의 core를 가진 slot 2개가 있는 것으로 나옵니다. 그래서 rank, 즉 GPU 1개마다 slot이 5개 (0-4) 있는 것으로 나오는데, 만약 그게 아니라 8-core POWER8 chip이 장착된 서버라면 수작업으로 0-4가 아닌 0-3으로 수정해주셔야 합니다.
u0017649@sys-89075:~$ cat 4x2x3.rf
#2017-09-14 04:45:51 by rank_gen
#dims = 4x2x3
#host = sys-89074,sys-89075,sys-89076,sys-89077,sys-89078,sys-89079
#dimX = 4
#dimY = 2
#dimZ = 3
#sockets = 2
#cores = 10
rank 0=sys-89074 slot=0:0-4
rank 6=sys-89074 slot=0:5-9
rank 12=sys-89074 slot=1:0-4
rank 18=sys-89074 slot=1:5-9
rank 3=sys-89075 slot=0:0-4
rank 9=sys-89075 slot=0:5-9
rank 15=sys-89075 slot=1:0-4
rank 21=sys-89075 slot=1:5-9
rank 1=sys-89076 slot=0:0-4
rank 7=sys-89076 slot=0:5-9
rank 13=sys-89076 slot=1:0-4
rank 19=sys-89076 slot=1:5-9
rank 4=sys-89077 slot=0:0-4
rank 10=sys-89077 slot=0:5-9
rank 16=sys-89077 slot=1:0-4
rank 22=sys-89077 slot=1:5-9
rank 2=sys-89078 slot=0:0-4
rank 8=sys-89078 slot=0:5-9
rank 14=sys-89078 slot=1:0-4
rank 20=sys-89078 slot=1:5-9
rank 5=sys-89079 slot=0:0-4
rank 11=sys-89079 slot=0:5-9
rank 17=sys-89079 slot=1:0-4
rank 23=sys-89079 slot=1:5-9
Caffe는 그렇게 쉽게 됩니다만, tensorflow는 그보다 좀 어렵습니다. 일단 별도의 ddl-tensorflow라는 debian package가 PowerAI 4.0에 포함되어 있는데, 이는 사실 tensorflow DDL에 꼭 필요한 것이 아니라, tensorflow DDL을 좀더 쉽게 사용하실 수 있도록 해주는 Google Slim에 기반한 script들과 example 파일들을 제공해주는 것입니다. 정작 tensorflow는 별도로 설치하셔야 하는데, 물론 그건 PowerAI에서 제공되는 tensorflow를 apt-get install 명령으로 설치하시면 됩니다.
$ sudo apt-get install ddl-tensorflow tensorflow
$ dpkg -L ddl-tensorflow
/.
/opt
/opt/DL
/opt/DL/ddl-tensorflow
/opt/DL/ddl-tensorflow/examples
/opt/DL/ddl-tensorflow/examples/mnist
/opt/DL/ddl-tensorflow/examples/mnist/ddl_mnist.py
/opt/DL/ddl-tensorflow/examples/mnist/README.md
/opt/DL/ddl-tensorflow/examples/slim
/opt/DL/ddl-tensorflow/examples/slim/BUILD
/opt/DL/ddl-tensorflow/examples/slim/WORKSPACE
/opt/DL/ddl-tensorflow/examples/slim/scripts
/opt/DL/ddl-tensorflow/examples/slim/scripts/finetune_inception_resnet_v2_on_flowers.sh
/opt/DL/ddl-tensorflow/examples/slim/scripts/train_lenet_on_mnist.sh
/opt/DL/ddl-tensorflow/examples/slim/scripts/finetune_resnet_v1_50_on_flowers.sh
/opt/DL/ddl-tensorflow/examples/slim/scripts/finetune_inception_v3_on_flowers.sh
/opt/DL/ddl-tensorflow/examples/slim/scripts/train_cifarnet_on_cifar10.sh
/opt/DL/ddl-tensorflow/examples/slim/scripts/finetune_inception_v1_on_flowers.sh
/opt/DL/ddl-tensorflow/examples/slim/train-alexnet.sh
/opt/DL/ddl-tensorflow/examples/slim/deployment
/opt/DL/ddl-tensorflow/examples/slim/deployment/__init__.py
...
이 ddl-tensorflow를 사용하시기 위해서는 PYTHONPATH 등의 환경변수 설정을 위해 source 명령으로 아래와 같이 ddl-tensorflow-activate를 수행해주셔야 합니다.
$ source /opt/DL/ddl-tensorflow/bin/ddl-tensorflow-activate
이제 ddl-tensorflow-install-samples 명령을 사용하시어 지정하는 directory에 sample들을 설치하실 수 있습니다.
nimbix@JARVICENAE-0A0A1835:~$ ddl-tensorflow-install-samples /data
Write into existing directory /data? (yN)
y
Copying examples/ into /data...
Success
가장 간단한 것으로, 손글씨 숫자를 판독하는 MNIST가 들어 있습니다.
nimbix@JARVICENAE-0A0A1835:~$ cd /data/examples/mnist
nimbix@JARVICENAE-0A0A1835:/data/examples/mnist$ ls
ddl_mnist.py README.md
여기에 나온 것처럼, tensorflow는 명령어라기보다는 python에서 불러 사용하는 library로 되어 있기 때문에, 결국 multi-node 병렬처리를 하기 위해서는 python script를 위의 ddl_mnist.py에서처럼 작성해야 합니다.
일단 4-GPU 서버 2대(sys-89074와 sys-89075)로 수행하는 환경이라고 가정하고 아래와 같이 rank file을 먼저 만듭니다.
nimbix@JARVICENAE-0A0A1835:/data/examples/mnist$ python /opt/DL/ddl/bin/rank_gen.py 4x2x1 sys-89074,sys-89075 > 4x2.rf
nimbix@JARVICENAE-0A0A1835:/data/examples/mnist$ cat 4x2.rf
#2017-09-15 03:19:14 by rank_gen
#dims = 4x2x1
#host = sys-89074,sys-89075
#dimX = 4
#dimY = 2
#dimZ = 1
#sockets = 2
#cores = 10
rank 0=sys-89074 slot=0:0-4
rank 2=sys-89074 slot=0:5-9
rank 4=sys-89074 slot=1:0-4
rank 6=sys-89074 slot=1:5-9
rank 1=sys-89075 slot=0:0-4
rank 3=sys-89075 slot=0:5-9
rank 5=sys-89075 slot=1:0-4
rank 7=sys-89075 slot=1:5-9
이제 다음과 같이 수행하면 됩니다.
nimbix@JARVICENAE-0A0A1835:/data/examples/mnist$ mpirun -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -n 8 -rf 4x2.rf python ddl_mnist.py
(사실 mnist는 워낙 작은 dataset만 사용하므로, 병렬화의 의미가 없습니다. 그래서인지 ddl_mnist.py는 위에서 제가 예로 든 것처럼 4x2 구조는 애초에 불가능하고, 저 아래에 보시듯이 -mode r:2로 되어 있어 그냥 GPU 2장으로 병렬화하는 것만 가능합니다.)
결국 문제는 tensorflow를 병렬로 수행하기 위해서 python script를 어떻게 작성해야 하느냐인데, 이 부분에 대해서는 저도 개발자가 아닌 관계로 별 도움을 못 드리겠습니다. (사실 제겐 흰건 글씨요 검은건 공백이며, 깜빡이는 것은 커서 정도로만 보입니다.)
대신, 다소 깁니다만, 아래에 PowerAI에 포함된 ddl_mnist.py의 내용을 그대로 올려두겠습니다.
nimbix@JARVICENAE-0A0A1835:/data/examples/mnist$ vi ddl_mnist.py
import tensorflow as tf
import numpy as np
############################################################################
# IBM PowerAI Distributed Deep Learning (DDL) setup
############################################################################
# Disable GPU memory preallocation
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
############################################################################
# DDL Initialize BEGIN
############################################################################
# Load DDL operator
ddl = tf.load_op_library('/opt/DL/ddl-tensorflow/lib/ddl_MDR.so')
# DDL initializes MPI on CPU
# ddl.init takes two inputs
# 1) the number of GPUs to utilize on each host in training.
# this number is not the number of GPUs to use for each leaner. It simply tells DDL that there are X GPUs in each host to be used for training
# 2) DDL options (refer to README for details)
with tf.Session(config=config) as sess:
with tf.device('/cpu:0'):
rank, size, gpuid = sess.run(ddl.init(2, mode = '-mode r:2 -dump_iter 100'))
# MPI info and assigned GPU
print [rank, size, gpuid]
############################################################################
# DDL Initialize END
############################################################################
# Perform all TensorFlow computation within gpuid
with tf.device('/gpu:%d' %gpuid):
##############################################################################
# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
# Parameters
learning_rate = 0.001
training_iters = 200000
batch_size = 100
display_step = 1
# Network Parameters
n_input = 784 # MNIST data input (img shape: 28*28)
n_classes = 10 # MNIST total classes (0-9 digits)
dropout = 0.75 # Dropout, probability to keep units
# tf Graph input
x = tf.placeholder(tf.float32, [None, n_input])
y = tf.placeholder(tf.float32, [None, n_classes])
keep_prob = tf.placeholder(tf.float32) #dropout (keep probability)
# Create some wrappers for simplicity
def conv2d(x, W, b, strides=1):
# Conv2D wrapper, with bias and relu activation
x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding='SAME')
x = tf.nn.bias_add(x, b)
return tf.nn.relu(x)
def maxpool2d(x, k=2):
# MaxPool2D wrapper
return tf.nn.max_pool(x, ksize=[1, k, k, 1], strides=[1, k, k, 1],
padding='SAME')
# Create model
def conv_net(x, weights, biases, dropout):
# Reshape input picture
x = tf.reshape(x, shape=[-1, 28, 28, 1])
# Convolution Layer
conv1 = conv2d(x, weights['wc1'], biases['bc1'])
# Max Pooling (down-sampling)
conv1 = maxpool2d(conv1, k=2)
# Convolution Layer
conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])
# Max Pooling (down-sampling)
conv2 = maxpool2d(conv2, k=2)
# Fully connected layer
# Reshape conv2 output to fit fully connected layer input
fc1 = tf.reshape(conv2, [-1, weights['wd1'].get_shape().as_list()[0]])
fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])
fc1 = tf.nn.relu(fc1)
# Apply Dropout
fc1 = tf.nn.dropout(fc1, dropout)
# Output, class prediction
out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])
return out
# Store layers weight & bias
weights = {
############################################################################
# DDL BROADCAST BEGIN
############################################################################
# This step ensures that all learners start with the same initial parameters
# 5x5 conv, 1 input, 32 outputs
'wc1': tf.Variable(ddl.bcast(tf.random_normal([5, 5, 1, 32]))),
# 5x5 conv, 32 inputs, 64 outputs
'wc2': tf.Variable(ddl.bcast(tf.random_normal([5, 5, 32, 64]))),
# fully connected, 7*7*64 inputs, 1024 outputs
'wd1': tf.Variable(ddl.bcast(tf.random_normal([7*7*64, 1024]))),
# 1024 inputs, 10 outputs (class prediction)
'out': tf.Variable(ddl.bcast(tf.random_normal([1024, n_classes])))
############################################################################
# DDL BROADCAST END
############################################################################
}
biases = {
'bc1': tf.Variable(ddl.bcast(tf.random_normal([32]))),
'bc2': tf.Variable(ddl.bcast(tf.random_normal([64]))),
'bd1': tf.Variable(ddl.bcast(tf.random_normal([1024]))),
'out': tf.Variable(ddl.bcast(tf.random_normal([n_classes])))
}
# Construct model
pred = conv_net(x, weights, biases, keep_prob)
# Define loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
############################################################################
# DDL ALLREDUCE BEGIN
############################################################################
# Collect the gradients and the corresponding parameters w.r.t the given cost
grads_and_vars = optimizer.compute_gradients(cost)
# Separate out the tuple
grads, vars = zip(*grads_and_vars)
# This step takes the average of the gradients on all the learners
grads_and_vars_ddl = zip(ddl.all_reduce_n(grads, op='avg'), vars)
# Update the parameters with the averaged gradient
objective = optimizer.apply_gradients(grads_and_vars_ddl)
############################################################################
# DDL ALLREDUCE END
############################################################################
# Evaluate model
correct_pred = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
##############################################################################
def split(a, n):
k, m = divmod(len(a), n)
return (a[i * k + min(i, m):(i + 1) * k + min(i + 1, m)] for i in xrange(n))
# Launch the graph
with tf.Session(config=config) as sess:
sess.run(tf.global_variables_initializer())
step = 1
# Keep training until reach max iterations
while step * batch_size < training_iters:
# Each learner will read batch_size*size samples and
# use only the portion correspoding to the current learner (or rank)
batch_x, batch_y = mnist.train.next_batch(batch_size*size)
batch_x = np.split(batch_x,size)[rank]
batch_y = np.split(batch_y,size)[rank]
# Run optimization op (backprop)
sess.run(objective, feed_dict={x: batch_x, y: batch_y,
keep_prob: dropout})
if step % display_step == 0:
# Calculate batch loss and accuracy
loss, acc = sess.run([cost, accuracy], feed_dict={x: batch_x,
y: batch_y,
keep_prob: 1.})
print("MPI "+str(rank)+"] Iter " + str(step*batch_size) + ", Minibatch Loss= " + \
"{:.6f}".format(loss) + ", Training Accuracy= " + \
"{:.5f}".format(acc))
step += 1
print("MPI "+str(rank)+"] Optimization Finished!")
# Calculate accuracy for 256 mnist test images
print("MPI "+str(rank)+"] Testing Accuracy:", \
sess.run(accuracy, feed_dict={x: mnist.test.images[:256],
y: mnist.test.labels[:256],
keep_prob: 1.}))
일단 caffe는 IBM 버전 caffe (caffe-ibm)에 DDL 옵션이 통합되어 있으므로 별도 debian 패키지를 설치할 필요가 없습니다. 이 caffe-ibm도 내부적으로는 OpenMPI를 이용하는 것이므로 관련 library들이 설치되기는 해야 합니다만, 이는 caffe-ibm을 설치할 때 함께 자동으로 설치되므로 따로 신경쓰지 않으셔도 됩니다.
nimbix@JARVICENAE-0A0A1835:/data/mnist$ dpkg -l | grep openmpi
ii libopenmpi2-cuda:ppc64el 2.0.1-4ibm1 ppc64el high performance message passing library -- shared library
ii openmpi-bin-cuda 2.0.1-4ibm1 ppc64el high performance message passing library -- binaries
ii openmpi-common-cuda 2.0.1-4ibm1 all high performance message passing library -- common files
ii openmpi-doc-cuda 2.0.1-4ibm1 all high performance message passing library -- man pages
가령 위에서 보는 것과 같이 CUDA-aware OpenMPI를 설치하고나면, mpirun이라는 MPI utility가 설치됩니다. 이 mpirun이라는 것은 여러단계의 soft link가 걸린 orterun이라는 명령어이고, 결국 아래와 같이 openmpi-bin-cuda에서 제공됩니다.
nimbix@JARVICENAE-0A0A1835:/data/mnist$ dpkg -S /usr/bin/orterun
openmpi-bin-cuda: /usr/bin/orterun
IBM 버전 caffe에서의 DDL 사용법은 알고 보면 단순합니다. 다음 4가지만 아시면 됩니다.
1) caffe 명령을 수행할 때 -ddl 옵션을 준다
2) Train/Validation용 dataset은 모든 서버에서 동일한 위치(directory)에 존재해야 한다 (병렬파일시스템 또는 NFS가 편리)
3) 모든 서버는 암호 없이 ssh가 되도록 ssh-keygen과 ssh-copy-id가 되어 있어야 한다
4) 환경변수 등을 다른 서버 노드에도 전달하기 위해서는 mpirun 명령을 사용하는 것이 편하다
다른 것은 다 쉽습니다만 1)번 항목이 조금 어렵게 느껴질 수도 있습니다. 복잡한 부분은 다 빼고, 그냥 쉽게 보면 이렇습니다.
DDL 옵션을 쓴다고 해서 caffe가 여러분이 가진 GPU서버 및 network 환경을 스스로 이해하고 그에 맞게 자동으로 최적화할 수는 없습니다. 따라서 그런 환경, 즉 topology를 caffe에게 여러분이 직접 알려주셔야 합니다. 그게 -ddl 옵션의 mode입니다. 쉽게 예를 들어 설명하면 다음과 같습니다.
$ mpirun -x PATH -x LD_LIBRARY_PATH -n 12 -rf 4x3.rf caffe train -solver /data/mnist/lenet_solver.prototxt -gpu 0 -bvlc -ddl "-mode n:4x3x1 -dev_sync 1"
- mpirun은 여러대의 서버 노드에 동일한 명령을 동일한 환경변수 (-x 옵션)을 써서 수행해주는 병렬환경 명령어입니다.
- 4x3.rf라는 이름의 파일은 rank file입니다. 이 속에 병렬 서버 환경의 toplogy가 들어있습니다. 이걸 어떻게 만드는지는 아래에서 다루겠습니다.
- -n 12라는 것은 MPI client의 총 숫자이며, 쉽게 말해 training에 이용하려는 GPU의 갯수입니다.
- -gpu 0에서, 왜 12개가 아니라 gpu 0이라고 1개로 지정했는지 의아하실 수 있는데, MPI 환경에서는 각각의 GPU가 하나의 learner가 됩니다. 따라서 실제 물리적 서버 1대에 GPU가 몇 장 장착되어있든 상관없이 모두 -gpu 0, 즉 GPU는 1개로 지정한 것입니다.
- "-mode n:4x3x1"에서 n이라는 것은 NCCL (NVIDIA Collective Communications Library, 니클이라고 읽습니다)을 이용하라는 뜻입니다. 4x3x1은 4장의 GPU를 가진 서버 3대가 하나의 rack에 들어있다는 뜻입니다. 사실 어느 rack에 들어있느냐가 중요한 것은 아닌데, 보통 병렬수퍼컴 환경에서는 한대의 rack 안에 장착된 서버끼리는 좀더 고속의 low latency network으로 연결되어있기 때문에 이렇게 rack 표시까지 해주는 것입니다. 만약 4장의 GPU를 가진 서버가 6대씩 장착된 rack이 5대있다면 4x6x5로 표시됩니다.
- dev_sync에서 0은 GPU간 sync를 하지 말라는 것이고, 1은 통신 시작할 때 sync하라는 뜻, 2는 시작할 때와 끝낼 때 각각 sync하라는 뜻입니다.
잠깐, 데이터는 어디에 있는지 어떻게 지정하냐고요 ? 저 위에 지정된 solver 파일, 즉 lenet_solver.prototxt에 neural network이 지정되어 있고, 다시 그 neural network의 prototxt 파일 속에 데이터 위치가 지정되어 있습니다. 아래처럼요.
$ vi lenet_solver.prototxt
#net: "examples/mnist/lenet_train_test.prototxt"
net: "/data/mnist/lenet_train_test.prototxt"
$ vi lenet_train_test.prototxt
...
# source: "examples/mnist/mnist_train_lmdb"
source: "/data/mnist/mnist_train_lmdb"
...
# source: "examples/mnist/mnist_test_lmdb"
source: "/data/mnist/mnist_test_lmdb"
여러 서버 노드들의 GPU마다 수행될 learner들이 어떻게 data를 나누어 가져가느냐고요 ? 가급적이면 서버 노드마다 미리 파티셔닝되어 적절히 분배된 data들을 넣어두는 것이 좋습니다. Data를 N개의 learner들이 읽어갈 때, 각자 순차적으로 파일 이름들이 뒤섞여 들어간 목록으로부터 data를 읽어가는데, 만약 이 data가 물리적으로 미리 파티셔닝하여 노드 별로 분배해놓은 것이 아니라면 그냥 1번 training을 끝낼 때마다 전체 data를 N번 (N epochs) training한 것과 같은 효과를 냅니다. 저 data들의 저장소는 여러 노드에서 동시에 access할 수 있도록 IBM Spectrum Scale (구명칭 GPFS) 같은 병렬 파일시스템으로 하든가, 그게 없다면 성능이 떨어지더라도 NFS 같은 것으로 구성하는 것이 좋습니다.
이제 저 rf 파일, 즉 랭크 파일을 어떻게 만드는지 보시겠습니다. 그냥 손으로, 즉 vi 에디터 같은 것을 이용해서 만드셔도 됩니다만, PowerAI에서 기본 제공되는 rank_gen.py를 이용해서 다음과 같이 만드시는 것이 편합니다.
$ python /opt/DL/ddl/bin/rank_gen.py 4x2x3 sys-89074,sys-89075,sys-89076,sys-89077,sys-89078,sys-89079 > 4x2x3.rf
위에서 콤마(,)로 분리된 이름들이 서버 이름들입니다. 4x2x3이니 4장의 GPU를 가진 서버가 총 6대 있는 것이니, 서버 이름은 반드시 6대를 적으셔야 합니다. 이렇게 만들어진 4x2x3.rf 파일의 내용은 아래와 같습니다. rank_gen.py는 기본적으로 10-core POWER8 chip 2장을 장착한 Minsky 서버를 기준으로 만들기 때문에 아래와 같이 10개의 core를 가진 slot 2개가 있는 것으로 나옵니다. 그래서 rank, 즉 GPU 1개마다 slot이 5개 (0-4) 있는 것으로 나오는데, 만약 그게 아니라 8-core POWER8 chip이 장착된 서버라면 수작업으로 0-4가 아닌 0-3으로 수정해주셔야 합니다.
u0017649@sys-89075:~$ cat 4x2x3.rf
#2017-09-14 04:45:51 by rank_gen
#dims = 4x2x3
#host = sys-89074,sys-89075,sys-89076,sys-89077,sys-89078,sys-89079
#dimX = 4
#dimY = 2
#dimZ = 3
#sockets = 2
#cores = 10
rank 0=sys-89074 slot=0:0-4
rank 6=sys-89074 slot=0:5-9
rank 12=sys-89074 slot=1:0-4
rank 18=sys-89074 slot=1:5-9
rank 3=sys-89075 slot=0:0-4
rank 9=sys-89075 slot=0:5-9
rank 15=sys-89075 slot=1:0-4
rank 21=sys-89075 slot=1:5-9
rank 1=sys-89076 slot=0:0-4
rank 7=sys-89076 slot=0:5-9
rank 13=sys-89076 slot=1:0-4
rank 19=sys-89076 slot=1:5-9
rank 4=sys-89077 slot=0:0-4
rank 10=sys-89077 slot=0:5-9
rank 16=sys-89077 slot=1:0-4
rank 22=sys-89077 slot=1:5-9
rank 2=sys-89078 slot=0:0-4
rank 8=sys-89078 slot=0:5-9
rank 14=sys-89078 slot=1:0-4
rank 20=sys-89078 slot=1:5-9
rank 5=sys-89079 slot=0:0-4
rank 11=sys-89079 slot=0:5-9
rank 17=sys-89079 slot=1:0-4
rank 23=sys-89079 slot=1:5-9
Caffe는 그렇게 쉽게 됩니다만, tensorflow는 그보다 좀 어렵습니다. 일단 별도의 ddl-tensorflow라는 debian package가 PowerAI 4.0에 포함되어 있는데, 이는 사실 tensorflow DDL에 꼭 필요한 것이 아니라, tensorflow DDL을 좀더 쉽게 사용하실 수 있도록 해주는 Google Slim에 기반한 script들과 example 파일들을 제공해주는 것입니다. 정작 tensorflow는 별도로 설치하셔야 하는데, 물론 그건 PowerAI에서 제공되는 tensorflow를 apt-get install 명령으로 설치하시면 됩니다.
$ sudo apt-get install ddl-tensorflow tensorflow
$ dpkg -L ddl-tensorflow
/.
/opt
/opt/DL
/opt/DL/ddl-tensorflow
/opt/DL/ddl-tensorflow/examples
/opt/DL/ddl-tensorflow/examples/mnist
/opt/DL/ddl-tensorflow/examples/mnist/ddl_mnist.py
/opt/DL/ddl-tensorflow/examples/mnist/README.md
/opt/DL/ddl-tensorflow/examples/slim
/opt/DL/ddl-tensorflow/examples/slim/BUILD
/opt/DL/ddl-tensorflow/examples/slim/WORKSPACE
/opt/DL/ddl-tensorflow/examples/slim/scripts
/opt/DL/ddl-tensorflow/examples/slim/scripts/finetune_inception_resnet_v2_on_flowers.sh
/opt/DL/ddl-tensorflow/examples/slim/scripts/train_lenet_on_mnist.sh
/opt/DL/ddl-tensorflow/examples/slim/scripts/finetune_resnet_v1_50_on_flowers.sh
/opt/DL/ddl-tensorflow/examples/slim/scripts/finetune_inception_v3_on_flowers.sh
/opt/DL/ddl-tensorflow/examples/slim/scripts/train_cifarnet_on_cifar10.sh
/opt/DL/ddl-tensorflow/examples/slim/scripts/finetune_inception_v1_on_flowers.sh
/opt/DL/ddl-tensorflow/examples/slim/train-alexnet.sh
/opt/DL/ddl-tensorflow/examples/slim/deployment
/opt/DL/ddl-tensorflow/examples/slim/deployment/__init__.py
...
이 ddl-tensorflow를 사용하시기 위해서는 PYTHONPATH 등의 환경변수 설정을 위해 source 명령으로 아래와 같이 ddl-tensorflow-activate를 수행해주셔야 합니다.
$ source /opt/DL/ddl-tensorflow/bin/ddl-tensorflow-activate
이제 ddl-tensorflow-install-samples 명령을 사용하시어 지정하는 directory에 sample들을 설치하실 수 있습니다.
nimbix@JARVICENAE-0A0A1835:~$ ddl-tensorflow-install-samples /data
Write into existing directory /data? (yN)
y
Copying examples/ into /data...
Success
가장 간단한 것으로, 손글씨 숫자를 판독하는 MNIST가 들어 있습니다.
nimbix@JARVICENAE-0A0A1835:~$ cd /data/examples/mnist
nimbix@JARVICENAE-0A0A1835:/data/examples/mnist$ ls
ddl_mnist.py README.md
여기에 나온 것처럼, tensorflow는 명령어라기보다는 python에서 불러 사용하는 library로 되어 있기 때문에, 결국 multi-node 병렬처리를 하기 위해서는 python script를 위의 ddl_mnist.py에서처럼 작성해야 합니다.
일단 4-GPU 서버 2대(sys-89074와 sys-89075)로 수행하는 환경이라고 가정하고 아래와 같이 rank file을 먼저 만듭니다.
nimbix@JARVICENAE-0A0A1835:/data/examples/mnist$ python /opt/DL/ddl/bin/rank_gen.py 4x2x1 sys-89074,sys-89075 > 4x2.rf
nimbix@JARVICENAE-0A0A1835:/data/examples/mnist$ cat 4x2.rf
#2017-09-15 03:19:14 by rank_gen
#dims = 4x2x1
#host = sys-89074,sys-89075
#dimX = 4
#dimY = 2
#dimZ = 1
#sockets = 2
#cores = 10
rank 0=sys-89074 slot=0:0-4
rank 2=sys-89074 slot=0:5-9
rank 4=sys-89074 slot=1:0-4
rank 6=sys-89074 slot=1:5-9
rank 1=sys-89075 slot=0:0-4
rank 3=sys-89075 slot=0:5-9
rank 5=sys-89075 slot=1:0-4
rank 7=sys-89075 slot=1:5-9
이제 다음과 같이 수행하면 됩니다.
nimbix@JARVICENAE-0A0A1835:/data/examples/mnist$ mpirun -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -n 8 -rf 4x2.rf python ddl_mnist.py
(사실 mnist는 워낙 작은 dataset만 사용하므로, 병렬화의 의미가 없습니다. 그래서인지 ddl_mnist.py는 위에서 제가 예로 든 것처럼 4x2 구조는 애초에 불가능하고, 저 아래에 보시듯이 -mode r:2로 되어 있어 그냥 GPU 2장으로 병렬화하는 것만 가능합니다.)
결국 문제는 tensorflow를 병렬로 수행하기 위해서 python script를 어떻게 작성해야 하느냐인데, 이 부분에 대해서는 저도 개발자가 아닌 관계로 별 도움을 못 드리겠습니다. (사실 제겐 흰건 글씨요 검은건 공백이며, 깜빡이는 것은 커서 정도로만 보입니다.)
대신, 다소 깁니다만, 아래에 PowerAI에 포함된 ddl_mnist.py의 내용을 그대로 올려두겠습니다.
nimbix@JARVICENAE-0A0A1835:/data/examples/mnist$ vi ddl_mnist.py
import tensorflow as tf
import numpy as np
############################################################################
# IBM PowerAI Distributed Deep Learning (DDL) setup
############################################################################
# Disable GPU memory preallocation
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
############################################################################
# DDL Initialize BEGIN
############################################################################
# Load DDL operator
ddl = tf.load_op_library('/opt/DL/ddl-tensorflow/lib/ddl_MDR.so')
# DDL initializes MPI on CPU
# ddl.init takes two inputs
# 1) the number of GPUs to utilize on each host in training.
# this number is not the number of GPUs to use for each leaner. It simply tells DDL that there are X GPUs in each host to be used for training
# 2) DDL options (refer to README for details)
with tf.Session(config=config) as sess:
with tf.device('/cpu:0'):
rank, size, gpuid = sess.run(ddl.init(2, mode = '-mode r:2 -dump_iter 100'))
# MPI info and assigned GPU
print [rank, size, gpuid]
############################################################################
# DDL Initialize END
############################################################################
# Perform all TensorFlow computation within gpuid
with tf.device('/gpu:%d' %gpuid):
##############################################################################
# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
# Parameters
learning_rate = 0.001
training_iters = 200000
batch_size = 100
display_step = 1
# Network Parameters
n_input = 784 # MNIST data input (img shape: 28*28)
n_classes = 10 # MNIST total classes (0-9 digits)
dropout = 0.75 # Dropout, probability to keep units
# tf Graph input
x = tf.placeholder(tf.float32, [None, n_input])
y = tf.placeholder(tf.float32, [None, n_classes])
keep_prob = tf.placeholder(tf.float32) #dropout (keep probability)
# Create some wrappers for simplicity
def conv2d(x, W, b, strides=1):
# Conv2D wrapper, with bias and relu activation
x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding='SAME')
x = tf.nn.bias_add(x, b)
return tf.nn.relu(x)
def maxpool2d(x, k=2):
# MaxPool2D wrapper
return tf.nn.max_pool(x, ksize=[1, k, k, 1], strides=[1, k, k, 1],
padding='SAME')
# Create model
def conv_net(x, weights, biases, dropout):
# Reshape input picture
x = tf.reshape(x, shape=[-1, 28, 28, 1])
# Convolution Layer
conv1 = conv2d(x, weights['wc1'], biases['bc1'])
# Max Pooling (down-sampling)
conv1 = maxpool2d(conv1, k=2)
# Convolution Layer
conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])
# Max Pooling (down-sampling)
conv2 = maxpool2d(conv2, k=2)
# Fully connected layer
# Reshape conv2 output to fit fully connected layer input
fc1 = tf.reshape(conv2, [-1, weights['wd1'].get_shape().as_list()[0]])
fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])
fc1 = tf.nn.relu(fc1)
# Apply Dropout
fc1 = tf.nn.dropout(fc1, dropout)
# Output, class prediction
out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])
return out
# Store layers weight & bias
weights = {
############################################################################
# DDL BROADCAST BEGIN
############################################################################
# This step ensures that all learners start with the same initial parameters
# 5x5 conv, 1 input, 32 outputs
'wc1': tf.Variable(ddl.bcast(tf.random_normal([5, 5, 1, 32]))),
# 5x5 conv, 32 inputs, 64 outputs
'wc2': tf.Variable(ddl.bcast(tf.random_normal([5, 5, 32, 64]))),
# fully connected, 7*7*64 inputs, 1024 outputs
'wd1': tf.Variable(ddl.bcast(tf.random_normal([7*7*64, 1024]))),
# 1024 inputs, 10 outputs (class prediction)
'out': tf.Variable(ddl.bcast(tf.random_normal([1024, n_classes])))
############################################################################
# DDL BROADCAST END
############################################################################
}
biases = {
'bc1': tf.Variable(ddl.bcast(tf.random_normal([32]))),
'bc2': tf.Variable(ddl.bcast(tf.random_normal([64]))),
'bd1': tf.Variable(ddl.bcast(tf.random_normal([1024]))),
'out': tf.Variable(ddl.bcast(tf.random_normal([n_classes])))
}
# Construct model
pred = conv_net(x, weights, biases, keep_prob)
# Define loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
############################################################################
# DDL ALLREDUCE BEGIN
############################################################################
# Collect the gradients and the corresponding parameters w.r.t the given cost
grads_and_vars = optimizer.compute_gradients(cost)
# Separate out the tuple
grads, vars = zip(*grads_and_vars)
# This step takes the average of the gradients on all the learners
grads_and_vars_ddl = zip(ddl.all_reduce_n(grads, op='avg'), vars)
# Update the parameters with the averaged gradient
objective = optimizer.apply_gradients(grads_and_vars_ddl)
############################################################################
# DDL ALLREDUCE END
############################################################################
# Evaluate model
correct_pred = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
##############################################################################
def split(a, n):
k, m = divmod(len(a), n)
return (a[i * k + min(i, m):(i + 1) * k + min(i + 1, m)] for i in xrange(n))
# Launch the graph
with tf.Session(config=config) as sess:
sess.run(tf.global_variables_initializer())
step = 1
# Keep training until reach max iterations
while step * batch_size < training_iters:
# Each learner will read batch_size*size samples and
# use only the portion correspoding to the current learner (or rank)
batch_x, batch_y = mnist.train.next_batch(batch_size*size)
batch_x = np.split(batch_x,size)[rank]
batch_y = np.split(batch_y,size)[rank]
# Run optimization op (backprop)
sess.run(objective, feed_dict={x: batch_x, y: batch_y,
keep_prob: dropout})
if step % display_step == 0:
# Calculate batch loss and accuracy
loss, acc = sess.run([cost, accuracy], feed_dict={x: batch_x,
y: batch_y,
keep_prob: 1.})
print("MPI "+str(rank)+"] Iter " + str(step*batch_size) + ", Minibatch Loss= " + \
"{:.6f}".format(loss) + ", Training Accuracy= " + \
"{:.5f}".format(acc))
step += 1
print("MPI "+str(rank)+"] Optimization Finished!")
# Calculate accuracy for 256 mnist test images
print("MPI "+str(rank)+"] Testing Accuracy:", \
sess.run(accuracy, feed_dict={x: mnist.test.images[:256],
y: mnist.test.labels[:256],
keep_prob: 1.}))
2017년 7월 3일 월요일
Minsky 서버(ppc64le)에서 tensorflow serving 설치하기
Tensorflow serving은 tensorflow를 이용해 training 시킨 model을 운영 환경으로 편리하게 deploy시켜주기 위한 SW platform입니다. 이를 build하기 위한 절차는 https://tensorflow.github.io/serving/setup 에 잘 나와 있습니다만, 여기서는 Minsky 서버의 아키텍처인 ppc64le, 즉 IBM POWER8 환경에서도 사용이 가능한지 검증을 해보겠습니다.
결론부터 이야기하면 잘 됩니다.
먼저, 관련된 Ubuntu OS package들을 설치합니다.
u0017496@sys-88049:~$ sudo apt-get install -y build-essential curl libcurl3-dev git libfreetype6-dev libpng12-dev libzmq3-dev pkg-config python-dev python-numpy python-pip software-properties-common swig zip zlib1g-dev openjdk-8-jdk openjdk-8-jdk-headless
이어서, 앞선 posting( http://hwengineer.blogspot.kr/2017/05/minsky-continuum-anaconda.html )을 참조하여 ppc64le Ubuntu에 mini-conda를 설치합니다. 그러고 난 뒤에 아래와 같이 tensorflow, tensorflow-gpu, bazel, cudatoolkit, cudnn 6.0 등의 conda package를 설치합니다.
u0017496@sys-88049:~$ which conda
/home/u0017496/miniconda2/bin/conda
u0017496@sys-88049:~$ conda install tensorflow tensorflow-gpu
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda2:
The following NEW packages will be INSTALLED:
cudatoolkit: 8.0-0
cudnn: 6.0.21-0
funcsigs: 1.0.2-py27_0
libprotobuf: 3.2.0-0
mock: 2.0.0-py27_0
numpy: 1.12.1-py27_0
openblas: 0.2.19-0
pbr: 1.10.0-py27_0
protobuf: 3.2.0-py27_0
tensorflow: 1.1.0-np112py27_0
tensorflow-gpu: 1.1.0-np112py27_0
werkzeug: 0.12.2-py27_0
The following packages will be UPDATED:
conda: 4.3.14-py27_0 --> 4.3.18-py27_0
Proceed ([y]/n)? y
cudatoolkit-8. 100% |####################################| Time: 0:00:26 12.45 MB/s
cudnn-6.0.21-0 100% |####################################| Time: 0:00:14 12.99 MB/s
openblas-0.2.1 100% |####################################| Time: 0:00:00 12.72 MB/s
libprotobuf-3. 100% |####################################| Time: 0:00:00 14.28 MB/s
funcsigs-1.0.2 100% |####################################| Time: 0:00:00 22.66 MB/s
numpy-1.12.1-p 100% |####################################| Time: 0:00:00 11.26 MB/s
werkzeug-0.12. 100% |####################################| Time: 0:00:00 9.26 MB/s
protobuf-3.2.0 100% |####################################| Time: 0:00:00 6.76 MB/s
pbr-1.10.0-py2 100% |####################################| Time: 0:00:00 6.81 MB/s
mock-2.0.0-py2 100% |####################################| Time: 0:00:00 24.93 MB/s
conda-4.3.18-p 100% |####################################| Time: 0:00:00 7.88 MB/s
tensorflow-1.1 100% |####################################| Time: 0:00:01 12.84 MB/s
tensorflow-gpu 100% |####################################| Time: 0:00:07 12.54 MB/s
u0017496@sys-88049:~$ conda install bazel curl pkg-config
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda2:
The following NEW packages will be INSTALLED:
bazel: 0.4.5-0
curl: 7.52.1-0
pkg-config: 0.28-1
Proceed ([y]/n)? y
bazel-0.4.5-0. 100% |####################################| Time: 0:00:08 16.43 MB/s
curl-7.52.1-0. 100% |####################################| Time: 0:00:00 9.12 MB/s
pkg-config-0.2 100% |####################################| Time: 0:00:00 16.10 MB/s
u0017496@sys-88049:~$ which bazel
/home/u0017496/miniconda2/bin/bazel
u0017496@sys-88049:~$ which pip
/home/u0017496/miniconda2/bin/pip
Tensorflow serving을 위해서는 먼저 grpcio package를 pip 명령으로 설치해야 합니다. 이때 google protobuf도 자동으로 함께 설치됩니다.
u0017496@sys-88049:~$ pip install grpcio
Collecting grpcio
Downloading grpcio-1.4.0.tar.gz (9.1MB)
100% |████████████████████████████████| 9.1MB 131kB/s
Requirement already satisfied: six>=1.5.2 in ./miniconda2/lib/python2.7/site-packages (from grpcio)
Collecting protobuf>=3.3.0 (from grpcio)
Downloading protobuf-3.3.0.tar.gz (271kB)
100% |████████████████████████████████| 276kB 2.4MB/s
Collecting futures>=2.2.0 (from grpcio)
Downloading futures-3.1.1-py2-none-any.whl
Requirement already satisfied: enum34>=1.0.4 in ./miniconda2/lib/python2.7/site-packages (from grpcio)
Requirement already satisfied: setuptools in ./miniconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg (from protobuf>=3.3.0->grpcio)
Building wheels for collected packages: grpcio, protobuf
Running setup.py bdist_wheel for grpcio ... /
...
Successfully built grpcio protobuf
Installing collected packages: protobuf, futures, grpcio
Found existing installation: protobuf 3.2.0
Uninstalling protobuf-3.2.0:
Successfully uninstalled protobuf-3.2.0
Successfully installed futures-3.1.1 grpcio-1.4.0 protobuf-3.3.0
이제 github에서 tensorflow serving의 source를 다운로드 받습니다.
u0017496@sys-88049:~$ git clone --recurse-submodules https://github.com/tensorflow/serving
그중 먼저 tensorflow directory에 들어가 configure를 수행합니다. 이때 대부분의 질문에는 default로 enter만 누르면 됩니다만, CUDA support와 함께 build하겠느냐는 것과, cudnn library의 위치는 아래와 같이 직접 입력하셔야 합니다. cudnn library의 위치는 OS에서 설치한 것 말고, conda install로 설치한 것의 directory를 입력하십시요.
u0017496@sys-88049:~$ cd serving/tensorflow
u0017496@sys-88049:~/serving/tensorflow$ ./configure
Extracting Bazel installation...
......................
You have bazel 0.4.5- installed.
Please specify the location of python. [Default is /home/u0017496/miniconda2/bin/python]:
Found possible Python library paths:
/home/u0017496/miniconda2/lib/python2.7/site-packages
Please input the desired Python library path to use. Default is [/home/u0017496/miniconda2/lib/python2.7/site-packages]
Using python library path: /home/u0017496/miniconda2/lib/python2.7/site-packages
Do you wish to build TensorFlow with MKL support? [y/N]
No MKL support will be enabled for TensorFlow
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:
Do you wish to use jemalloc as the malloc implementation? [Y/n]
jemalloc enabled
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N]
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N]
No Hadoop File System support will be enabled for TensorFlow
Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N]
No XLA support will be enabled for TensorFlow
Do you wish to build TensorFlow with VERBS support? [y/N]
No VERBS support will be enabled for TensorFlow
Do you wish to build TensorFlow with OpenCL support? [y/N]
No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with CUDA support? [y/N] y
CUDA support will be enabled for TensorFlow
Do you want to use clang as CUDA compiler? [y/N]
nvcc will be used as CUDA compiler
Please specify the CUDA SDK version you want to use, e.g. 7.0. [Leave empty to default to CUDA 8.0]:
Please specify the location where CUDA toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 6.0]:
Please specify the location where cuDNN library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /home/u0017496/miniconda2/pkgs/cudnn-6.0.21-0/lib
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]:
Do you wish to build TensorFlow with MPI support? [y/N]
MPI support will not be enabled for TensorFlow
Configuration finished
이제 다시 한칸 위의 directory로 올라가 bazel build를 수행하면 됩니다. 이 과정은 상당히 오래 걸리고, 또 disk 공간도 약 700MB 정도 차지합니다. 특히 여기서는 Power Development Platform이라는 POWER8 기반의 cloud 환경에서 가상화된 CPU 1개를 이용했기 때문에 거의 2시간 가까이 걸렸습니다.
u0017496@sys-88049:~/serving/tensorflow$ cd ..
u0017496@sys-88049:~/serving$ bazel build tensorflow_serving/...
..............
WARNING: /home/u0017496/.cache/bazel/_bazel_u0017496/71c9acd4ac5e4078b5a9612ad32f9c09/external/org_tensorflow/third_party/py/python_configure.bzl:31:3: Python Configuration Warning: 'PYTHON_LIB_PATH' environment variable is not set, using '/usr/local/lib/python2.7/dist-packages' as default.
WARNING: /home/u0017496/serving/tensorflow_serving/servables/tensorflow/BUILD:525:1: in cc_library rule //tensorflow_serving/servables/tensorflow:classifier: target '//tensorflow_serving/servables/tensorflow:classifier' depends on deprecated target '@org_tensorflow//tensorflow/contrib/session_bundle:session_bundle': Use SavedModel Loader instead.
WARNING: /home/u0017496/serving/tensorflow_serving/servables/tensorflow/BUILD:525:1: in cc_library rule //tensorflow_serving/servables/tensorflow:classifier: target '//tensorflow_serving/servables/tensorflow:classifier' depends on deprecated target '@org_tensorflow//tensorflow/contrib/session_bundle:signature': Use SavedModel instead.
..............
[1,125 / 4,270] Compiling external/protobuf/src/google/protobuf/compiler/cpp/cpp_m\
essage_field.cc
...
INFO: From Compiling external/org_tensorflow/tensorflow/tools/tfprof/internal/print_model_analysis.cc:
In file included from external/org_tensorflow/tensorflow/tools/tfprof/internal/tfprof_stats.h:41:0,
from external/org_tensorflow/tensorflow/tools/tfprof/internal/advisor/checker.h:20,
from external/org_tensorflow/tensorflow/tools/tfprof/internal/advisor/accelerator_utilization_checker.h:19,
from external/org_tensorflow/tensorflow/tools/tfprof/internal/advisor/tfprof_advisor.h:19,
from external/org_tensorflow/tensorflow/tools/tfprof/internal/print_model_analysis.cc:26:
external/org_tensorflow/tensorflow/tools/tfprof/internal/tfprof_op.h: In member function 'virtual bool tensorflow::tfprof::TFOp::ShouldShowIfExtra(tensorflow::tfprof::ShowMultiNode*, const tensorflow::tfprof::Options&, int)':
external/org_tensorflow/tensorflow/tools/tfprof/internal/tfprof_op.h:61:29: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (opts.min_occurrence > node->node->graph_nodes().size()) {
^
INFO: Elapsed time: 6744.345s, Critical Path: 5770.33s
위의 bazel build 결과 아래와 같은 binary file이 생성되고, 수행해보면 아래와 같은 usage가 display 됩니다.
u0017496@sys-88049:~/serving$ file bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, interpreter /lib64/ld64.so.2, for GNU/Linux 3.2.0, BuildID[md5/uuid]=a9e44548b6dbbe05d18b5711e5e29e1a, not stripped
u0017496@sys-88049:~/serving$ bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server
usage: bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server
Flags:
--port=8500 int32 port to listen on
--enable_batching=false bool enable batching
--batching_parameters_file="" string If non-empty, read an ascii BatchingParameters protobuf from the supplied file name and use the contained values instead of the defaults.
--model_config_file="" string If non-empty, read an ascii ModelServerConfig protobuf from the supplied file name, and serve the models in that file. (If used, --model_name, --model_base_path and --model_version_policy are ignored.)
--model_name="default" string name of model (ignored if --model_config_file flag is set
--model_base_path="" string path to export (ignored if --model_config_file flag is set, otherwise required)
--model_version_policy="LATEST_VERSION" string The version policy which determines the number of model versions to be served at the same time. The default value is LATEST_VERSION, which will serve only the latest version. See file_system_storage_path_source.proto for the list of possible VersionPolicy. (Ignored if --model_config_file flag is set)
--file_system_poll_wait_seconds=1 int32 interval in seconds between each poll of the file system for new model version
--use_saved_model=true bool If true, use SavedModel in the server; otherwise, use SessionBundle. It is used by tensorflow serving team to control the rollout of SavedModel and is not expected to be set by users directly.
--tensorflow_session_parallelism=0 int64 Number of threads to use for running a Tensorflow session. Auto-configured by default.Note that this option is ignored if --platform_config_file is non-empty.
--platform_config_file="" string If non-empty, read an ascii PlatformConfigMap protobuf from the supplied file name, and use that platform config instead of the Tensorflow platform. (If used, --enable_batching and --use_saved_model are ignored.)
결론부터 이야기하면 잘 됩니다.
먼저, 관련된 Ubuntu OS package들을 설치합니다.
u0017496@sys-88049:~$ sudo apt-get install -y build-essential curl libcurl3-dev git libfreetype6-dev libpng12-dev libzmq3-dev pkg-config python-dev python-numpy python-pip software-properties-common swig zip zlib1g-dev openjdk-8-jdk openjdk-8-jdk-headless
이어서, 앞선 posting( http://hwengineer.blogspot.kr/2017/05/minsky-continuum-anaconda.html )을 참조하여 ppc64le Ubuntu에 mini-conda를 설치합니다. 그러고 난 뒤에 아래와 같이 tensorflow, tensorflow-gpu, bazel, cudatoolkit, cudnn 6.0 등의 conda package를 설치합니다.
u0017496@sys-88049:~$ which conda
/home/u0017496/miniconda2/bin/conda
u0017496@sys-88049:~$ conda install tensorflow tensorflow-gpu
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda2:
The following NEW packages will be INSTALLED:
cudatoolkit: 8.0-0
cudnn: 6.0.21-0
funcsigs: 1.0.2-py27_0
libprotobuf: 3.2.0-0
mock: 2.0.0-py27_0
numpy: 1.12.1-py27_0
openblas: 0.2.19-0
pbr: 1.10.0-py27_0
protobuf: 3.2.0-py27_0
tensorflow: 1.1.0-np112py27_0
tensorflow-gpu: 1.1.0-np112py27_0
werkzeug: 0.12.2-py27_0
The following packages will be UPDATED:
conda: 4.3.14-py27_0 --> 4.3.18-py27_0
Proceed ([y]/n)? y
cudatoolkit-8. 100% |####################################| Time: 0:00:26 12.45 MB/s
cudnn-6.0.21-0 100% |####################################| Time: 0:00:14 12.99 MB/s
openblas-0.2.1 100% |####################################| Time: 0:00:00 12.72 MB/s
libprotobuf-3. 100% |####################################| Time: 0:00:00 14.28 MB/s
funcsigs-1.0.2 100% |####################################| Time: 0:00:00 22.66 MB/s
numpy-1.12.1-p 100% |####################################| Time: 0:00:00 11.26 MB/s
werkzeug-0.12. 100% |####################################| Time: 0:00:00 9.26 MB/s
protobuf-3.2.0 100% |####################################| Time: 0:00:00 6.76 MB/s
pbr-1.10.0-py2 100% |####################################| Time: 0:00:00 6.81 MB/s
mock-2.0.0-py2 100% |####################################| Time: 0:00:00 24.93 MB/s
conda-4.3.18-p 100% |####################################| Time: 0:00:00 7.88 MB/s
tensorflow-1.1 100% |####################################| Time: 0:00:01 12.84 MB/s
tensorflow-gpu 100% |####################################| Time: 0:00:07 12.54 MB/s
u0017496@sys-88049:~$ conda install bazel curl pkg-config
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/u0017496/miniconda2:
The following NEW packages will be INSTALLED:
bazel: 0.4.5-0
curl: 7.52.1-0
pkg-config: 0.28-1
Proceed ([y]/n)? y
bazel-0.4.5-0. 100% |####################################| Time: 0:00:08 16.43 MB/s
curl-7.52.1-0. 100% |####################################| Time: 0:00:00 9.12 MB/s
pkg-config-0.2 100% |####################################| Time: 0:00:00 16.10 MB/s
u0017496@sys-88049:~$ which bazel
/home/u0017496/miniconda2/bin/bazel
u0017496@sys-88049:~$ which pip
/home/u0017496/miniconda2/bin/pip
Tensorflow serving을 위해서는 먼저 grpcio package를 pip 명령으로 설치해야 합니다. 이때 google protobuf도 자동으로 함께 설치됩니다.
u0017496@sys-88049:~$ pip install grpcio
Collecting grpcio
Downloading grpcio-1.4.0.tar.gz (9.1MB)
100% |████████████████████████████████| 9.1MB 131kB/s
Requirement already satisfied: six>=1.5.2 in ./miniconda2/lib/python2.7/site-packages (from grpcio)
Collecting protobuf>=3.3.0 (from grpcio)
Downloading protobuf-3.3.0.tar.gz (271kB)
100% |████████████████████████████████| 276kB 2.4MB/s
Collecting futures>=2.2.0 (from grpcio)
Downloading futures-3.1.1-py2-none-any.whl
Requirement already satisfied: enum34>=1.0.4 in ./miniconda2/lib/python2.7/site-packages (from grpcio)
Requirement already satisfied: setuptools in ./miniconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg (from protobuf>=3.3.0->grpcio)
Building wheels for collected packages: grpcio, protobuf
Running setup.py bdist_wheel for grpcio ... /
...
Successfully built grpcio protobuf
Installing collected packages: protobuf, futures, grpcio
Found existing installation: protobuf 3.2.0
Uninstalling protobuf-3.2.0:
Successfully uninstalled protobuf-3.2.0
Successfully installed futures-3.1.1 grpcio-1.4.0 protobuf-3.3.0
이제 github에서 tensorflow serving의 source를 다운로드 받습니다.
u0017496@sys-88049:~$ git clone --recurse-submodules https://github.com/tensorflow/serving
그중 먼저 tensorflow directory에 들어가 configure를 수행합니다. 이때 대부분의 질문에는 default로 enter만 누르면 됩니다만, CUDA support와 함께 build하겠느냐는 것과, cudnn library의 위치는 아래와 같이 직접 입력하셔야 합니다. cudnn library의 위치는 OS에서 설치한 것 말고, conda install로 설치한 것의 directory를 입력하십시요.
u0017496@sys-88049:~$ cd serving/tensorflow
u0017496@sys-88049:~/serving/tensorflow$ ./configure
Extracting Bazel installation...
......................
You have bazel 0.4.5- installed.
Please specify the location of python. [Default is /home/u0017496/miniconda2/bin/python]:
Found possible Python library paths:
/home/u0017496/miniconda2/lib/python2.7/site-packages
Please input the desired Python library path to use. Default is [/home/u0017496/miniconda2/lib/python2.7/site-packages]
Using python library path: /home/u0017496/miniconda2/lib/python2.7/site-packages
Do you wish to build TensorFlow with MKL support? [y/N]
No MKL support will be enabled for TensorFlow
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:
Do you wish to use jemalloc as the malloc implementation? [Y/n]
jemalloc enabled
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N]
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N]
No Hadoop File System support will be enabled for TensorFlow
Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N]
No XLA support will be enabled for TensorFlow
Do you wish to build TensorFlow with VERBS support? [y/N]
No VERBS support will be enabled for TensorFlow
Do you wish to build TensorFlow with OpenCL support? [y/N]
No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with CUDA support? [y/N] y
CUDA support will be enabled for TensorFlow
Do you want to use clang as CUDA compiler? [y/N]
nvcc will be used as CUDA compiler
Please specify the CUDA SDK version you want to use, e.g. 7.0. [Leave empty to default to CUDA 8.0]:
Please specify the location where CUDA toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 6.0]:
Please specify the location where cuDNN library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /home/u0017496/miniconda2/pkgs/cudnn-6.0.21-0/lib
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]:
Do you wish to build TensorFlow with MPI support? [y/N]
MPI support will not be enabled for TensorFlow
Configuration finished
이제 다시 한칸 위의 directory로 올라가 bazel build를 수행하면 됩니다. 이 과정은 상당히 오래 걸리고, 또 disk 공간도 약 700MB 정도 차지합니다. 특히 여기서는 Power Development Platform이라는 POWER8 기반의 cloud 환경에서 가상화된 CPU 1개를 이용했기 때문에 거의 2시간 가까이 걸렸습니다.
u0017496@sys-88049:~/serving/tensorflow$ cd ..
u0017496@sys-88049:~/serving$ bazel build tensorflow_serving/...
..............
WARNING: /home/u0017496/.cache/bazel/_bazel_u0017496/71c9acd4ac5e4078b5a9612ad32f9c09/external/org_tensorflow/third_party/py/python_configure.bzl:31:3: Python Configuration Warning: 'PYTHON_LIB_PATH' environment variable is not set, using '/usr/local/lib/python2.7/dist-packages' as default.
WARNING: /home/u0017496/serving/tensorflow_serving/servables/tensorflow/BUILD:525:1: in cc_library rule //tensorflow_serving/servables/tensorflow:classifier: target '//tensorflow_serving/servables/tensorflow:classifier' depends on deprecated target '@org_tensorflow//tensorflow/contrib/session_bundle:session_bundle': Use SavedModel Loader instead.
WARNING: /home/u0017496/serving/tensorflow_serving/servables/tensorflow/BUILD:525:1: in cc_library rule //tensorflow_serving/servables/tensorflow:classifier: target '//tensorflow_serving/servables/tensorflow:classifier' depends on deprecated target '@org_tensorflow//tensorflow/contrib/session_bundle:signature': Use SavedModel instead.
..............
[1,125 / 4,270] Compiling external/protobuf/src/google/protobuf/compiler/cpp/cpp_m\
essage_field.cc
...
INFO: From Compiling external/org_tensorflow/tensorflow/tools/tfprof/internal/print_model_analysis.cc:
In file included from external/org_tensorflow/tensorflow/tools/tfprof/internal/tfprof_stats.h:41:0,
from external/org_tensorflow/tensorflow/tools/tfprof/internal/advisor/checker.h:20,
from external/org_tensorflow/tensorflow/tools/tfprof/internal/advisor/accelerator_utilization_checker.h:19,
from external/org_tensorflow/tensorflow/tools/tfprof/internal/advisor/tfprof_advisor.h:19,
from external/org_tensorflow/tensorflow/tools/tfprof/internal/print_model_analysis.cc:26:
external/org_tensorflow/tensorflow/tools/tfprof/internal/tfprof_op.h: In member function 'virtual bool tensorflow::tfprof::TFOp::ShouldShowIfExtra(tensorflow::tfprof::ShowMultiNode*, const tensorflow::tfprof::Options&, int)':
external/org_tensorflow/tensorflow/tools/tfprof/internal/tfprof_op.h:61:29: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (opts.min_occurrence > node->node->graph_nodes().size()) {
^
INFO: Elapsed time: 6744.345s, Critical Path: 5770.33s
위의 bazel build 결과 아래와 같은 binary file이 생성되고, 수행해보면 아래와 같은 usage가 display 됩니다.
u0017496@sys-88049:~/serving$ file bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, interpreter /lib64/ld64.so.2, for GNU/Linux 3.2.0, BuildID[md5/uuid]=a9e44548b6dbbe05d18b5711e5e29e1a, not stripped
u0017496@sys-88049:~/serving$ bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server
usage: bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server
Flags:
--port=8500 int32 port to listen on
--enable_batching=false bool enable batching
--batching_parameters_file="" string If non-empty, read an ascii BatchingParameters protobuf from the supplied file name and use the contained values instead of the defaults.
--model_config_file="" string If non-empty, read an ascii ModelServerConfig protobuf from the supplied file name, and serve the models in that file. (If used, --model_name, --model_base_path and --model_version_policy are ignored.)
--model_name="default" string name of model (ignored if --model_config_file flag is set
--model_base_path="" string path to export (ignored if --model_config_file flag is set, otherwise required)
--model_version_policy="LATEST_VERSION" string The version policy which determines the number of model versions to be served at the same time. The default value is LATEST_VERSION, which will serve only the latest version. See file_system_storage_path_source.proto for the list of possible VersionPolicy. (Ignored if --model_config_file flag is set)
--file_system_poll_wait_seconds=1 int32 interval in seconds between each poll of the file system for new model version
--use_saved_model=true bool If true, use SavedModel in the server; otherwise, use SessionBundle. It is used by tensorflow serving team to control the rollout of SavedModel and is not expected to be set by users directly.
--tensorflow_session_parallelism=0 int64 Number of threads to use for running a Tensorflow session. Auto-configured by default.Note that this option is ignored if --platform_config_file is non-empty.
--platform_config_file="" string If non-empty, read an ascii PlatformConfigMap protobuf from the supplied file name, and use that platform config instead of the Tensorflow platform. (If used, --enable_batching and --use_saved_model are ignored.)
2017년 6월 15일 목요일
Minsky 서버의 Ubuntu 16.04 ppc64le에 JDK 7 설치하기
ppc64le 아키텍처인 Minsky 서버는 Ubuntu 16.04 (또는 Redhat 7.3) 이후부터 지원됩니다. 그러다보니 Ubuntu 16.04에 포함된 openJDK8을 주로 사용하는데, 간혹 JDK7을 필요로 하는 경우도 있을 수 있습니다. 이때는 아래와 같이 설치하여 사용하실 수 있는데, 크게 2가지 방법, 즉 IBM java SDK 7을 설치하거나 또는 openJDK7을 설치하는 방법이 있습니다. 둘 중 원하시는 것을 선택하시면 됩니다.
1) IBM SDK 7 설치
아래 URL에서 ibm-java-ppc64le-sdk-7.1-4.5.bin 을 다운로드 받습니다. Minsky는 POWER8의 big endian이 아닌, x86과 같은 little endian을 지원하므로 'Linux on Power Systems 64-bit'가 아닌 'Linux on Power Systems 64-bit LE'을 택해야 합니다.
https://developer.ibm.com/javasdk/downloads/sdk7r1/
Linux on Power Systems 64-bit LE
Installable package (InstallAnywhere as root)
(File name: ibm-java-ppc64le-sdk-7.1-4.5.bin, Size: 128MB)
u0017496@sys-87548:~$ chmod a+x ibm-java-ppc64le-sdk-7.1-4.5.bin
u0017496@sys-87548:~$ sudo ./ibm-java-ppc64le-sdk-7.1-4.5.bin
...
===============================================================================
Installation Complete
---------------------
Congratulations. IBM 64-bit SDK for Linux, Java Technology Edition, Version 7.1
has been successfully installed to:
/opt/ibm/java-ppc64le-71
PRESS <ENTER> TO EXIT THE INSTALLER:
설치 directory가 아래와 같으므로, user profile의 PATH의 맨 앞에 다음과 같이 설정해서 쓰시면 됩니다.
u0017496@sys-87548:~$ export PATH=/opt/ibm/java-ppc64le-71/bin:$PATH
u0017496@sys-87548:~$ which java
/opt/ibm/java-ppc64le-71/bin/java
u0017496@sys-87548:~$ java -version
java version "1.7.0"
Java(TM) SE Runtime Environment (build pxl6470_27sr4fp5-20170421_02(SR4 FP5))
IBM J9 VM (build 2.7, JRE 1.7.0 Linux ppc64le-64 Compressed References 20170406_343211 (JIT enabled, AOT enabled)
J9VM - R27_Java727_SR4_20170406_2107_B343211
JIT - tr.r13.java_20170406_343211
GC - R27_Java727_SR4_20170406_2107_B343211_CMPRSS
J9CL - 20170406_343211)
JCL - 20170420_01 based on Oracle jdk7u141-b11
2) openJDK 7 설치
Ubuntu 16에는 JDK7이 지원되지 않습니다만, 아래와 같이 PPA repository에서 제공되기는 합니다. 다만, 이 설치 과정 중에 Oracle installer를 download 받아서 그걸 이용하는 과정이 있는데, ppc64le는 Oracle과 친하지 않아 그 과정은 실패합니다. 그러나 그와는 무관하게 JDK7은 설치되고 사용할 수 있습니다.
u0017496@sys-87548:~$ sudo add-apt-repository ppa:openjdk-r/ppa
More info: https://launchpad.net/~openjdk-r/+archive/ubuntu/ppa
Press [ENTER] to continue or ctrl-c to cancel adding it
gpg: keyring `/tmp/tmpgyngeisu/secring.gpg' created
gpg: keyring `/tmp/tmpgyngeisu/pubring.gpg' created
gpg: requesting key 86F44E2A from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpgyngeisu/trustdb.gpg: trustdb created
gpg: key 86F44E2A: public key "Launchpad OpenJDK builds (all archs)" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK
u0017496@sys-87548:~$ sudo apt-get update
u0017496@sys-87548:~$ apt-cache pkgnames | grep openjdk-7
openjdk-7-jre-zero
openjdk-7-jre-lib
openjdk-7-source
oracle-jdk7-installer
openjdk-7-demo
openjdk-7-jre-headless
openjdk-7-dbg
openjdk-7-doc
openjdk-7-jdk
openjdk-7-jre
u0017496@sys-87548:~$ sudo apt-get install openjdk-7-jdk
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
at-spi2-core dbus-x11 gconf-service gconf-service-backend gconf2 gconf2-common libatk-bridge2.0-0
libatk-wrapper-java libatk-wrapper-java-jni libatspi2.0-0 libavahi-glib1 libbonobo2-0
libbonobo2-common libcanberra0 libgconf-2-4 libgnome-2-0 libgnome2-common libgnomevfs2-0
libgnomevfs2-common liborbit-2-0 libsctp1 libtdb1 libvorbisfile3 libxaw7 libxcb-shape0 libxft2
libxmu6 libxpm4 libxv1 libxxf86dga1 openjdk-7-jre openjdk-7-jre-headless sound-theme-freedesktop
x11-utils
Suggested packages:
gconf-defaults-service libbonobo2-bin libcanberra-gtk0 libcanberra-pulse desktop-base
libgnomevfs2-bin libgnomevfs2-extra gamin | fam gnome-mime-data lksctp-tools openjdk-7-demo
openjdk-7-source visualvm libnss-mdns sun-java6-fonts fonts-ipafont-gothic fonts-ipafont-mincho
ttf-wqy-microhei | ttf-wqy-zenhei fonts-indic mesa-utils
The following NEW packages will be installed:
at-spi2-core dbus-x11 gconf-service gconf-service-backend gconf2 gconf2-common libatk-bridge2.0-0
libatk-wrapper-java libatk-wrapper-java-jni libatspi2.0-0 libavahi-glib1 libbonobo2-0
libbonobo2-common libcanberra0 libgconf-2-4 libgnome-2-0 libgnome2-common libgnomevfs2-0
libgnomevfs2-common liborbit-2-0 libsctp1 libtdb1 libvorbisfile3 libxaw7 libxcb-shape0 libxft2
libxmu6 libxpm4 libxv1 libxxf86dga1 openjdk-7-jdk openjdk-7-jre openjdk-7-jre-headless
sound-theme-freedesktop x11-utils
0 upgraded, 35 newly installed, 0 to remove and 39 not upgraded.
1 not fully installed or removed.
Need to get 54.6 MB of archives.
After this operation, 91.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial/main ppc64el libbonobo2-common all 2.32.1-3 [34.7 kB]
Get:2 http://ppa.launchpad.net/openjdk-r/ppa/ubuntu xenial/main ppc64el openjdk-7-jre-headless ppc64el 7u95-2.6.4-3 [38.6 MB]
...
Processing triggers for hicolor-icon-theme (0.15-0ubuntu1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Setting up oracle-java7-installer (7u80+7u60arm-0~webupd8~1) ...
Please report to author unsupported platform 'ppc64le'.
Proceeding without web browser plugin support
Please report to author unsupported platform 'ppc64le'.
Proceeding without web browser plugin support
...
AuthParam=1497491097_5726d7e0ebd89a8675567ef8793c3443 [following]
--2017-06-14 21:42:57-- http://download.oracle.com/otn-pub/java/jdk/7u60-b19/jdk-7u60-linux-.tar.gz?AuthParam=1497491097_5726d7e0ebd89a8675567ef8793c3443
Connecting to download.oracle.com (download.oracle.com)|65.158.47.80|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-06-14 21:42:57 ERROR 404: Not Found.
download failed
Oracle JDK 7 is NOT installed.
...
Setting up gconf2 (3.2.6-3ubuntu6) ...
Setting up libgnomevfs2-common (1:2.24.4-6.1ubuntu1) ...
Setting up libgnomevfs2-0:ppc64el (1:2.24.4-6.1ubuntu1) ...
Setting up libgnome2-common (2.32.1-5ubuntu1) ...
Setting up libgnome-2-0:ppc64el (2.32.1-5ubuntu1) ...
Processing triggers for libc-bin (2.23-0ubuntu7) ...
Errors were encountered while processing:
oracle-java7-installer
E: Sub-process /usr/bin/dpkg returned an error code (1)
그러나 이는 oracle-java7-installer의 error일 뿐, JDK 7 자체는 설치가 된 것이며 사용 가능합니다.
u0017496@sys-87548:~$ dpkg -l | grep jdk-7
ii openjdk-7-demo 7u95-2.6.4-3 ppc64el Java runtime based on OpenJDK (demos and examples)
ii openjdk-7-jdk:ppc64el 7u95-2.6.4-3 ppc64el OpenJDK Development Kit (JDK)
ii openjdk-7-jre:ppc64el 7u95-2.6.4-3 ppc64el OpenJDK Java runtime, using Hotspot JIT
ii openjdk-7-jre-headless:ppc64el 7u95-2.6.4-3 ppc64el OpenJDK Java runtime, using Hotspot JIT (headless)
u0017496@sys-87548:~$ dpkg -L openjdk-7-jdk | grep bin
/usr/lib/jvm/java-7-openjdk-ppc64el/bin
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/serialver
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/native2ascii
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jps
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/extcheck
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/javadoc
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jcmd
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jstatd
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jstack
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jsadebugd
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jarsigner
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/javah
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jmap
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/xjc
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jdb
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/javac
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jstat
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/idlj
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/wsimport
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jinfo
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/javap
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/appletviewer
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/rmic
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/schemagen
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jhat
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/wsgen
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jrunscript
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jar
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jconsole
이 JDK7을 이용하려면 위의 IBM java SDK처럼 PATH 환경 변수를 조정하여 사용할 수도 있고, 이건 openjdk이므로 'update-alternatives --config java' 명령으로 아래처럼 link를 바꿔줄 수도 있습니다.
u0017496@sys-87548:~$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
u0017496@sys-87548:~$ sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-8-openjdk-ppc64el/jre/bin/java 1081 auto mode
1 /usr/lib/jvm/java-7-openjdk-ppc64el/jre/bin/java 1071 manual mode
2 /usr/lib/jvm/java-8-openjdk-ppc64el/jre/bin/java 1081 manual mode
Press <enter> to keep the current choice[*], or type selection number: 1
update-alternatives: using /usr/lib/jvm/java-7-openjdk-ppc64el/jre/bin/java to provide /usr/bin/java (java) in manual mode
위에서처럼 1번을 택해서 JDK8 대신 JDK7을 택한 뒤, java 명령을 내리면 아래처럼 버전 7이 link되어 있음을 확인할 수 있습니다.
u0017496@sys-87548:~$ java -version
java version "1.7.0_95"
OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-3)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
1) IBM SDK 7 설치
https://developer.ibm.com/javasdk/downloads/sdk7r1/
Linux on Power Systems 64-bit LE
Installable package (InstallAnywhere as root)
(File name: ibm-java-ppc64le-sdk-7.1-4.5.bin, Size: 128MB)
u0017496@sys-87548:~$ chmod a+x ibm-java-ppc64le-sdk-7.1-4.5.bin
u0017496@sys-87548:~$ sudo ./ibm-java-ppc64le-sdk-7.1-4.5.bin
...
===============================================================================
Installation Complete
---------------------
Congratulations. IBM 64-bit SDK for Linux, Java Technology Edition, Version 7.1
has been successfully installed to:
/opt/ibm/java-ppc64le-71
PRESS <ENTER> TO EXIT THE INSTALLER:
설치 directory가 아래와 같으므로, user profile의 PATH의 맨 앞에 다음과 같이 설정해서 쓰시면 됩니다.
u0017496@sys-87548:~$ export PATH=/opt/ibm/java-ppc64le-71/bin:$PATH
u0017496@sys-87548:~$ which java
/opt/ibm/java-ppc64le-71/bin/java
u0017496@sys-87548:~$ java -version
java version "1.7.0"
Java(TM) SE Runtime Environment (build pxl6470_27sr4fp5-20170421_02(SR4 FP5))
IBM J9 VM (build 2.7, JRE 1.7.0 Linux ppc64le-64 Compressed References 20170406_343211 (JIT enabled, AOT enabled)
J9VM - R27_Java727_SR4_20170406_2107_B343211
JIT - tr.r13.java_20170406_343211
GC - R27_Java727_SR4_20170406_2107_B343211_CMPRSS
J9CL - 20170406_343211)
JCL - 20170420_01 based on Oracle jdk7u141-b11
2) openJDK 7 설치
Ubuntu 16에는 JDK7이 지원되지 않습니다만, 아래와 같이 PPA repository에서 제공되기는 합니다. 다만, 이 설치 과정 중에 Oracle installer를 download 받아서 그걸 이용하는 과정이 있는데, ppc64le는 Oracle과 친하지 않아 그 과정은 실패합니다. 그러나 그와는 무관하게 JDK7은 설치되고 사용할 수 있습니다.
u0017496@sys-87548:~$ sudo add-apt-repository ppa:openjdk-r/ppa
More info: https://launchpad.net/~openjdk-r/+archive/ubuntu/ppa
Press [ENTER] to continue or ctrl-c to cancel adding it
gpg: keyring `/tmp/tmpgyngeisu/secring.gpg' created
gpg: keyring `/tmp/tmpgyngeisu/pubring.gpg' created
gpg: requesting key 86F44E2A from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpgyngeisu/trustdb.gpg: trustdb created
gpg: key 86F44E2A: public key "Launchpad OpenJDK builds (all archs)" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK
u0017496@sys-87548:~$ sudo apt-get update
u0017496@sys-87548:~$ apt-cache pkgnames | grep openjdk-7
openjdk-7-jre-zero
openjdk-7-jre-lib
openjdk-7-source
oracle-jdk7-installer
openjdk-7-demo
openjdk-7-jre-headless
openjdk-7-dbg
openjdk-7-doc
openjdk-7-jdk
openjdk-7-jre
u0017496@sys-87548:~$ sudo apt-get install openjdk-7-jdk
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
at-spi2-core dbus-x11 gconf-service gconf-service-backend gconf2 gconf2-common libatk-bridge2.0-0
libatk-wrapper-java libatk-wrapper-java-jni libatspi2.0-0 libavahi-glib1 libbonobo2-0
libbonobo2-common libcanberra0 libgconf-2-4 libgnome-2-0 libgnome2-common libgnomevfs2-0
libgnomevfs2-common liborbit-2-0 libsctp1 libtdb1 libvorbisfile3 libxaw7 libxcb-shape0 libxft2
libxmu6 libxpm4 libxv1 libxxf86dga1 openjdk-7-jre openjdk-7-jre-headless sound-theme-freedesktop
x11-utils
Suggested packages:
gconf-defaults-service libbonobo2-bin libcanberra-gtk0 libcanberra-pulse desktop-base
libgnomevfs2-bin libgnomevfs2-extra gamin | fam gnome-mime-data lksctp-tools openjdk-7-demo
openjdk-7-source visualvm libnss-mdns sun-java6-fonts fonts-ipafont-gothic fonts-ipafont-mincho
ttf-wqy-microhei | ttf-wqy-zenhei fonts-indic mesa-utils
The following NEW packages will be installed:
at-spi2-core dbus-x11 gconf-service gconf-service-backend gconf2 gconf2-common libatk-bridge2.0-0
libatk-wrapper-java libatk-wrapper-java-jni libatspi2.0-0 libavahi-glib1 libbonobo2-0
libbonobo2-common libcanberra0 libgconf-2-4 libgnome-2-0 libgnome2-common libgnomevfs2-0
libgnomevfs2-common liborbit-2-0 libsctp1 libtdb1 libvorbisfile3 libxaw7 libxcb-shape0 libxft2
libxmu6 libxpm4 libxv1 libxxf86dga1 openjdk-7-jdk openjdk-7-jre openjdk-7-jre-headless
sound-theme-freedesktop x11-utils
0 upgraded, 35 newly installed, 0 to remove and 39 not upgraded.
1 not fully installed or removed.
Need to get 54.6 MB of archives.
After this operation, 91.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial/main ppc64el libbonobo2-common all 2.32.1-3 [34.7 kB]
Get:2 http://ppa.launchpad.net/openjdk-r/ppa/ubuntu xenial/main ppc64el openjdk-7-jre-headless ppc64el 7u95-2.6.4-3 [38.6 MB]
...
Processing triggers for hicolor-icon-theme (0.15-0ubuntu1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Setting up oracle-java7-installer (7u80+7u60arm-0~webupd8~1) ...
Please report to author unsupported platform 'ppc64le'.
Proceeding without web browser plugin support
Please report to author unsupported platform 'ppc64le'.
Proceeding without web browser plugin support
...
AuthParam=1497491097_5726d7e0ebd89a8675567ef8793c3443 [following]
--2017-06-14 21:42:57-- http://download.oracle.com/otn-pub/java/jdk/7u60-b19/jdk-7u60-linux-.tar.gz?AuthParam=1497491097_5726d7e0ebd89a8675567ef8793c3443
Connecting to download.oracle.com (download.oracle.com)|65.158.47.80|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-06-14 21:42:57 ERROR 404: Not Found.
download failed
Oracle JDK 7 is NOT installed.
...
Setting up gconf2 (3.2.6-3ubuntu6) ...
Setting up libgnomevfs2-common (1:2.24.4-6.1ubuntu1) ...
Setting up libgnomevfs2-0:ppc64el (1:2.24.4-6.1ubuntu1) ...
Setting up libgnome2-common (2.32.1-5ubuntu1) ...
Setting up libgnome-2-0:ppc64el (2.32.1-5ubuntu1) ...
Processing triggers for libc-bin (2.23-0ubuntu7) ...
Errors were encountered while processing:
oracle-java7-installer
E: Sub-process /usr/bin/dpkg returned an error code (1)
그러나 이는 oracle-java7-installer의 error일 뿐, JDK 7 자체는 설치가 된 것이며 사용 가능합니다.
u0017496@sys-87548:~$ dpkg -l | grep jdk-7
ii openjdk-7-demo 7u95-2.6.4-3 ppc64el Java runtime based on OpenJDK (demos and examples)
ii openjdk-7-jdk:ppc64el 7u95-2.6.4-3 ppc64el OpenJDK Development Kit (JDK)
ii openjdk-7-jre:ppc64el 7u95-2.6.4-3 ppc64el OpenJDK Java runtime, using Hotspot JIT
ii openjdk-7-jre-headless:ppc64el 7u95-2.6.4-3 ppc64el OpenJDK Java runtime, using Hotspot JIT (headless)
다음과 같이 JDK 명령어들도 잘 설치된 것을 보실 수 있습니다.
u0017496@sys-87548:~$ dpkg -L openjdk-7-jdk | grep bin
/usr/lib/jvm/java-7-openjdk-ppc64el/bin
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/serialver
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/native2ascii
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jps
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/extcheck
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/javadoc
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jcmd
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jstatd
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jstack
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jsadebugd
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jarsigner
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/javah
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jmap
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/xjc
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jdb
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/javac
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jstat
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/idlj
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/wsimport
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jinfo
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/javap
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/appletviewer
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/rmic
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/schemagen
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jhat
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/wsgen
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jrunscript
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jar
/usr/lib/jvm/java-7-openjdk-ppc64el/bin/jconsole
이 JDK7을 이용하려면 위의 IBM java SDK처럼 PATH 환경 변수를 조정하여 사용할 수도 있고, 이건 openjdk이므로 'update-alternatives --config java' 명령으로 아래처럼 link를 바꿔줄 수도 있습니다.
u0017496@sys-87548:~$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
u0017496@sys-87548:~$ sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-8-openjdk-ppc64el/jre/bin/java 1081 auto mode
1 /usr/lib/jvm/java-7-openjdk-ppc64el/jre/bin/java 1071 manual mode
2 /usr/lib/jvm/java-8-openjdk-ppc64el/jre/bin/java 1081 manual mode
Press <enter> to keep the current choice[*], or type selection number: 1
update-alternatives: using /usr/lib/jvm/java-7-openjdk-ppc64el/jre/bin/java to provide /usr/bin/java (java) in manual mode
위에서처럼 1번을 택해서 JDK8 대신 JDK7을 택한 뒤, java 명령을 내리면 아래처럼 버전 7이 link되어 있음을 확인할 수 있습니다.
u0017496@sys-87548:~$ java -version
java version "1.7.0_95"
OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-3)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
피드 구독하기:
글 (Atom)