caffe-ibm이 자랑하는 기능 중 하나인 DDL (Distributed Deep Learning)은 여러대의 GPU 서버에 들어있는 GPU들을 OpenMPI로 연결하여 하나의 큰 모델을 training할 수 있도록 해주는 기능입니다. 당연히 여러대의 GPU 서버를 연결하는 network의 latency와 bandwidth에 큰 영향을 받습니다. 여기서는 minsky1과 minsky2라는 hostname의 서버에, IP over Infiniband를 구성하고, 그 interface에 각각 ib1과 ib2라는 IP name을 /etc/hosts에 등록하여 별도의 고속 private network을 구성하여 DDL을 해봤습니다.
이 경우 4-GPU 서버가 2대이고 하나의 network으로 물려있으므로 그 topology를 알려주는 rank file은 아래와 같이 하면 됩니다.
$ cat 4x2x1.rf
rank 0=ib1 slot=0:0-3
rank 2=ib1 slot=0:4-7
rank 4=ib1 slot=1:0-3
rank 6=ib1 slot=1:4-7
rank 2=ib1 slot=0:4-7
rank 4=ib1 slot=1:0-3
rank 6=ib1 slot=1:4-7
rank 1=ib2 slot=0:0-3
rank 3=ib2 slot=0:4-7
rank 5=ib2 slot=1:0-3
rank 7=ib2 slot=1:4-7
rank 3=ib2 slot=0:4-7
rank 5=ib2 slot=1:0-3
rank 7=ib2 slot=1:4-7
그런데, 막상 돌려보니 아래와 같이 error 메시지가 나옵니다.
$ mpirun -x PATH -x LD_LIBRARY_PATH -n 8 -rf 4x2x1.rf caffe train
--solver=solver.prototxt -gpu 0 -ddl "-mode b:4x1x1 -dev_sync
1"
--------------------------------------------------------------------------
Failed to create a completion queue (CQ):
--------------------------------------------------------------------------
Failed to create a completion queue (CQ):
Hostname: minsky1
Requested CQE: 16384
Error: Cannot allocate memory
Requested CQE: 16384
Error: Cannot allocate memory
Check the CQE
attribute.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Open MPI has detected that there are UD-capable Verbs devices on your
system, but none of them were able to be setup properly. This may
indicate a problem on this system.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Open MPI has detected that there are UD-capable Verbs devices on your
system, but none of them were able to be setup properly. This may
indicate a problem on this system.
You job will continue, but Open
MPI will ignore the "ud" oob component
in this run.
in this run.
Hostname:
minsky1
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Failed to create a completion queue (CQ):
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Failed to create a completion queue (CQ):
Hostname: minsky2
Requested CQE: 16384
Error: Cannot allocate memory
Requested CQE: 16384
Error: Cannot allocate memory
Check the CQE
attribute.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Open MPI has detected that there are UD-capable Verbs devices on your
system, but none of them were able to be setup properly. This may
indicate a problem on this system.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Open MPI has detected that there are UD-capable Verbs devices on your
system, but none of them were able to be setup properly. This may
indicate a problem on this system.
You job will continue, but Open
MPI will ignore the "ud" oob component
in this run.
in this run.
Hostname:
minsky2
--------------------------------------------------------------------------
--------------------------------------------------------------------------
The rankfile that was used claimed that a host was either not
allocated or oversubscribed its slots. Please review your rank-slot
assignments and your host allocation to ensure a proper match. Also,
some systems may require using full hostnames, such as
"host1.example.com" (instead of just plain "host1").
--------------------------------------------------------------------------
--------------------------------------------------------------------------
The rankfile that was used claimed that a host was either not
allocated or oversubscribed its slots. Please review your rank-slot
assignments and your host allocation to ensure a proper match. Also,
some systems may require using full hostnames, such as
"host1.example.com" (instead of just plain "host1").
Host:
minsky1
--------------------------------------------------------------------------
--------------------------------------------------------------------------
이건 2가지 error입니다. 하나는 "Cannot allocate memory"이고, 다른 하나는 "either not
allocated or oversubscribed its slots" 인데, 각각 다른 원인에 의한 것입니다.
1) "Cannot allocate memory" error
이건 limits 값 때문입니다. 다음과 같이 ulimit 값을 보면 max locked memory가 기본으로는 64로 되어 있습니다.
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15880
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 15880
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
이걸 풀어주기 위해서는 아래와 같이 limits.conf의 맨 끝에 해당 user에 대한 limit를 풀어준 뒤, 반드시 re-login을 하셔야 합니다. 이걸로 해결됩니다.
$ sudo vi /etc/security/limits.conf
...
user1 soft memlock -1
user1 hard memlock -1
2) "either not allocated or oversubscribed its slots" error
이건 정말 제가 예상 못 했던 것인데, 구글링을 해보니 뜻 밖에도 mpirun 등의 MPI command는 IP name이 아니라 hostname에 민감한 것 같습니다. 즉, 저 GPU 서버들의 hostname이자 ethernet interface의 IP name이 minsky1 (10.1.1.1), minsky2 (10.1.1.2)이고, infiniband interface의 IP name이 ib1 (9.1.1.1), ib2 (9.1.1.2)인데, 이렇게 hostname과 rank file 안에 들어가는 IP name이 각각 다르면 안되나 봅니다.
이 경우 다음과 같이 두 서버의 hostname을 IB interface의 이름인 ib1, ib2로 각각 바꿔주면 해결이 됩니다.
$ hostnamectl set-hostname ib1
$ hostnamectl set-hostname ib2
댓글 없음:
댓글 쓰기