/usr/local/lib64/python3.6/site-packages/caffe2/python/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/caffe2/python/__pycache__/muji.cpython-36.pyc (5566B)
3
Eg @ sj d Z ddlZddlmZ ddlmZ dd Zdd Zdd
d Z d d
Z
dd Z dd Z dd Z
dd ZdS )aJ muji.py does multi-gpu training for caffe2 with no need to change the c++
side code. Everything is defined on the computation graph level.
We support the following use cases:
- 2 gpus, where peer access is enabled between them.
- 4 gpus, where peer access are enabled between all of them.
- 4 gpus, where peer access are enabled in two groups,
between {1, 2} and {3, 4}
- 8 gpus, where peer access are enabled in two groups,
between {1, 2, 3, 4} and {5, 6, 7, 8}.
If above cases are not satisfied, a fallback function which does not rely on
peer access will be called.
N)
caffe2_pb2) workspacec C s t j }tj|_| |_|S )zVA utility function that returns a device option protobuf of the
specified gpu id.
)r DeviceOptionr Z
GpuDeviceType device_typeZ device_id)Zgpu_id
device_option r >/usr/local/lib64/python3.6/site-packages/caffe2/python/muji.pyOnGPU s r c C s t j } t j| _| S )N)r r ZCPUr )r r r r OnCPU! s r
_reducedc C s |dkrt tt|}t|t|kr@tdt|t|f tj }t|dkr|jd dkrtj|ddddf rt | |||S t|dkr|jd dkrtj|ddddf rt
| |||S t|dkr0|jd dkr0tj|ddddf r0tj|ddddf r0t | |||S t|dkrx|jd dkrxtj|ddddf rxt | |||S t
| |||S dS )zThe general Allreduce interface that reroutes the function calls.
CPUs and AMD GPUs are not supported because
GetGpuPeerAccessPattern is called to get gpu peer access pattern.
Nz6gpu_indices length and blobs length mismatch: %d vs %d r )listrangelen RuntimeErrorr ZGetGpuPeerAccessPatternshapenpall
Allreduce2
Allreduce4Allreduce4Group2
Allreduce8AllreduceFallback)netblobs
reduced_affix gpu_indicespatternr r r Allreduce' s 44V:r c
C sL |\}}|\}}| j ||g|| t|d}|jg || t|d} || fS )zUAllreduce for 2 gpus.
Algorithm: 0r <- 0 + 1, 1r <- 0r, where r means "reduced"
)r )Addr Copy)
r r r r abgpu_agpu_b a_reduced b_reducedr r r r @ s r c C s |\}}}}|\}} }
} | j ||gt|| t|d} | j ||gt|| t|
d}
| j |
| t|d} | jg |
t|
d}
| jg t|| t| d}|
jg t|| t| d}| ||
|fS )zAllreduce for 4 gpus.
Algorithm: 2 level reduction.
0r <- 0 + 1, 2r <- 2 + 3
0r <- 0r + 2r
2r <- 0r,
1r <- 0r, 3r <- 2r
)r )r! strr r" )r r r r r# r$ cdr% r& gpu_cgpu_dr' c_reducedr( d_reducedr r r r P s*
r c C s |\}}}}|\}} }
} | j ||gt|| t|d} | j ||gt|| t|
d}
|
jg t|
d t|d}| j || t|d} | jg |
t|
d}
| jg t|| t| d}|
jg t|| t| d}| ||
|fS )zAllreduce for 4 gpus where peer access are enabled in {0,1} and {2,3}
Algorithm: 2 level reduction.
0r <- 0 + 1, 2r <- 2 + 3
0r <- 0r + 2r
2r <- 0r,
1r <- 0r, 3r <- 2r
)r _copy)r! r) r r" )r r r r r# r$ r* r+ r% r& r, r- r' r. Zc_reduced_copyr( r/ r r r r x s2
r c C sz dgd }x@d
D ]8}| j || ||d g|| | t|| d||< qW xDdD ]<}| j || ||d gt|| | t|| d||< qRW |d jg t|d d t|d d}|d j ||d t|d d|d< |d jg |d t|d d|d< x6dD ].}||d jg || t|| d||< qW x:dD ]2}||d jg || | t|| d||< q@W |S )zAllreduce for 8 gpus.
Algorithm: 3 level reduction.
0r <- 0 + 1, 2r <- 2 + 3, 4r <- 4 + 5, 6r <- 6 + 7
0r <- 0r + 2r, 4r <- 4r + 6r
0r <- 0r + 4r
4r <- 0r
2r <- 0r, 6r <- 4r
1r <- 0r, 3r <- 2r, 5r <- 4r, 7r <- 6r
Nr r r r
)r r0 )r r r
r1 )r r
)r r1 )r2 r3 r4 r5 )r! r r) r" )r r r r reducediZreduced_4_copyr r r r sD
r c C s dgt | }|dkr@| j|d |d | t|d d|d< n |d |d< |d d }xZtdt |D ]H}| j|| |t|d d}| j||d g|d t|d d|d< qhW x>tdt |D ],}| j|d || | t|| d||< qW |S )zA fallback option for Allreduce with no assumption on p2p.
Algorithm: a flat operation on gpu 0
0r <- 0
0r <- 0r + i for i in gpu_indices[1:]
ir <- 0r for i in gpu_indices[1:]
N r )r Z
_temp_copyr2 )r r" r r r! )r r r r r6 Z temp_namer7 tempr r r r s.
r )r N)__doc__Znumpyr Z caffe2.protor Z
caffe2.pythonr r r
r r r r r r r r r r
s
(.<