/usr/local/lib64/python3.6/site-packages/torch/nn/parallel/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/nn/parallel/__pycache__/comm.cpython-36.pyc (10466B)
3
Eg$* @ s d dl Z d dlZd dlmZ d dlmZmZmZmZm Z m
Z
d dl m Z dddddZ
dd d
Zdd d Zdd
dZdddddZdddddZdS ) N)nccl)
_take_tensors_flatten_dense_tensors_unflatten_dense_tensors_reorder_tensors_as_get_device_index_handle_complex)List)outc C s^ t | } |dk|dkA s(tdj|||dk rLdd |D }tjj| |S tjj| |S dS )a Broadcasts a tensor to specified GPU devices.
Args:
tensor (Tensor): tensor to broadcast. Can be on CPU or GPU.
devices (Iterable[torch.device, str or int], optional): an iterable of
GPU devices, among which to broadcast.
out (Sequence[Tensor], optional, keyword-only): the GPU tensors to
store output results.
.. note::
Exactly one of :attr:`devices` and :attr:`out` must be specified.
Returns:
- If :attr:`devices` is specified,
a tuple containing copies of :attr:`tensor`, placed on
:attr:`devices`.
- If :attr:`out` is specified,
a tuple containing :attr:`out` tensors, each containing a copy of
:attr:`tensor`.
NzSExactly one of 'devices' and 'out' must be specified, but got devices={} and out={}c S s g | ] }t |qS )r ).0dr r B/usr/local/lib64/python3.6/site-packages/torch/nn/parallel/comm.py
# s z broadcast..)r RuntimeErrorformattorch_CZ
_broadcastZ_broadcast_out)tensordevicesr
r r r broadcast s
r c C s, dd |D }dd | D } t jj| ||S )a/ Broadcasts a sequence tensors to the specified GPUs.
Small tensors are first coalesced into a buffer to reduce the number
of synchronizations.
Args:
tensors (sequence): tensors to broadcast. Must be on the same device,
either CPU or GPU.
devices (Iterable[torch.device, str or int]): an iterable of GPU
devices, among which to broadcast.
buffer_size (int): maximum size of the buffer used for coalescing
Returns:
A tuple containing copies of :attr:`tensor`, placed on :attr:`devices`.
c S s g | ] }t |qS r )r )r r
r r r r 8 s z'broadcast_coalesced..c S s g | ] }t |qS r )r )r tr r r r 9 s )r r Z_broadcast_coalesced)tensorsr buffer_sizer r r broadcast_coalesced) s r c sb t |dd}| d j }d xt| D ]v\}}|jjdksBtd|j |krR| |j |kr&djdd |j D }djd
d |D }td j |||q&W dkrt
d t | d
kr| d S t j
| rtj| }t j| | d nptj| jj|} fddt| D } | | d j|dd }x*| d
d D ]}
|j|
j|dd q@W |S )a Sums tensors from multiple GPUs.
All inputs should have matching shapes, dtype, and layout. The output tensor
will be of the same shape, dtype, and layout.
Args:
inputs (Iterable[Tensor]): an iterable of tensors to add.
destination (int, optional): a device on which the output will be
placed (default: current device).
Returns:
A tensor containing an elementwise sum of all inputs, placed on the
:attr:`destination` device.
T)optionalr Ncpuz+reduce_add expects all inputs to be on GPUsxc s s | ]}t |V qd S )N)str)r r r r r T s z reduce_add..c s s | ]}t |V qd S )N)r )r r r r r r U s z2input {} has invalid size: got {}, but expected {}zLreduce_add expects destination to be on the same GPU with one of the tensors )outputrootc s g | ]\}}| kr|qS r r )r ir )
root_indexr r r c s z reduce_add..)deviceZ non_blocking)r size enumerater& typeAssertionErrorZ
get_devicejoin
ValueErrorr r lenr Z is_availabler Z
empty_likereduceZtoZadd_) inputs destinationZ
input_sizer$ inpgotexpectedresultZdestination_deviceZnonroototherr )r% r
reduce_add= s2
r6 c s dd | D }g }g }xt | D ]|}tdd |D rZt||}|j| |j|d q x.t ||D ] \}} |j| jr| j n| qfW |j|d d q W fdd|D }
xLt |
D ]@} dd | D } t| |}
x"t|
| d D ]} |j| j qW qW tt ||S )
a] Sums tensors from multiple GPUs.
Small tensors are first coalesced into a buffer to reduce the number
of synchronizations.
Args:
inputs (Iterable[Iterable[Tensor]]): iterable of iterables that
contain tensors from a single device.
destination (int, optional): a device on which the output will be
placed (default: current device).
buffer_size (int): maximum size of the buffer used for coalescing
Returns:
A tuple of tensors containing an elementwise sum of each group of
inputs, placed on the ``destination`` device.
c S s g | ]}g qS r r )r _r r r r ~ s z(reduce_add_coalesced..c s s | ] }|j V qd S )N) is_sparse)r r r r r r s z'reduce_add_coalesced..r r! c s g | ]}t | qS r )r )r r )r r r r s c S s g | ] }t |qS r )r )r chunkr r r r s )
zipallr6 appendr8 Zto_denser datatupler )r/ r0 r Z
dense_tensorsr" Z ref_orderZtensor_at_gpusr4 Zcollr ZitrschunksZ flat_tensorsZ flat_resultr )r r reduce_add_coalescedk s$
rA c C s| t | } |dkr6dd |D }ttjj| ||||S |dk rLtdj||dk rbtdj|ttjj| |||S dS )a1 Scatters tensor across multiple GPUs.
Args:
tensor (Tensor): tensor to scatter. Can be on CPU or GPU.
devices (Iterable[torch.device, str or int], optional): an iterable of
GPU devices, among which to scatter.
chunk_sizes (Iterable[int], optional): sizes of chunks to be placed on
each device. It should match :attr:`devices` in length and sums to
``tensor.size(dim)``. If not specified, :attr:`tensor` will be divided
into equal chunks.
dim (int, optional): A dimension along which to chunk :attr:`tensor`.
Default: ``0``.
streams (Iterable[Stream], optional): an iterable of Streams, among
which to execute the scatter. If not specified, the default stream will
be utilized.
out (Sequence[Tensor], optional, keyword-only): the GPU tensors to
store output results. Sizes of these tensors must match that of
:attr:`tensor`, except for :attr:`dim`, where the total size must
sum to ``tensor.size(dim)``.
.. note::
Exactly one of :attr:`devices` and :attr:`out` must be specified. When
:attr:`out` is specified, :attr:`chunk_sizes` must not be specified and
will be inferred from sizes of :attr:`out`.
Returns:
- If :attr:`devices` is specified,
a tuple containing chunks of :attr:`tensor`, placed on
:attr:`devices`.
- If :attr:`out` is specified,
a tuple containing :attr:`out` tensors, each containing a chunk of
:attr:`tensor`.
Nc S s g | ] }t |qS r )r )r r
r r r r s zscatter..zK'devices' must not be specified when 'out' is specified, but got devices={}zS'chunk_sizes' must not be specified when 'out' is specified, but got chunk_sizes={})r r? r r Z_scatterr r Z _scatter_out)r r Z chunk_sizesdimstreamsr
r r r scatter s "rD c C sp dd | D } |dkrF|d kr(t jd t|ddd}tjj| ||S |dk r\tdj|tjj| ||S dS )
a Gathers tensors from multiple GPU devices.
Args:
tensors (Iterable[Tensor]): an iterable of tensors to gather.
Tensor sizes in all dimensions other than :attr:`dim` have to match.
dim (int, optional): a dimension along which the tensors will be
concatenated. Default: ``0``.
destination (torch.device, str, or int, optional): the output device.
Can be CPU or CUDA. Default: the current CUDA device.
out (Tensor, optional, keyword-only): the tensor to store gather result.
Its sizes must match those of :attr:`tensors`, except for :attr:`dim`,
where the size must equal ``sum(tensor.size(dim) for tensor in tensors)``.
Can be on CPU or CUDA.
.. note::
:attr:`destination` must not be specified when :attr:`out` is specified.
Returns:
- If :attr:`destination` is specified,
a tensor located on :attr:`destination` device, that is a result of
concatenating :attr:`tensors` along :attr:`dim`.
- If :attr:`out` is specified,
the :attr:`out` tensor, now containing results of concatenating
:attr:`tensors` along :attr:`dim`.
c S s g | ] }t |qS r )r )r r r r r r s zgather..Nr! zjUsing -1 to represent CPU tensor is deprecated. Please use a device object or string instead, e.g., "cpu".T)Z allow_cpur zS'destination' must not be specified when 'out' is specified, but got destination={}r: ) warningswarnr r r Z_gatherr r Z _gather_out)r rB r0 r
r r r gather s rG )N)r )N)Nr )NNr N)r N)rE r Z
torch.cudar Z torch._utilsr r r r r r typingr r r r6 rA rD rG r r r r s !
.
-2