/usr/local/lib64/python3.6/site-packages/torch/distributed/nn/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/distributed/nn/__pycache__/functional.cpython-36.pyc (9166B)
3
Eg @ s d dl Z d dlmZ d dljZejjfddZd ejjfddZ d ejjfddZ
ej j ejjfd d
Z
ejjfd d Zejjfd
dZej j ejjfddZG dd deZG dd deZG dd deZG dd deZG dd deZG dd d eZG d d d eZdS ) N)Functionc C s t j||| S )a
Broadcasts the tensor to the whole group.
``tensor`` must have the same number of elements in all processes
participating in the collective.
Arguments:
tensor (Tensor): Data to be sent if ``src`` is the rank of current
process.
src (int): Source rank.
group (ProcessGroup, optional): The process group to work on.
Returns:
Tensor: Received tensor from the broadcast op.
)
_Broadcastapply)tensorsrcgroup r K/usr/local/lib64/python3.6/site-packages/torch/distributed/nn/functional.py broadcast s r
c C s t j||| S )aT
Gathers a list of tensors in a single process.
Arguments:
tensor (Tensor): Input tensor.
dst (int, optional): Destination rank (default is 0).
group (ProcessGroup, optional): The process group to work on.
Returns:
tuple[Tensor]: List of appropriately-sized tensors with the gathered data.
)_Gatherr )r dstr r r r gather s r
c C s t j||f| S )a
Scatters a list of tensors to all processes in a group.
Each process will receive exactly one tensor and store its data in the
``tensor`` argument.
Arguments:
tensors (list[Tensor]): List of tensors to scatter on the source rank.
Receivers must pass ``None`.
src (int, optional): Source rank (default is 0).
group (ProcessGroup, optional): The process group to work on.
Returns:
Tensor: Output tensor from the scatter operation.
)_Scatterr )tensorsr r r r r scatter) s r c C s t j|||| S )a
Reduces the tensor data across all machines.
Only the process with rank ``dst`` is going to receive the final result.
Arguments:
tensor (Tensor): Input of the collective.
dst (int): Destination rank.
op (optional): One of the values from
``torch.distributed.ReduceOp``
enum. Specifies an operation used for element-wise reductions.
group (ProcessGroup, optional): The process group to work on.
Returns:
Tensor: Output of the collective.
)_Reducer )r r opr r r r reduce= s r c C s t j|| S )a
Gathers tensors from the whole group in a list.
Arguments:
tensor (Tensor): Tensor to be broadcast from current process.
group (ProcessGroup, optional): The process group to work on.
Returns:
tuple[Tensor]): Output of the collective.
)
_AllGatherr )r r r r r
all_gatherR s r c C s t j|f| S )ai
Each process scatters list of input tensors to all processes in a group and
return gathered list of tensors in output list.
Arguments:
tensors (list[Tensor]): List of tensors to scatter one per rank.
group (ProcessGroup, optional): The process group to work on.
Returns:
tuple[Tensor]): Output of the collective.
) _AlltoAllr )r r r r r
all_to_alla s
r c C s t j||| S )a*
Reduces the tensor data across all machines in such a way that all get
the final result.
After the call the returned tensor is going to be bitwise
identical in all processes.
Arguments:
tensor (Tensor): Input of the collective.
op (optional): One of the values from
``torch.distributed.ReduceOp``
enum. Specifies an operation used for element-wise reductions.
group (ProcessGroup, optional): The process group to work on.
Returns:
Tensor: Output of the collective
)
_AllReducer )r r r r r r
all_reduceq s r c @ s$ e Zd Zedd Zedd ZdS )r c C s2 || _ || _tj | _|j }tj|||d |S )N)r )r r distget_rankrankcloner
)ctxr r r r r r forward s
z_Broadcast.forwardc C s6 t j| jtjj| j|}| j| jkr,|j d d |fS )N) r r r r ReduceOpSUMr r Zzero_)r grad_outputgxr r r backward s z_Broadcast.backwardN)__name__
__module__ __qualname__ staticmethodr r$ r r r r r s r c @ s$ e Zd Zedd Zedd ZdS )r c sh || _ || _ fddttj|dD }tj|d|krNtj |||d ntj d ||d t|S )Nc s g | ]}t j qS r )torch
zeros_like).0i)r r r
s z#_Gather.forward..)r )r r ranger get_world_sizer r
tuple)r r r r Z tensor_listr )r r r s z_Gather.forwardc G s dt j| j| jf| f S )N)NN)r r r r )r grad_outputsr r r r$ s z_Gather.backwardN)r% r& r' r( r r$ r r r r r s r c @ s$ e Zd Zedd Zedd ZdS )r c sr || _ || _t fdd D s&ttj d }tj|d|kr\tj|t ||d ntj|d ||d |S )Nc 3 s" | ]}|j d j kV qdS )r N)size)r+ t)r r r s z#_Scatter.forward..r )r )
r r allAssertionErrorr) r* r r r list)r r r r outputr )r r r s z_Scatter.forwardc C s dt j| j| j| S )N)NN)r r r r )r r" r r r r$ s z_Scatter.backwardN)r% r& r' r( r r$ r r r r r s r c @ s$ e Zd Zedd Zedd ZdS )r c C s* || _ || _|j }tj||||d |S )N)r r )r r r r r )r r r r r r r r r s
z_Reduce.forwardc C s dt j| j| j|f S )N)NNN)r r r r )r r" r r r r$ s z_Reduce.backwardN)r% r& r' r( r r$ r r r r r s r c @ s$ e Zd Zedd Zedd ZdS )r c s< || _ fddttj|dD }tj| |d t|S )Nc s g | ]}t j qS r )r)
empty_like)r+ r, )r r r r- s z&_AllGather.forward..)r )r r. r r/ r r0 )r r r out_tensor_listr )r r r s z_AllGather.forwardc G s. t j| jf| }tjtj|dd}d |fS )Nr )Zdim)r r r r) sumstack)r r1 Zgxsr# r r r r$ s z_AllGather.backwardN)r% r& r' r( r r$ r r r r r s r c @ s$ e Zd Zedd Zedd ZdS )r c s || _ fddttj|dD }d gtj|d }tj|d}tj|dtjjkrxXttj|dD ].}d }||krt }tj || |||d qhW ntj
|t |d t |S )Nc s g | ]}t j | qS r )r) r9 )r+ r, )r r r r- s z%_AlltoAll.forward..)r ) r r. r r/ r get_backendZBackendZGLOOr7 r r r0 )r r r r: reqsZmy_rankr, to_sendr )r r r s z_AlltoAll.forwardc G s dt j| jf| S )N)N)r r r )r r1 r r r r$ s z_AlltoAll.backwardN)r% r& r' r( r r$ r r r r r s r c @ s$ e Zd Zedd Zedd ZdS )r c C s( || _ || _|j }tj|||d |S )N)r r )r r r r r )r r r r r r r r s
z_AllReduce.forwardc C s dt j| j| j|f S )N)NN)r r r r )r r" r r r r$ s z_AllReduce.backwardN)r% r& r' r( r r$ r r r r r s r )r) Ztorch.autogradr Ztorch.distributedZ distributedr r ZWORLDr
r
r r r! r r r r r r r r r r r r r r r s