/usr/local/lib64/python3.6/site-packages/torch/nn/parallel/__pycache__/
/usr/local/lib64/python3.6/site-packages/torch/nn/parallel/__pycache__
NameSizeModeActions
comm.cpython-36.pyc104660644editdlrm
data_parallel.cpython-36.pyc96340644editdlrm
distributed.cpython-36.pyc563480644editdlrm
parallel_apply.cpython-36.pyc29420644editdlrm
replicate.cpython-36.pyc45100644editdlrm
scatter_gather.cpython-36.pyc34900644editdlrm
_functions.cpython-36.pyc57950644editdlrm
__init__.cpython-36.pyc7760644editdlrm
Edit: /usr/local/lib64/python3.6/site-packages/torch/nn/parallel/__pycache__/distributed.cpython-36.pyc (56348B)
3 Eg4@sHddlZddlZddlZddlZddlZddlZddlmZddlZddl j Z ddl m Z mZddlmZmZmZddlmZmZdZe jrddlmZmZej jjrdZdd lmZdd lmZd d l m!Z!d dl"m#Z#d dl$m%Z%m&Z&m'Z'ddZ(ddZ)ddZ*ddZ+Gddde Z,GdddeZ-Gddde!eZ.dS)N)contextmanager)FunctionVariable)JoinJoinableJoinHook) tree_flattentree_unflattenF)ReduceOp_get_default_groupT)RRef)_get_device_index)Module) _get_stream)gather is_namedtuplescatter_kwargscCs:to t|t}|r$t|j\}}n t|\}}|||fS)N) RPC_AVAILABLE isinstancer r local_value)outputoutput_is_rrefoutput_tensor_listtreespecrI/usr/local/lib64/python3.6/site-packages/torch/nn/parallel/distributed.py_tree_flatten_with_rref!s  rcCst||}|rt|}|S)N)r r )rrrrrr_tree_unflatten_with_rref,s rcCsttr"t|tr"|jr"t|jSt|tjr4|gSt|tt frRt j t t|St|t rpt j t t|jSgS)zI Recursively find all tensors contained in the specified object. )rrr Zis_owner _find_tensorsrtorchTensorlisttuple itertoolschainmapdictvalues)objrrrr 3s   r c/Csddddddddd d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/g/}d0}x4|D],}|tjkrtj|nd1}|d2||f7}qlWt|dS)3NZRANKZ LOCAL_RANKZ WORLD_SIZEZ MASTER_PORTZ MASTER_ADDRZCUDA_VISIBLE_DEVICESZGLOO_SOCKET_IFNAMEZGLOO_DEVICE_TRANSPORTZNCCL_SOCKET_IFNAMEZNCCL_BLOCKING_WAITZ NCCL_DEBUGZNCCL_DEBUG_SUBSYSZNCCL_IB_DISABLEZNCCL_P2P_DISABLEZNCCL_P2P_LEVELZNCCL_SHM_DISABLEZNCCL_SOCKET_NTHREADSZNCCL_NSOCKS_PERTHREADZ NCCL_BUFFSIZEZ NCCL_NTHREADSZ NCCL_RINGSZNCCL_MAX_NCHANNELSZNCCL_MIN_NCHANNELSZNCCL_CHECKS_DISABLEZNCCL_CHECK_POINTERSZNCCL_LAUNCH_MODEZ NCCL_IB_HCAZNCCL_IB_TIMEOUTZNCCL_IB_RETRY_CNTZNCCL_IB_GID_INDEXZ NCCL_IB_SLZ NCCL_IB_TCZNCCL_IB_AR_THRESHOLDZNCCL_IB_CUDA_SUPPORTZNCCL_NET_GDR_LEVELZNCCL_NET_GDR_READZNCCL_SINGLE_RING_THRESHOLDZNCCL_LL_THRESHOLDZNCCL_TREE_THRESHOLDZ NCCL_ALGOZ NCCL_PROTOZNCCL_IGNORE_CPU_AFFINITYZNCCL_DEBUG_FILEZNCCL_COLLNET_ENABLEZNCCL_TOPO_FILEZNCCL_TOPO_DUMP_FILEZNCCL_ASYNC_ERROR_HANDLINGzN/Az env:%s=%s )osenvironprint)Zrelevant_env_varsZformatted_outputvarvaluerrr_dump_DDP_relevant_env_varsFsh r1c@s$eZdZeddZeddZdS)_DDPSinkcGs|jd||_||_|S)NF)Zset_materialize_gradsreducer state_dict)ctxr3r4inputsrrrforwards z_DDPSink.forwardcGs6|j}|jdr.|jddkr.tjj|jjd|S)N static_graphnum_iterationsr)NN)r4rZ_execution_engineZqueue_callbackr3Z_delay_all_reduce)r5Z grad_outputsr4rrrbackwardsz_DDPSink.backwardN)__name__ __module__ __qualname__ staticmethodr7r:rrrrr2s r2cs2eZdZfddZddZedddZZS) _DDPJoinHookcs8t|tstd|jj||_||j_tjdS)z; Sets config variables for internal usage. zQDDP join hook requires passing in a DistributedDataParallel instance as the stateN) rDistributedDataParallelAssertionErrorloggerZ_set_uneven_input_joinddp_divide_by_initial_world_sizesuper__init__)selfrCdivide_by_initial_world_size) __class__rrrFs   z_DDPJoinHook.__init__cCsr|j}|jj|j|jdd}|j|jdjdk}||_|sNdS|j |j rd|j |jj dS)zq Shadows the DDP collective communication operations in the forward and backward passes. T)is_joined_rankrN) rCr3_rebuild_buckets_check_and_sync_module_buffers)_check_global_requires_backward_grad_syncwaitresultitemrequire_forward_param_sync_match_all_reduce_for_bwd_passfind_unused_parameters_match_unused_params_allreduceZ_push_all_rebuilt_params)rGrCworkZshould_sync_backwardsrrr main_hooks  z_DDPJoinHook.main_hook)is_last_joinercCs|jj|dS)zj Syncs the final model to ensure that the model is the same across all processes. N)rC_sync_final_model)rGrWrrr post_hooksz_DDPJoinHook.post_hook)r;r<r=rFrVboolrY __classcell__rr)rIrr?s #r?c seZdZdZd[fdd Zd\d d Zd d Zd dZddZfddZ ddZ ddZ ddZ d]ddZ ddZeddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd^fd)d* Zd+d,Zd-d.Zd/d0Zd1d2Zd3d4Zd5d6Zd_eeed7d8d9Zd:d;Ze dd?Z"e#e$d@dAdBZ%dCdDZ&d`dEdFZ'dGdHZ(dIdJZ)dKdLZ*dMdNZ+dOdPZ,e dQdRZ-e.dSdTZ/dUdVZ0dWdXZ1dYdZZ2Z3S)ar@a93Implements distributed data parallelism that is based on ``torch.distributed`` package at the module level. This container parallelizes the application of the given module by splitting the input across the specified devices by chunking in the batch dimension. The module is replicated on each machine and each device, and each such replica handles a portion of the input. During the backwards pass, gradients from each node are averaged. The batch size should be larger than the number of GPUs used locally. See also: :ref:`distributed-basics` and :ref:`cuda-nn-ddp-instead`. The same constraints on input as in :class:`torch.nn.DataParallel` apply. Creation of this class requires that ``torch.distributed`` to be already initialized, by calling :func:`torch.distributed.init_process_group`. ``DistributedDataParallel`` is proven to be significantly faster than :class:`torch.nn.DataParallel` for single-node multi-GPU data parallel training. To use ``DistributedDataParallel`` on a host with N GPUs, you should spawn up ``N`` processes, ensuring that each process exclusively works on a single GPU from 0 to N-1. This can be done by either setting ``CUDA_VISIBLE_DEVICES`` for every process or by calling: >>> torch.cuda.set_device(i) where i is from 0 to N-1. In each process, you should refer the following to construct this module: >>> torch.distributed.init_process_group( >>> backend='nccl', world_size=N, init_method='...' >>> ) >>> model = DistributedDataParallel(model, device_ids=[i], output_device=i) In order to spawn up multiple processes per node, you can use either ``torch.distributed.launch`` or ``torch.multiprocessing.spawn``. .. note:: Please refer to `PyTorch Distributed Overview `__ for a brief introduction to all features related to distributed training. .. note:: ``DistributedDataParallel`` can be used in conjunction with :class:`torch.distributed.optim.ZeroRedundancyOptimizer` to reduce per-rank optimizer states memory footprint. Please refer to `ZeroRedundancyOptimizer recipe `__ for more details. .. note:: ``nccl`` backend is currently the fastest and highly recommended backend when using GPUs. This applies to both single-node and multi-node distributed training. .. note:: This module also supports mixed-precision distributed training. This means that your model can have different types of parameters such as mixed types of ``fp16`` and ``fp32``, the gradient reduction on these mixed types of parameters will just work fine. .. note:: If you use ``torch.save`` on one process to checkpoint the module, and ``torch.load`` on some other processes to recover it, make sure that ``map_location`` is configured properly for every process. Without ``map_location``, ``torch.load`` would recover the module to devices where the module was saved from. .. note:: When a model is trained on ``M`` nodes with ``batch=N``, the gradient will be ``M`` times smaller when compared to the same model trained on a single node with ``batch=M*N`` if the loss is summed (NOT averaged as usual) across instances in a batch (because the gradients between different nodes are averaged). You should take this into consideration when you want to obtain a mathematically equivalent training process compared to the local training counterpart. But in most cases, you can just treat a DistributedDataParallel wrapped model, a DataParallel wrapped model and an ordinary model on a single GPU as the same (E.g. using the same learning rate for equivalent batch size). .. note:: Parameters are never broadcast between processes. The module performs an all-reduce step on gradients and assumes that they will be modified by the optimizer in all processes in the same way. Buffers (e.g. BatchNorm stats) are broadcast from the module in process of rank 0, to all other replicas in the system in every iteration. .. note:: If you are using DistributedDataParallel in conjunction with the :ref:`distributed-rpc-framework`, you should always use :meth:`torch.distributed.autograd.backward` to compute gradients and :class:`torch.distributed.optim.DistributedOptimizer` for optimizing parameters. Example:: >>> import torch.distributed.autograd as dist_autograd >>> from torch.nn.parallel import DistributedDataParallel as DDP >>> from torch import optim >>> from torch.distributed.optim import DistributedOptimizer >>> from torch.distributed.rpc import RRef >>> >>> t1 = torch.rand((3, 3), requires_grad=True) >>> t2 = torch.rand((3, 3), requires_grad=True) >>> rref = rpc.remote("worker1", torch.add, args=(t1, t2)) >>> ddp_model = DDP(my_model) >>> >>> # Setup optimizer >>> optimizer_params = [rref] >>> for param in ddp_model.parameters(): >>> optimizer_params.append(RRef(param)) >>> >>> dist_optim = DistributedOptimizer( >>> optim.SGD, >>> optimizer_params, >>> lr=0.05, >>> ) >>> >>> with dist_autograd.context() as context_id: >>> pred = ddp_model(rref.to_here()) >>> loss = loss_func(pred, loss) >>> dist_autograd.backward(context_id, loss) >>> dist_optim.step() .. note:: To let a non-DDP model load a state dict from a DDP model, :meth:`~torch.nn.modules.utils.consume_prefix_in_state_dict_if_present` needs to be applied to strip the prefix "module." in the DDP state dict before loading. .. warning:: Constructor, forward method, and differentiation of the output (or a function of the output of this module) are distributed synchronization points. Take that into account in case different processes might be executing different code. .. warning:: This module assumes all parameters are registered in the model by the time it is created. No parameters should be added nor removed later. Same applies to buffers. .. warning:: This module assumes all parameters are registered in the model of each distributed processes are in the same order. The module itself will conduct gradient ``allreduce`` following the reverse order of the registered parameters of the model. In other words, it is users' responsibility to ensure that each distributed process has the exact same model and thus the exact same parameter registration order. .. warning:: This module allows parameters with non-rowmajor-contiguous strides. For example, your model may contain some parameters whose :class:`torch.memory_format` is ``torch.contiguous_format`` and others whose format is ``torch.channels_last``. However, corresponding parameters in different processes must have the same strides. .. warning:: This module doesn't work with :func:`torch.autograd.grad` (i.e. it will only work if gradients are to be accumulated in ``.grad`` attributes of parameters). .. warning:: If you plan on using this module with a ``nccl`` backend or a ``gloo`` backend (that uses Infiniband), together with a DataLoader that uses multiple workers, please change the multiprocessing start method to ``forkserver`` (Python 3 only) or ``spawn``. Unfortunately Gloo (that uses Infiniband) and NCCL2 are not fork safe, and you will likely experience deadlocks if you don't change this setting. .. warning:: Forward and backward hooks defined on :attr:`module` and its submodules won't be invoked anymore, unless the hooks are initialized in the :meth:`forward` method. .. warning:: You should never try to change your model's parameters after wrapping up your model with ``DistributedDataParallel``. Because, when wrapping up your model with ``DistributedDataParallel``, the constructor of ``DistributedDataParallel`` will register the additional gradient reduction functions on all the parameters of the model itself at the time of construction. If you change the model's parameters afterwards, gradient redunction functions no longer match the correct set of parameters. .. warning:: Using ``DistributedDataParallel`` in conjunction with the :ref:`distributed-rpc-framework` is experimental and subject to change. Args: module (Module): module to be parallelized device_ids (list of int or torch.device): CUDA devices. 1) For single-device modules, ``device_ids`` can contain exactly one device id, which represents the only CUDA device where the input module corresponding to this process resides. Alternatively, ``device_ids`` can also be ``None``. 2) For multi-device modules and CPU modules, ``device_ids`` must be ``None``. When ``device_ids`` is ``None`` for both cases, both the input data for the forward pass and the actual module must be placed on the correct device. (default: ``None``) output_device (int or torch.device): Device location of output for single-device CUDA modules. For multi-device modules and CPU modules, it must be ``None``, and the module itself dictates the output location. (default: ``device_ids[0]`` for single-device modules) broadcast_buffers (bool): Flag that enables syncing (broadcasting) buffers of the module at beginning of the ``forward`` function. (default: ``True``) process_group: The process group to be used for distributed data all-reduction. If ``None``, the default process group, which is created by :func:`torch.distributed.init_process_group`, will be used. (default: ``None``) bucket_cap_mb: ``DistributedDataParallel`` will bucket parameters into multiple buckets so that gradient reduction of each bucket can potentially overlap with backward computation. :attr:`bucket_cap_mb` controls the bucket size in MegaBytes (MB). (default: 25) find_unused_parameters (bool): Traverse the autograd graph from all tensors contained in the return value of the wrapped module's ``forward`` function. Parameters that don't receive gradients as part of this graph are preemptively marked as being ready to be reduced. In addition, parameters that may have been used in the wrapped module's ``forward`` function but were not part of loss computation and thus would also not receive gradients are preemptively marked as ready to be reduced. (default: ``False``) check_reduction: This argument is deprecated. gradient_as_bucket_view (bool): When set to ``True``, gradients will be views pointing to different offsets of ``allreduce`` communication buckets. This can reduce peak memory usage, where the saved memory size will be equal to the total gradients size. Moreover, it avoids the overhead of copying between gradients and ``allreduce`` communication buckets. When gradients are views, ``detach_()`` cannot be called on the gradients. If hitting such errors, please fix it by referring to the :meth:`~torch.optim.Optimizer.zero_grad` function in ``torch/optim/optimizer.py`` as a solution. Attributes: module (Module): the module to be parallelized. Example:: >>> torch.distributed.init_process_group(backend='nccl', world_size=4, init_method='...') >>> net = torch.nn.parallel.DistributedDataParallel(model, pg) NrTFc stt|jtj|d|_tdd|jDs@|jtd|dk r`t |dkr`|jt dt dd|jDdk|_ dd|jD} t | dkr|jt d j | t | d |_|dkst |d ks|jd ks|j r|s|r|jt d j ||d d|jDd|_d|_n.dd|D|_|dkr>|d }t|d|_|dkr^t|_n||_d|_||_||_t |jjd j|_||_||_d|_d|_| |_t|dr|j|_ng|_| rt j!dx0|jD]$} t"| t#j$j%j&r|jtdqWt'd|_(t'|dd|_)t*j+j,dddk|_-|j.\} }t/j0|j| |j1d dt/j2t/j3j4kr||j5| }ni}|j6| ||d|_7dS)Ncss|] }|jVqdS)N) requires_grad).0prrr sz3DistributedDataParallel.__init__..zhDistributedDataParallel is not needed when a module doesn't have any parameter that requires a gradient.rz8device_ids can only be None or contain a single element.cSsh|] }|jqSr)device)r^r_rrr sz3DistributedDataParallel.__init__..cSsh|] }|jjqSr)ratype)r^r_rrrrbszuDistributedDataParallel's input module must be on the same type of devices, but input module parameters locate in {}.rcpuzDistributedDataParallel device_ids and output_device arguments only work with single-device/multiple-device GPU modules or CPU modules, but got device_ids {}, output_device {}, and module parameters {}.cSsh|] }|jqSr)ra)r^r_rrrrbscSsg|]}t|dqS)T)r )r^xrrr sz4DistributedDataParallel.__init__..TF!_ddp_params_and_buffers_to_ignorezhThe `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.zModules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modulesiZPYTORCH_DDP_USE_SIDE_STREAM1)authoritative_rankii)8rEr@rFrrBany parameters_log_and_throw RuntimeErrorlen ValueErrorZis_multi_device_moduleformatr# device_type device_ids output_devicer r process_groupr8dimmodulerabroadcast_buffersrSrequire_backward_grad_syncrQgradient_as_bucket_viewhasattrrgparameters_to_ignorewarningswarnrr!nn parameterZUninitializedParameterintbroadcast_bucket_sizebucket_bytes_capr,r-get!use_side_stream_for_tensor_copies_build_params_for_reducerdistZ_verify_model_across_ranks_sync_params_and_buffers_get_debug_mode_DistributedDebugLevelOFF_build_param_to_name_mapping_ddp_init_helper_has_rebuilt_buckets)rGrwrsrtrvrxruZ bucket_cap_mbrSZcheck_reductionrzZdistinct_device_typesparamrlexpect_sparse_gradientparam_to_name_mapping)rIrrrFs                z DistributedDataParallel.__init__cCsTg}x.|jjjD]\}}||jkr|j|qWt|dkrP|j||j|dS)Nr)rwr4itemsr|appendro _distributed_broadcast_coalescedr)rGrjZ module_statesnamerrrrrNs  z0DistributedDataParallel._sync_params_and_bufferscCs0|jdk r$|jjt|d|||dS)Nz: )rBZset_error_and_logstr)rGZerr_typeerr_msgrrrrmYs z&DistributedDataParallel._log_and_throwc Csd|_tj|dtj|jg|d\}}tj|tt|tt||j||j|j |j |tj |_ tj |j |_ |j j|j |j j|jjj|jdkrgn|j|jdkrdn|j|j|j|jdS)a@ Initialization helper function that does the following: (1) bucketing the parameters for reductions (2) resetting the bucketing states (3) registering the grad hooks (4) Logging constructin-time DDP logging data (5) passing a handle of DDP to SyncBatchNorm Layer rNr)r9rZ"_compute_bucket_assignment_by_sizeZ_DEFAULT_FIRST_BUCKET_BYTESrZReducerr#reversedrurSrzr3LoggerrBZ set_loggerZset_construction_data_and_logrwrIr;rsrtrx_passing_sync_batchnorm_handle)rGrlrrZbucket_indicesZper_bucket_size_limitsrrrr^s0     z(DistributedDataParallel._ddp_init_helpercCs*|jtj|j}|d=|d=|d=|S)Nrur3rB)_check_default_groupcopy__dict__)rGattrsrrr __getstate__s  z$DistributedDataParallel.__getstate__cst|_tt|j||jjdd|jjdd|j\}}tj tj j kr\|j |}ni}|j ||||jr||jdS)NrQTry)r rurEr@ __setstate__r setdefaultrrrrrrrr8_set_static_graph)rGstaterlrr)rIrrrs  z$DistributedDataParallel.__setstate__csjfddjjDg}tfdd|D}dd|D}ddfdd|D}j||fS)Ncs:g|]2\}fdd|jddDD] }||fq(qS)cs.g|]&\}}|jrd|jkr|qS).)r]r|)r^ param_namer) module_namerGrrrfszPDistributedDataParallel._build_params_for_reducer...F)recurse)named_parameters)r^rwr)rG)rrrfs zEDistributedDataParallel._build_params_for_reducer..csg|]}fdd|DqS)cs,g|]$\}}|krj| r||fqSr)add)r^mr_)memorrrfszPDistributedDataParallel._build_params_for_reducer...r)r^Z replica_mps)rrrrfscSsg|]}tdd|DqS)css|]\}}|VqdS)Nr)r^_rrrrr`szODistributedDataParallel._build_params_for_reducer...)r#)r^replicarrrrfscSs&t|tjjst|tjjr"|jSdS)NF)rr!rZ EmbeddingZ EmbeddingBagsparse)rwrrrproduces_sparse_gradients zSDistributedDataParallel._build_params_for_reducer..produces_sparse_gradientcs"g|]}tfdd|DqS)c3s|]\}}|VqdS)Nr)r^rwr)rrrr`szODistributedDataParallel._build_params_for_reducer...)r#)r^r)rrrrfs)rw named_modulesset_assign_modules_buffers)rGZmodules_and_parametersrlrr)rrrGrrs   z1DistributedDataParallel._build_params_for_reducercs,ddjjD}fdd|D_dS)aU Assigns module buffers to self.modules_buffers which are then used to broadcast across ranks when broadcast_buffers=True. Note that this must be called every time buffers need to be synced because buffers can be reassigned by user module, see https://github.com/pytorch/pytorch/issues/63916. cSsg|]\}}||fqSrr)r^ buffer_namebufferrrrrfszCDistributedDataParallel._assign_modules_buffers..csg|]\}}|jkr|qSr)r|)r^rr)rGrrrfsN)rwZ named_buffersmodules_buffers)rGZnamed_module_buffersr)rGrrs  z/DistributedDataParallel._assign_modules_buffersc sfddttdD}td}i}xz|jjD]l\}}xb|jddD]R\}}|d|} | |jkrP|jrP||kr|jt d| d||} | || <qPWq:Wt|t|kr|jt d t|d t|d |S) Ncsi|]}|d|qS)rr)r^i)rlrr szHDistributedDataParallel._build_param_to_name_mapping..rF)rrzParam with name zt found in module parameters, but not DDP parameters. This indicates a bug in DDP, please report an issue to PyTorch.zUExpected param to name mapping to cover all parameters, but got conflicting lengths: z vs zA. This indicates a bug in DDP, please report an issue to PyTorch.) rangerorrwrrr|r]rmrp) rGrlZparam_to_param_indexZ param_setZparam_index_to_param_fqnrrwrrZfqnZ param_indexr)rlrrs$ z4DistributedDataParallel._build_param_to_name_mappingccs@dd}x2|r|jn|gD]}x||D] }|Vq*WqWdS)z: Returns a generator of module parameters css8t|dr|jjn |jdd}x|D] }|Vq&WdS)N_former_parametersF)r)r{rr)rl)rZpsr_rrrmodel_parameterss  zADistributedDataParallel._get_parameters..model_parametersN)modules)rGrrrr_rrr_get_parameterss z'DistributedDataParallel._get_parametersc CsFd}y|jtkrd}Wntk r0d}YnX|rB|jtddS)NFTzDDP Pickling/Unpickling are only supported when using DDP with the default process group. That is, when you have called init_process_group and have not passed process_group argument to DDP constructor)rur rnrm)rGZpickle_not_supportedrrrr+s  z,DistributedDataParallel._check_default_groupc cs$|j}d|_z dVWd||_XdS)aV A context manager to disable gradient synchronizations across DDP processes. Within this context, gradients will be accumulated on module variables, which will later be synchronized in the first forward-backward pass exiting the context. Example:: >>> ddp = torch.nn.parallel.DistributedDataParallel(model, pg) >>> with ddp.no_sync(): >>> for input in inputs: >>> ddp(input).backward() # no synchronization, accumulate grads >>> ddp(another_input).backward() # synchronize grads FN)ry)rGZold_require_backward_grad_syncrrrno_sync=s  zDistributedDataParallel.no_syncc Os&tjjjd*tjrB|jrB|jj|jd7_|j j t j |}|r`|j j ||jtjr|j jrtjdd|_|jr|j|jjr|jdd|jr|j|||jd\}}|j|d|d}n |j||}tjo|jr0d|_|jr"|j r"|j jtt|n |j jgnd|_WdQRX|jrR|j sf|jr"|jdkr"|j|jd}t |\}}}d d t!t"|D} x6t#|D]*\} }tj$|r|j%dkr|| | <qWt&j'|j |f|} x0t!t"| D] } | | dkr| | | | <qWt(| ||}|S) NzDistributedDataParallel.forwardrz4Reducer buckets have been rebuilt in this iteration.TF)rJr)r8r9cSsg|]}dqS)Nr)r^rrrrrfsz3DistributedDataParallel.forward..))r!ZautogradZprofilerZrecord_functionZis_grad_enabledryrBZset_runtime_stats_and_logr9r3Zprepare_for_forwardrZnotify_join_contextZ_set_forward_pass_work_handlerDrKlogginginforrQ _sync_params _join_configenablerMrs to_kwargsrwrSr8Zprepare_for_backwardr#r rrro enumerateZ is_tensorZgrad_fnr2applyr) rGr6kwargsrUrr4rrrZoutput_placeholdersrZpassthrough_tensor_listrrrr7Ts\          zDistributedDataParallel.forwardcCst||||jdS)N)rv)rrv)rGr6rrsrrrscatterszDistributedDataParallel.scatterc s(fddz |}WddX|S)z< Recursively moves input to the target_gpu. csZttjrjtjdkr$fSjs6jfSt}tjj|j}WdQRXtjj$tjj }|j ||j |WdQRX|fSt rfddt tDSttrtdkrtt tSttrtdkrddt tDSttrTtdkrTfddt tjDSgS)Ncudacsg|]}t|qSr)rc)r^args)r*rrrfszIDistributedDataParallel._recursive_to..to_map..rcSsg|] }t|qSr)r#)r^rrrrrfscsg|]}t|qSr)rc)r^r)r*rrrfs)rr!r"rarZtorrstreamcurrent_streamZ wait_streamZ record_streamrzipr'r$ror#r(r)r*rrr)rG target_gputo_map)r*rrs,     z5DistributedDataParallel._recursive_to..to_mapNr)rGr6rresr)rGrrr _recursive_tos   z%DistributedDataParallel._recursive_tocCs|r|j||ng}|r$|j||ng}t|t|kr^|jddtt|t|Dn4t|t|kr|jddtt|t|Dt|}t|}||fS)NcSsg|]}fqSrr)r^rrrrrfsz5DistributedDataParallel.to_kwargs..cSsg|]}iqSrr)r^rrrrrfs)rroextendrr$)rGr6rZ device_idrrrrs&$z!DistributedDataParallel.to_kwargscCst|||jdS)N)rv)rrv)rGoutputsrtrrrrszDistributedDataParallel.gathercstt|j||S)N)rEr@train)rGmode)rIrrrszDistributedDataParallel.traincCs(tjd|jd}tj||jd|jS)Nr)ra)group)r!zerosrar all_reducerurP)rGZall_active_procsrrr(_schedule_shadow_all_reduce_for_fwd_passsz@DistributedDataParallel._schedule_shadow_all_reduce_for_fwd_passcCsD| r|jrtjd|jd}ntjd|jd}tj||jdd}|S)Nr)raT)rZasync_op)ryr!Zonesrarrrru)rGrJZrequires_sync_tensorrUrrrrMs  zADistributedDataParallel._check_global_requires_backward_grad_synccCs,|jr(|j|jd}|j|j|j|dS)NF)will_sync_module_buffers_find_common_rank_distributed_rankrrr)rGrjrrrrLsz6DistributedDataParallel._check_and_sync_module_bufferscCs"|j|j||_|j|jddS)N)rj)rrZ_authoritative_rankr)rGrWrrrrXs z)DistributedDataParallel._sync_final_modelcCsLg}|jj}x"|D]}|jj|}|j|qWx|D] }|jq8WdS)N)r3Z_get_zeros_like_grad_bucketsZ_run_comm_hookrrN)rGZ comm_workZ grad_bucketsZ grad_bucketrUrrrrRs    z6DistributedDataParallel._match_all_reduce_for_bwd_passcCs|jj}|jj|dS)N)r3Z_get_local_used_mapsruZ allreduce)rGZlocally_used_param_mapsrrrrT1s z6DistributedDataParallel._match_unused_params_allreduce)rHrthrow_on_early_terminationcCst|g|||dS)a1 A context manager to be used in conjunction with an instance of :class:`torch.nn.parallel.DistributedDataParallel` to be able to train with uneven inputs across participating processes. This context manager will keep track of already-joined DDP processes, and "shadow" the forward and backward passes by inserting collective communication operations to match with the ones created by non-joined DDP processes. This will ensure each collective call has a corresponding call by already-joined DDP processes, preventing hangs or errors that would otherwise happen when training with uneven inputs across processes. Alternatively, if the flag ``throw_on_early_termination`` is specified to be ``True``, all trainers will throw an error once one rank runs out of inputs, allowing these errors to be caught and handled according to application logic. Once all DDP processes have joined, the context manager will broadcast the model corresponding to the last joined process to all processes to ensure the model is the same across all processes (which is guaranteed by DDP). To use this to enable training with uneven inputs across processes, simply wrap this context manager around your training loop. No further modifications to the model or data loading is required. .. warning:: If the model or training loop this context manager is wrapped around has additional distributed collective operations, such as ``SyncBatchNorm`` in the model's forward pass, then the flag ``throw_on_early_termination`` must be enabled. This is because this context manager is not aware of non-DDP collective communication. This flag will cause all ranks to throw when any one rank exhausts inputs, allowing these errors to be caught and recovered from across all ranks. Args: divide_by_initial_world_size (bool): If ``True``, will divide gradients by the initial ``world_size`` DDP training was launched with. If ``False``, will compute the effective world size (number of ranks that have not depleted their inputs yet) and divide gradients by that during allreduce. Set ``divide_by_initial_world_size=True`` to ensure every input sample including the uneven inputs have equal weight in terms of how much they contribute to the global gradient. This is achieved by always dividing the gradient by the initial ``world_size`` even when we encounter uneven inputs. If you set this to ``False``, we divide the gradient by the remaining number of nodes. This ensures parity with training on a smaller ``world_size`` although it also means the uneven inputs would contribute more towards the global gradient. Typically, you would want to set this to ``True`` for cases where the last few inputs of your training job are uneven. In extreme cases, where there is a large discrepancy in the number of inputs, setting this to ``False`` might provide better results. enable (bool): Whether to enable uneven input detection or not. Pass in ``enable=False`` to disable in cases where you know that inputs are even across participating processes. Default is ``True``. throw_on_early_termination (bool): Whether to throw an error or continue training when at least one rank has exhausted inputs. If ``True``, will throw upon the first rank reaching end of data. If ``False``, will continue training with a smaller effective world size until all ranks are joined. Note that if this flag is specified, then the flag ``divide_by_initial_world_size`` would be ignored. Default is ``False``. Example:: >>> import torch >>> import torch.distributed as dist >>> import os >>> import torch.multiprocessing as mp >>> import torch.nn as nn >>> # On each spawned worker >>> def worker(rank): >>> dist.init_process_group("nccl", rank=rank, world_size=2) >>> torch.cuda.set_device(rank) >>> model = nn.Linear(1, 1, bias=False).to(rank) >>> model = torch.nn.parallel.DistributedDataParallel( >>> model, device_ids=[rank], output_device=rank >>> ) >>> # Rank 1 gets one more input than rank 0. >>> inputs = [torch.tensor([1]).float() for _ in range(10 + rank)] >>> with model.join(): >>> for _ in range(5): >>> for inp in inputs: >>> loss = model(inp).sum() >>> loss.backward() >>> # Without the join() API, the below synchronization will hang >>> # blocking for rank 1's allreduce to complete. >>> torch.cuda.synchronize(device=rank) )rH)r)rGrHrrrrrjoin5s dzDistributedDataParallel.joincKs|jdd}t||dS)a Returns the DDP join hook, which enables training on uneven inputs by shadowing the collective communications in the forward and backward passes. Arguments: kwargs (dict): a :class:`dict` containing any keyword arguments to modify the behavior of the join hook at run time; all :class:`Joinable` instances sharing the same join context manager are forwarded the same value for ``kwargs``. The hook supports the following keyword arguments: divide_by_initial_world_size (bool, optional): If ``True``, then gradients are divided by the initial world size that DDP was launched with. If ``False``, then gradients are divided by the effective world size (i.e. the number of non-joined processes), meaning that the uneven inputs contribute more toward the global gradient. Typically, this should be set to ``True`` if the degree of unevenness is small but can be set to ``False`` in extreme cases for possibly better results. Default is ``True``. rHT)rH)rr?)rGrrHrrr join_hooks z!DistributedDataParallel.join_hookcCs|jS)N)ra)rGrrr join_devicesz#DistributedDataParallel.join_devicecCs|jS)N)ru)rGrrrjoin_process_groupsz*DistributedDataParallel.join_process_group)rhookcCs,|j||jj|jtj|j||dS)a Registers a communication hook which is an enhancement that provides a flexible hook to users where they can specify how DDP aggregates gradients across multiple workers. This hook would be very useful for researchers to try out new ideas. For example, this hook can be used to implement several algorithms like GossipGrad and gradient compression which involve different communication strategies for parameter syncs while running Distributed DataParallel training. Args: state (object): Passed to the hook to maintain any state information during the training process. Examples include error feedback in gradient compression, peers to communicate with next in GossipGrad, etc. It is locally stored by each worker and shared by all the gradient tensors on the worker. hook (callable): Callable with the following signature: ``hook(state: object, bucket: dist.GradBucket) -> torch.futures.Future[torch.Tensor]``: This function is called once the bucket is ready. The hook can perform whatever processing is needed and return a Future indicating completion of any async work (ex: allreduce). If the hook doesn't perform any communication, it still must return a completed Future. The Future should hold the new value of grad bucket's tensors. Once a bucket is ready, c10d reducer would call this hook and use the tensors returned by the Future and copy grads to individual parameters. Note that the future's return type must be a single tensor. We also provide an API called ``get_future`` to retrieve a Future associated with the completion of ``c10d.ProcessGroup.Work``. ``get_future`` is currently supported for NCCL and also supported for most operations on GLOO and MPI, except for peer to peer operations (send/recv). .. warning :: Grad bucket's tensors will not be predivided by world_size. User is responsible to divide by the world_size in case of operations like allreduce. .. warning :: DDP communication hook can only be registered once and should be registered before calling backward. .. warning :: The Future object that hook returns should contain a single tensor that has the same shape with the tensors inside grad bucket. .. warning :: ``get_future`` API supports NCCL, and partially GLOO and MPI backends (no support for peer-to-peer operations like send/recv) and will return a ``torch.futures.Future``. Example:: Below is an example of a noop hook that returns the same tensor. >>> def noop(state: object, bucket: dist.GradBucket): -> torch.futures.Future[torch.Tensor] >>> fut = torch.futures.Future() >>> fut.set_result(bucket.buffer()) >>> return fut >>> ddp.register_comm_hook(state=None, hook=noop) Example:: Below is an example of a Parallel SGD algorithm where gradients are encoded before allreduce, and then decoded after allreduce. >>> def encode_and_decode(state: object, bucket: dist.GradBucket): -> torch.futures.Future[torch.Tensor] >>> encoded_tensor = encode(bucket.buffer()) # encode gradients >>> fut = torch.distributed.all_reduce(encoded_tensor).get_future() >>> # Define the then callback to decode. >>> def decode(fut): >>> decoded_tensor = decode(fut.value()[0]) # decode gradients >>> return decoded_tensor >>> return fut.then(decode) >>> ddp.register_comm_hook(state=None, hook=encode_and_decode) N)_check_comm_hookrB_set_comm_hook_namer=rZ_register_comm_hookr3)rGrrrrrregister_comm_hooksM z*DistributedDataParallel.register_comm_hookcCs"|jjt|tj|j|dS)a Registers a built-in communication hook that specifies how DDP aggregates gradients across multiple workers. The built-in hooks aim to provide efficient C++ implementations for certain hooks, which might not be as efficient if implemented in Python using a Python communication hook. Args: comm_hook_type (dist.BuiltinCommHookType): type of communication hook, such as ALLREDUCE, FP16_COMPRESS, etc. .. warning :: DDP communication hook can only be registered once and should be registered before calling backward. Example:: Below is an example of a FP16 compression where gradients are compressed into 16-bit floating-point numbers before allreduce, and then decompressed after allreduce. >>> ddp._register_builtin_comm_hook(dist.BuiltinCommHookType.FP16_COMPRESS) N)rBrrr_register_builtin_comm_hookr3)rGZcomm_hook_typerrrrsz3DistributedDataParallel._register_builtin_comm_hookcCstj|j|||dS)N)rZ_broadcast_coalescedru)rGZtensors buffer_sizerjrrrr3sz8DistributedDataParallel._distributed_broadcast_coalescedcCs|jo|jot|jdkS)Nr)rQrxror)rGrrrr:sz0DistributedDataParallel.will_sync_module_bufferscCsNtj|r |ndg|jd}tj|tj|jd|jdkrF|j t d|jS)Nr)ra)oprzuBUG! Expected rank_cond to be true for at least one process. This indicates a bug in PyTorch, please report an issue.rr) r!Ztensorrarrr MAXrurPrmrp)rGZ input_rankZ rank_condZ rank_to_userrrrAs   z)DistributedDataParallel._find_common_rankc CsVtjD|jrH|jjr*|j|jd}nd}|j|j|j |j |WdQRXdS)NTr) r!Zno_gradrrrrrrrrr)rGrjrrrrQs  z$DistributedDataParallel._sync_paramscCs<x6|jD]*}t|tjjjr |jdkr |jtdq WdS)Nrdz/SyncBatchNorm layers only work with GPU modules)rrr!rZ SyncBatchNormrrrmrp)rGrwZlayerrrrris  z6DistributedDataParallel._passing_sync_batchnorm_handlecCst|s|jtdtj|}|jdjtjkrN|jdjtj krN|jt d|j tjkrz|j t j jt jkrz|jt d|jdkrt jjdkstt jjjddd kstj stj st jjjdkr|jtd dS)Nz$Communication hook must be callable.Zbucketz@Communication hook: bucket annotation should be dist.GradBucket.zSCommunication hook: return annotation should be torch.futures.Future[torch.Tensor].bf16_compress_hookbf16_compress_wrapper_hookrr r zEBF16 all reduce communication hook required CUDA 11+ and NCCL 2.9.7+.)rr)rrr)callablerm TypeErrorinspect signaturerl annotation_emptyrZ GradBucketrpreturn_annotationr!futuresFuturer"r;versionrrsplit is_availableZis_nccl_availableZnccl)rGrsigrrrrqs(       z(DistributedDataParallel._check_comm_hookcCs tj|jS)N)rZget_rankru)rGrrrrsz)DistributedDataParallel._distributed_rankcCs ||_dS)a; Sets parameters and buffers to be ignored by DDP. Expected format for parameters is the fully qualified name: {module_name}.{param_name}, and similarly, {module_name}.{buffer_name} for buffers. For example: params_to_ignore = [] # NB: model here is vanilla PyTorch module, not yet wrapped with DDP. for module_name, module in model.named_modules(): for param_name, param in module.named_parameters(recurse=False): if should_ignore(param): # Create expected format fqn = f"{module_name}.{param_name}" params_to_ignore.append(fqn) torch.nn.parallel.DistributedDataParallel._set_params_and_buffers_to_ignore_for_model( model, params_to_ignore ) N)rg)rwZparams_and_buffers_to_ignorerrr+_set_params_and_buffers_to_ignore_for_modelszCDistributedDataParallel._set_params_and_buffers_to_ignore_for_modelcCs|jj}|j|jS)a This interface can be called after DistributedDataParallel() is constructed. It returns a dictionary of logging data. It could help for debugging and analysis. The loggind data includes DistributedDataParallel constructor input parameters, some internal states of DistributedDataParallel and performance metrics. Simply print the dictorinary and see what these metrics are. THis is a prototype interface and subject to change in the future. )rB_get_ddp_logging_dataZstrs_mapZints_map)rGZddp_logging_datarrrrs z-DistributedDataParallel._get_ddp_logging_datacCs$|dkr|jtd|jj|dS)a. This interface allows users to set sample_rate of collecting runtime stats. The runtime stats will be recorded for the first 10 iterations, after 10 iteratons runtime stats will be recorded once every "sample_rate" training iterations. In default, runtime stats are recorded for the first 10 iterations, after 10 iterations runtime stats are recorded once every "kDDPRuntimeLoggingSampleRate=100" training iterations. This is a prototype interface and subject to change in the future. rzADDP runtime logging sample rate should be equal or greater than 1N)rmrpr3$_set_ddp_runtime_logging_sample_rate)rGZ sample_raterrrrs zrrrrr[rr)rIrr@sjxx 9 5!  ]+    g  Q $   r@)/rrr%rr,r} contextlibrr!Ztorch.distributedZ distributedrZtorch.autogradrrZ!torch.distributed.algorithms.joinrrrZtorch.utils._pytreerr rrZ"torch.distributed.distributed_c10dr r ZrpcZtorch.distributed.rpcr Z torch._utilsr rrZ _functionsrZscatter_gatherrrrrrr r1r2r?r@rrrrs8        >9