/usr/local/lib64/python3.6/site-packages/torch/nn/modules/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/nn/modules/__pycache__/batchnorm.cpython-36.pyc (31039B)
3
Egh @ s d dl mZmZ d dlZd dlmZ d dlmZmZmZ ddl m
Z ddl m Z dd l
mZ dd
lmZ dd lmZ G d d
d
eZG dd deZG dd deeZG dd deZG dd deeZG dd deZG dd deeZG dd deZG d d d eeZ G d d d eZdS ) )OptionalAnyN)Tensor) ParameterUninitializedParameterUninitializedBuffer )
functional)init )
SyncBatchNorm)LazyModuleMixin)Modulec s e Zd ZU dZdZdddddgZee e
e e
deeee e d d fd
d
Zd dddZd dddZdd Zdd Z fddZ ZS ) _NormBasez+Common base of _InstanceNorm and _BatchNormr track_running_statsmomentumeps num_featuresaffineh㈵>皙?TN)r r r r r returnc s ||d}t t| j || _|| _|| _|| _|| _| jrftt j
|f|| _ tt j
|f|| _ n| j
dd | j
dd | jr| jdt j|f| | jdt j|f| | | | jdt jd dt jid d
|j D | n$| jdd | jdd | jdd | j d S ) N)devicedtypeweightbias running_mean running_varnum_batches_trackedr r c S s i | ]\}}|d kr||qS )r ).0kvr r F/usr/local/lib64/python3.6/site-packages/torch/nn/modules/batchnorm.py
9 s z&_NormBase.__init__..)r )superr __init__r r r r r r torchemptyr r Zregister_parameterZregister_bufferZzerosZonestensorlongitemsreset_parameters) selfr r r r r r r factory_kwargs) __class__r r# r& s2
z_NormBase.__init__)r c C s* | j r&| jj | jjd | jj d S )Nr )r r Zzero_r Zfill_r )r- r r r# reset_running_statsA s
z _NormBase.reset_running_statsc C s* | j | jr&tj| j tj| j d S )N)r0 r r
Zones_r Zzeros_r )r- r r r# r, I s z_NormBase.reset_parametersc C s t d S )N)NotImplementedError)r- inputr r r# _check_input_dimO s z_NormBase._check_input_dimc C s dj f | jS )Nzj{num_features}, eps={eps}, momentum={momentum}, affine={affine}, track_running_stats={track_running_stats})format__dict__)r- r r r#
extra_reprR s z_NormBase.extra_reprc
sf |j dd }|d ks |dk rF| jrF|d } | |krFtjdtjd|| < tt| j||||||| d S )Nversionr r r )r )getr r' r) r* r% r _load_from_state_dict)
r- Z
state_dictprefixZlocal_metadatastrictZ missing_keysZunexpected_keysZ
error_msgsr7 Znum_batches_tracked_key)r/ r r# r9 X s
z _NormBase._load_from_state_dict)r r TTNN)__name__
__module__ __qualname____doc___versionZ
__constants__intr floatr r boolr r r& r0 r, r3 r6 r9
__classcell__r r )r/ r# r s(
r c s. e Zd Zd
fdd Zeeddd Z ZS )
_BatchNormh㈵>皙?TNc s* ||d}t t| j|||||f| d S )N)r r )r% rE r& ) r- r r r r r r r r. )r/ r r# r& w s
z_BatchNorm.__init__)r2 r c C s | j | | jd krd}n| j}| jrb| jrb| jd k rb| jd | _| jd kr\dt| j }n| j}| jrnd}n| jd ko| jd k}tj || j s| jr| jnd | j s| jr| jnd | j
| j ||| j S )Ng r g ?T)
r3 r trainingr r rB r r F
batch_normr r r )r- r2 exponential_average_factor bn_trainingr r r# forward s.
z_BatchNorm.forward)rF rG TTNN)r< r= r> r& r rM rD r r )r/ r# rE v s rE c sN e Zd ZU eed dd fdd
Zdd fdd Zddd
d Z Z S )
_LazyNormBaseh㈵>皙?TN)r c s ||d}t t| jd||ddf| || _|| _| jrPtf || _tf || _| jrtf || _ tf || _
t j ddt j
idd |j D | _d S )N)r r r Fr c S s i | ]\}}|d kr||qS )r r )r r! r" r r r# r$ s z*_LazyNormBase.__init__..)r )r% rN r& r r r r r r r r r' r) r* r+ r )r- r r r r r r r. )r/ r r# r& s$
z_LazyNormBase.__init__c s" | j r | jdkr t j d S )Nr )has_uninitialized_paramsr r% r, )r- )r/ r r# r, s z _LazyNormBase.reset_parametersc C s | j r|jd | _| jrZt| jts*tt| jts:t| jj | jf | jj | jf | j
r| j j | jf | j j | jf | j
d S )Nr )rQ shaper r
isinstancer r AssertionErrorr Z materializer r r r, )r- r2 r r r# initialize_parameters s z#_LazyNormBase.initialize_parameters)rO rP TTNN)
r< r= r> r r r r& r, rU rD r r )r/ r# rN s
rN c @ s e Zd ZdZdd ZdS ) BatchNorm1da
Applies Batch Normalization over a 2D or 3D input (a mini-batch of 1D
inputs with optional additional channel dimension) as described in the paper
`Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift `__ .
.. math::
y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
The mean and standard-deviation are calculated per-dimension over
the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors
of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are set
to 1 and the elements of :math:`\beta` are set to 0. The standard-deviation is calculated
via the biased estimator, equivalent to `torch.var(input, unbiased=False)`.
Also by default, during training this layer keeps running estimates of its
computed mean and variance, which are then used for normalization during
evaluation. The running estimates are kept with a default :attr:`momentum`
of 0.1.
If :attr:`track_running_stats` is set to ``False``, this layer then does not
keep running estimates, and batch statistics are instead used during
evaluation time as well.
.. note::
This :attr:`momentum` argument is different from one used in optimizer
classes and the conventional notion of momentum. Mathematically, the
update rule for running statistics here is
:math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`,
where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the
new observed value.
Because the Batch Normalization is done over the `C` dimension, computing statistics
on `(N, L)` slices, it's common terminology to call this Temporal Batch Normalization.
Args:
num_features: :math:`C` from an expected input of size
:math:`(N, C, L)` or :math:`L` from input of size :math:`(N, L)`
eps: a value added to the denominator for numerical stability.
Default: 1e-5
momentum: the value used for the running_mean and running_var
computation. Can be set to ``None`` for cumulative moving average
(i.e. simple average). Default: 0.1
affine: a boolean value that when set to ``True``, this module has
learnable affine parameters. Default: ``True``
track_running_stats: a boolean value that when set to ``True``, this
module tracks the running mean and variance, and when set to ``False``,
this module does not track such statistics, and initializes statistics
buffers :attr:`running_mean` and :attr:`running_var` as ``None``.
When these buffers are ``None``, this module always uses batch statistics.
in both training and eval modes. Default: ``True``
Shape:
- Input: :math:`(N, C)` or :math:`(N, C, L)`
- Output: :math:`(N, C)` or :math:`(N, C, L)` (same shape as input)
Examples::
>>> # With Learnable Parameters
>>> m = nn.BatchNorm1d(100)
>>> # Without Learnable Parameters
>>> m = nn.BatchNorm1d(100, affine=False)
>>> input = torch.randn(20, 100)
>>> output = m(input)
c C s. |j dkr*|j dkr*tdj|j d S )Nr z'expected 2D or 3D input (got {}D input))dim
ValueErrorr4 )r- r2 r r r# r3 ) s z BatchNorm1d._check_input_dimN)r< r= r> r? r3 r r r r# rV s ArV c @ s e Zd ZdZeZdd ZdS )LazyBatchNorm1da6 A :class:`torch.nn.BatchNorm1d` module with lazy initialization of
the ``num_features`` argument of the :class:`BatchNorm1d` that is inferred
from the ``input.size(1)``.
The attributes that will be lazily initialized are `weight`, `bias`,
`running_mean` and `running_var`.
Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation
on lazy modules and their limitations.
Args:
eps: a value added to the denominator for numerical stability.
Default: 1e-5
momentum: the value used for the running_mean and running_var
computation. Can be set to ``None`` for cumulative moving average
(i.e. simple average). Default: 0.1
affine: a boolean value that when set to ``True``, this module has
learnable affine parameters. Default: ``True``
track_running_stats: a boolean value that when set to ``True``, this
module tracks the running mean and variance, and when set to ``False``,
this module does not track such statistics, and initializes statistics
buffers :attr:`running_mean` and :attr:`running_var` as ``None``.
When these buffers are ``None``, this module always uses batch statistics.
in both training and eval modes. Default: ``True``
c C s. |j dkr*|j dkr*tdj|j d S )Nr rW z'expected 2D or 3D input (got {}D input))rX rY r4 )r- r2 r r r# r3 L s z LazyBatchNorm1d._check_input_dimN)r< r= r> r? rV
cls_to_becomer3 r r r r# rZ 0 s rZ c @ s e Zd ZdZdd ZdS ) BatchNorm2da Applies Batch Normalization over a 4D input (a mini-batch of 2D inputs
with additional channel dimension) as described in the paper
`Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift `__ .
.. math::
y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
The mean and standard-deviation are calculated per-dimension over
the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors
of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are set
to 1 and the elements of :math:`\beta` are set to 0. The standard-deviation is calculated
via the biased estimator, equivalent to `torch.var(input, unbiased=False)`.
Also by default, during training this layer keeps running estimates of its
computed mean and variance, which are then used for normalization during
evaluation. The running estimates are kept with a default :attr:`momentum`
of 0.1.
If :attr:`track_running_stats` is set to ``False``, this layer then does not
keep running estimates, and batch statistics are instead used during
evaluation time as well.
.. note::
This :attr:`momentum` argument is different from one used in optimizer
classes and the conventional notion of momentum. Mathematically, the
update rule for running statistics here is
:math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`,
where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the
new observed value.
Because the Batch Normalization is done over the `C` dimension, computing statistics
on `(N, H, W)` slices, it's common terminology to call this Spatial Batch Normalization.
Args:
num_features: :math:`C` from an expected input of size
:math:`(N, C, H, W)`
eps: a value added to the denominator for numerical stability.
Default: 1e-5
momentum: the value used for the running_mean and running_var
computation. Can be set to ``None`` for cumulative moving average
(i.e. simple average). Default: 0.1
affine: a boolean value that when set to ``True``, this module has
learnable affine parameters. Default: ``True``
track_running_stats: a boolean value that when set to ``True``, this
module tracks the running mean and variance, and when set to ``False``,
this module does not track such statistics, and initializes statistics
buffers :attr:`running_mean` and :attr:`running_var` as ``None``.
When these buffers are ``None``, this module always uses batch statistics.
in both training and eval modes. Default: ``True``
Shape:
- Input: :math:`(N, C, H, W)`
- Output: :math:`(N, C, H, W)` (same shape as input)
Examples::
>>> # With Learnable Parameters
>>> m = nn.BatchNorm2d(100)
>>> # Without Learnable Parameters
>>> m = nn.BatchNorm2d(100, affine=False)
>>> input = torch.randn(20, 100, 35, 45)
>>> output = m(input)
c C s" |j dkr tdj|j d S )N z!expected 4D input (got {}D input))rX rY r4 )r- r2 r r r# r3 s z BatchNorm2d._check_input_dimN)r< r= r> r? r3 r r r r# r\ S s Ar\ c @ s e Zd ZdZeZdd ZdS )LazyBatchNorm2da6 A :class:`torch.nn.BatchNorm2d` module with lazy initialization of
the ``num_features`` argument of the :class:`BatchNorm2d` that is inferred
from the ``input.size(1)``.
The attributes that will be lazily initialized are `weight`, `bias`,
`running_mean` and `running_var`.
Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation
on lazy modules and their limitations.
Args:
eps: a value added to the denominator for numerical stability.
Default: 1e-5
momentum: the value used for the running_mean and running_var
computation. Can be set to ``None`` for cumulative moving average
(i.e. simple average). Default: 0.1
affine: a boolean value that when set to ``True``, this module has
learnable affine parameters. Default: ``True``
track_running_stats: a boolean value that when set to ``True``, this
module tracks the running mean and variance, and when set to ``False``,
this module does not track such statistics, and initializes statistics
buffers :attr:`running_mean` and :attr:`running_var` as ``None``.
When these buffers are ``None``, this module always uses batch statistics.
in both training and eval modes. Default: ``True``
c C s" |j dkr tdj|j d S )Nr] z!expected 4D input (got {}D input))rX rY r4 )r- r2 r r r# r3 s z LazyBatchNorm2d._check_input_dimN)r< r= r> r? r\ r[ r3 r r r r# r^ s r^ c @ s e Zd ZdZdd ZdS ) BatchNorm3da Applies Batch Normalization over a 5D input (a mini-batch of 3D inputs
with additional channel dimension) as described in the paper
`Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift `__ .
.. math::
y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
The mean and standard-deviation are calculated per-dimension over
the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors
of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are set
to 1 and the elements of :math:`\beta` are set to 0. The standard-deviation is calculated
via the biased estimator, equivalent to `torch.var(input, unbiased=False)`.
Also by default, during training this layer keeps running estimates of its
computed mean and variance, which are then used for normalization during
evaluation. The running estimates are kept with a default :attr:`momentum`
of 0.1.
If :attr:`track_running_stats` is set to ``False``, this layer then does not
keep running estimates, and batch statistics are instead used during
evaluation time as well.
.. note::
This :attr:`momentum` argument is different from one used in optimizer
classes and the conventional notion of momentum. Mathematically, the
update rule for running statistics here is
:math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`,
where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the
new observed value.
Because the Batch Normalization is done over the `C` dimension, computing statistics
on `(N, D, H, W)` slices, it's common terminology to call this Volumetric Batch Normalization
or Spatio-temporal Batch Normalization.
Args:
num_features: :math:`C` from an expected input of size
:math:`(N, C, D, H, W)`
eps: a value added to the denominator for numerical stability.
Default: 1e-5
momentum: the value used for the running_mean and running_var
computation. Can be set to ``None`` for cumulative moving average
(i.e. simple average). Default: 0.1
affine: a boolean value that when set to ``True``, this module has
learnable affine parameters. Default: ``True``
track_running_stats: a boolean value that when set to ``True``, this
module tracks the running mean and variance, and when set to ``False``,
this module does not track such statistics, and initializes statistics
buffers :attr:`running_mean` and :attr:`running_var` as ``None``.
When these buffers are ``None``, this module always uses batch statistics.
in both training and eval modes. Default: ``True``
Shape:
- Input: :math:`(N, C, D, H, W)`
- Output: :math:`(N, C, D, H, W)` (same shape as input)
Examples::
>>> # With Learnable Parameters
>>> m = nn.BatchNorm3d(100)
>>> # Without Learnable Parameters
>>> m = nn.BatchNorm3d(100, affine=False)
>>> input = torch.randn(20, 100, 35, 45, 10)
>>> output = m(input)
c C s" |j dkr tdj|j d S )N z!expected 5D input (got {}D input))rX rY r4 )r- r2 r r r# r3 s z BatchNorm3d._check_input_dimN)r< r= r> r? r3 r r r r# r_ s Br_ c @ s e Zd ZdZeZdd ZdS )LazyBatchNorm3da6 A :class:`torch.nn.BatchNorm3d` module with lazy initialization of
the ``num_features`` argument of the :class:`BatchNorm3d` that is inferred
from the ``input.size(1)``.
The attributes that will be lazily initialized are `weight`, `bias`,
`running_mean` and `running_var`.
Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation
on lazy modules and their limitations.
Args:
eps: a value added to the denominator for numerical stability.
Default: 1e-5
momentum: the value used for the running_mean and running_var
computation. Can be set to ``None`` for cumulative moving average
(i.e. simple average). Default: 0.1
affine: a boolean value that when set to ``True``, this module has
learnable affine parameters. Default: ``True``
track_running_stats: a boolean value that when set to ``True``, this
module tracks the running mean and variance, and when set to ``False``,
this module does not track such statistics, and initializes statistics
buffers :attr:`running_mean` and :attr:`running_var` as ``None``.
When these buffers are ``None``, this module always uses batch statistics.
in both training and eval modes. Default: ``True``
c C s" |j dkr tdj|j d S )Nr` z!expected 5D input (got {}D input))rX rY r4 )r- r2 r r r# r3 ! s z LazyBatchNorm3d._check_input_dimN)r< r= r> r? r_ r[ r3 r r r r# ra s ra c sf e Zd ZdZdeeeeeee dd fdd
Z d d
Z
d d Z e e d
ddZ
edddZ ZS )r a Applies Batch Normalization over a N-Dimensional input (a mini-batch of [N-2]D inputs
with additional channel dimension) as described in the paper
`Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift `__ .
.. math::
y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
The mean and standard-deviation are calculated per-dimension over all
mini-batches of the same process groups. :math:`\gamma` and :math:`\beta`
are learnable parameter vectors of size `C` (where `C` is the input size).
By default, the elements of :math:`\gamma` are sampled from
:math:`\mathcal{U}(0, 1)` and the elements of :math:`\beta` are set to 0.
The standard-deviation is calculated via the biased estimator, equivalent to
`torch.var(input, unbiased=False)`.
Also by default, during training this layer keeps running estimates of its
computed mean and variance, which are then used for normalization during
evaluation. The running estimates are kept with a default :attr:`momentum`
of 0.1.
If :attr:`track_running_stats` is set to ``False``, this layer then does not
keep running estimates, and batch statistics are instead used during
evaluation time as well.
.. note::
This :attr:`momentum` argument is different from one used in optimizer
classes and the conventional notion of momentum. Mathematically, the
update rule for running statistics here is
:math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`,
where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the
new observed value.
Because the Batch Normalization is done for each channel in the ``C`` dimension, computing
statistics on ``(N, +)`` slices, it's common terminology to call this Volumetric Batch
Normalization or Spatio-temporal Batch Normalization.
Currently :class:`SyncBatchNorm` only supports
:class:`~torch.nn.DistributedDataParallel` (DDP) with single GPU per process. Use
:meth:`torch.nn.SyncBatchNorm.convert_sync_batchnorm()` to convert
:attr:`BatchNorm*D` layer to :class:`SyncBatchNorm` before wrapping
Network with DDP.
Args:
num_features: :math:`C` from an expected input of size
:math:`(N, C, +)`
eps: a value added to the denominator for numerical stability.
Default: ``1e-5``
momentum: the value used for the running_mean and running_var
computation. Can be set to ``None`` for cumulative moving average
(i.e. simple average). Default: 0.1
affine: a boolean value that when set to ``True``, this module has
learnable affine parameters. Default: ``True``
track_running_stats: a boolean value that when set to ``True``, this
module tracks the running mean and variance, and when set to ``False``,
this module does not track such statistics, and initializes statistics
buffers :attr:`running_mean` and :attr:`running_var` as ``None``.
When these buffers are ``None``, this module always uses batch statistics.
in both training and eval modes. Default: ``True``
process_group: synchronization of stats happen within each process group
individually. Default behavior is synchronization across the whole
world
Shape:
- Input: :math:`(N, C, +)`
- Output: :math:`(N, C, +)` (same shape as input)
.. note::
Synchronization of batchnorm statistics occurs only while training, i.e.
synchronization is disabled when ``model.eval()`` is set or if
``self.training`` is otherwise ``False``.
Examples::
>>> # With Learnable Parameters
>>> m = nn.SyncBatchNorm(100)
>>> # creating process group (optional)
>>> # ranks is a list of int identifying rank ids.
>>> ranks = list(range(8))
>>> r1, r2 = ranks[:4], ranks[4:]
>>> # Note: every rank calls into new_group for every
>>> # process group created, even if that rank is not
>>> # part of the group.
>>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]]
>>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1]
>>> # Without Learnable Parameters
>>> m = nn.BatchNorm3d(100, affine=False, process_group=process_group)
>>> input = torch.randn(20, 100, 35, 45, 10)
>>> output = m(input)
>>> # network is nn.BatchNorm layer
>>> sync_bn_network = nn.SyncBatchNorm.convert_sync_batchnorm(network, process_group)
>>> # only single gpu per process is currently supported
>>> ddp_sync_bn_network = torch.nn.parallel.DistributedDataParallel(
>>> sync_bn_network,
>>> device_ids=[args.local_rank],
>>> output_device=args.local_rank)
h㈵>皙?TN)r r r r r
process_groupr c
s0 ||d} t t| j|||||f| || _d S )N)r r )r% r r& rd )
r- r r r r r rd r r r. )r/ r r# r& s
zSyncBatchNorm.__init__c C s" |j dk r tdj|j d S )Nr z*expected at least 2D input (got {}D input))rX rY r4 )r- r2 r r r# r3 s z SyncBatchNorm._check_input_dimc C s |j ddkrtdd S )Nr r z9SyncBatchNorm number of input channels should be non-zero)sizerY )r- r2 r r r# _check_non_zero_input_channels s z,SyncBatchNorm._check_non_zero_input_channels)r2 r c
C sZ |j std| j| | j| | jd kr2d}n| j}| jr~| jr~| jd k sRt| jd | _| jd krxd| jj }n| j}| jrd}n| j
d ko| j d k}| j s| jr| j
nd }| j s| jr| j nd }|o| j}|rt j
jj}| jr| j}t j
j|}|dk}|s,tj|||| j| j||| jS |s6ttj|| j| j||| j||| S d S )Nz0SyncBatchNorm expected input tensor to be on GPUg r g ?T)Zis_cudarY r3 rf r rH r r rT itemr r r' Z distributedgroupZWORLDrd Zget_world_sizerI rJ r r r sync_batch_normapply) r- r2 rK rL r r Z need_syncrd Z
world_sizer r r# rM s\
zSyncBatchNorm.forwardc
C s |}t |tjjjjrtjj|j|j|j |j
|j |}|j
r`tj |j
|_
|j|_W dQ R X |j|_|j|_|j|_t|dr|j|_x(|j D ] \}}|j|| j|| qW ~|S )a Helper function to convert all :attr:`BatchNorm*D` layers in the model to
:class:`torch.nn.SyncBatchNorm` layers.
Args:
module (nn.Module): module containing one or more :attr:`BatchNorm*D` layers
process_group (optional): process group to scope synchronization,
default is the whole world
Returns:
The original :attr:`module` with the converted :class:`torch.nn.SyncBatchNorm`
layers. If the original :attr:`module` is a :attr:`BatchNorm*D` layer,
a new :class:`torch.nn.SyncBatchNorm` layer object will be returned
instead.
Example::
>>> # Network with nn.BatchNorm layer
>>> module = torch.nn.Sequential(
>>> torch.nn.Linear(20, 100),
>>> torch.nn.BatchNorm1d(100),
>>> ).cuda()
>>> # creating process group (optional)
>>> # ranks is a list of int identifying rank ids.
>>> ranks = list(range(8))
>>> r1, r2 = ranks[:4], ranks[4:]
>>> # Note: every rank calls into new_group for every
>>> # process group created, even if that rank is not
>>> # part of the group.
>>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]]
>>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1]
>>> sync_bn_module = torch.nn.SyncBatchNorm.convert_sync_batchnorm(module, process_group)
Nqconfig)rS r' nnmodulesZ batchnormrE r r r r r r Zno_gradr r r r r hasattrrk Znamed_childrenZ
add_moduleconvert_sync_batchnorm)clsmodulerd Z
module_outputnamechildr r r# ro s. #
z$SyncBatchNorm.convert_sync_batchnorm)rb rc TTNNN)N)r< r= r> r? rA rB rC r r r& r3 rf r rM classmethodro rD r r )r/ r# r & s c Qr ) typingr r r' r Ztorch.nn.parameterr r r r rI r
Z
_functionsr ri Zlazyr
rq r r rE rN rV rZ r\ r^ r_ ra r r r r# s$ hA/J#H!I!