/usr/local/lib64/python3.6/site-packages/torch/nn/modules/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/nn/modules/__pycache__/normalization.cpython-36.pyc (11360B)
3
ûEgþ) ã @ sÆ d dl Z d dlZd dlmZ ddlmZ ddlmZ ddl m
Z ddl m Z d d l m
Z
mZ d d
lmZmZmZ G d d „ d eƒZG d
d„ deƒZeeee ef ZG dd„ deƒZG dd„ deƒZdS )é N)Ú Parameteré )ÚModule)Ú
CrossMapLRN2dé )Ú
functional)Úinit)ÚTensorÚSize)ÚUnionÚListÚTuplec sf e Zd ZU dZddddgZeee e
deeeed d
œ‡ fd d „
Z e e d
œdd„Z
dd„ Z‡ ZS )ÚLocalResponseNormau Applies local response normalization over an input signal composed
of several input planes, where channels occupy the second dimension.
Applies normalization across channels.
.. math::
b_{c} = a_{c}\left(k + \frac{\alpha}{n}
\sum_{c'=\max(0, c-n/2)}^{\min(N-1,c+n/2)}a_{c'}^2\right)^{-\beta}
Args:
size: amount of neighbouring channels used for normalization
alpha: multiplicative factor. Default: 0.0001
beta: exponent. Default: 0.75
k: additive factor. Default: 1
Shape:
- Input: :math:`(N, C, *)`
- Output: :math:`(N, C, *)` (same shape as input)
Examples::
>>> lrn = nn.LocalResponseNorm(2)
>>> signal_2d = torch.randn(32, 5, 24, 24)
>>> signal_4d = torch.randn(16, 5, 7, 7, 7, 7)
>>> output_2d = lrn(signal_2d)
>>> output_4d = lrn(signal_4d)
ÚsizeÚalphaÚbetaÚkç-C ëâ6?ç è?ç ð?N)r r r r Úreturnc s* t t| ƒjƒ || _|| _|| _|| _d S )N)Úsuperr Ú__init__r r r r )Úselfr r r r )Ú __class__© úJ/usr/local/lib64/python3.6/site-packages/torch/nn/modules/normalization.pyr / s
zLocalResponseNorm.__init__)Úinputr c C s t j|| j| j| j| jƒS )N)ÚFZlocal_response_normr r r r )r r r r r Úforward6 s zLocalResponseNorm.forwardc C s dj f | jŽS )Nz){size}, alpha={alpha}, beta={beta}, k={k})ÚformatÚ__dict__)r r r r Ú
extra_repr: s z LocalResponseNorm.extra_repr)r r r )Ú__name__Ú
__module__Ú __qualname__Ú__doc__Ú
__constants__Úintr Úfloatr r r r r r r" Ú
__classcell__r r )r r r
s
r c s\ e Zd ZU eeeedeeeeddœ‡ fdd„
Z e
e
dœd d
„Z e d œd d
„Z
‡ ZS )r ç-C ëâ6?ç è?r N)r r r r r c s* t t| ƒjƒ || _|| _|| _|| _d S )N)r r r r r r r )r r r r r )r r r r D s
zCrossMapLRN2d.__init__)r r c C s t j|| j| j| j| jƒS )N)Ú_cross_map_lrn2dÚapplyr r r r )r r r r r r K s zCrossMapLRN2d.forward)r c C s dj f | jŽS )Nz){size}, alpha={alpha}, beta={beta}, k={k})r r! )r r r r r" O s zCrossMapLRN2d.extra_repr)r+ r, r )r# r$ r% r( r r) r r r r r r Ústrr" r* r r )r r r > s
r c sz e Zd ZU dZdddgZeedf e e
de ee
dd œ‡ fd
d „
Z
dd œd
d„Zeedœdd„Zed œdd„Z‡ ZS )Ú LayerNormaõ Applies Layer Normalization over a mini-batch of inputs as described in
the paper `Layer Normalization
`__
.. math::
y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
The mean and standard-deviation are calculated over the last `D` dimensions, where `D`
is the dimension of :attr:`normalized_shape`. For example, if :attr:`normalized_shape`
is ``(3, 5)`` (a 2-dimensional shape), the mean and standard-deviation are computed over
the last 2 dimensions of the input (i.e. ``input.mean((-2, -1))``).
:math:`\gamma` and :math:`\beta` are learnable affine transform parameters of
:attr:`normalized_shape` if :attr:`elementwise_affine` is ``True``.
The standard-deviation is calculated via the biased estimator, equivalent to
`torch.var(input, unbiased=False)`.
.. note::
Unlike Batch Normalization and Instance Normalization, which applies
scalar scale and bias for each entire channel/plane with the
:attr:`affine` option, Layer Normalization applies per-element scale and
bias with :attr:`elementwise_affine`.
This layer uses statistics computed from input data in both training and
evaluation modes.
Args:
normalized_shape (int or list or torch.Size): input shape from an expected input
of size
.. math::
[* \times \text{normalized\_shape}[0] \times \text{normalized\_shape}[1]
\times \ldots \times \text{normalized\_shape}[-1]]
If a single integer is used, it is treated as a singleton list, and this module will
normalize over the last dimension which is expected to be of that specific size.
eps: a value added to the denominator for numerical stability. Default: 1e-5
elementwise_affine: a boolean value that when set to ``True``, this module
has learnable per-element affine parameters initialized to ones (for weights)
and zeros (for biases). Default: ``True``.
Attributes:
weight: the learnable weights of the module of shape
:math:`\text{normalized\_shape}` when :attr:`elementwise_affine` is set to ``True``.
The values are initialized to 1.
bias: the learnable bias of the module of shape
:math:`\text{normalized\_shape}` when :attr:`elementwise_affine` is set to ``True``.
The values are initialized to 0.
Shape:
- Input: :math:`(N, *)`
- Output: :math:`(N, *)` (same shape as input)
Examples::
>>> # NLP Example
>>> batch, sentence_length, embedding_dim = 20, 5, 10
>>> embedding = torch.randn(batch, sentence_length, embedding_dim)
>>> layer_norm = nn.LayerNorm(embedding_dim)
>>> # Activate module
>>> layer_norm(embedding)
>>>
>>> # Image Example
>>> N, C, H, W = 20, 5, 10, 10
>>> input = torch.randn(N, C, H, W)
>>> # Normalize over the last three dimensions (i.e. the channel and spatial dimensions)
>>> # as shown in the image below
>>> layer_norm = nn.LayerNorm([C, H, W])
>>> output = layer_norm(input)
.. image:: ../_static/img/nn/layer_norm.jpg
:scale: 50 %
Únormalized_shapeÚepsÚelementwise_affine.çñh㈵øä>TN)r1 r2 r3 r c s˜ ||dœ}t t| ƒjƒ t|tjƒr*|f}t|ƒ| _|| _|| _ | j rtt
t j | jf|Žƒ| _
t
t j | jf|Žƒ| _n| jdd ƒ | jdd ƒ | jƒ d S )N)ÚdeviceÚdtypeÚweightÚbias)r r0 r Ú
isinstanceÚnumbersÚIntegralÚtupler1 r2 r3 r ÚtorchÚemptyr7 r8 Úregister_parameterÚreset_parameters)r r1 r2 r3 r5 r6 Úfactory_kwargs)r r r r ¤ s
zLayerNorm.__init__)r c C s" | j r tj| jƒ tj| jƒ d S )N)r3 r Úones_r7 Úzeros_r8 )r r r r r@ · s zLayerNorm.reset_parameters)r r c C s t j|| j| j| j| jƒS )N)r Z
layer_normr1 r7 r8 r2 )r r r r r r ¼ s zLayerNorm.forwardc C s dj f | jŽS )NzF{normalized_shape}, eps={eps}, elementwise_affine={elementwise_affine})r r! )r r r r r" À s zLayerNorm.extra_repr)r4 TNN)r# r$ r% r& r' r
r( r1 r) r2 Úboolr3 Ú_shape_tr r@ r r r/ r" r* r r )r r r0 V s
H
r0 c sz e Zd ZU dZddddgZeee e
deeee
dd œ‡ fd
d „
Z dd œd
d„Z
eedœdd„Zed œdd„Z‡ ZS )Ú GroupNorma] Applies Group Normalization over a mini-batch of inputs as described in
the paper `Group Normalization `__
.. math::
y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
The input channels are separated into :attr:`num_groups` groups, each containing
``num_channels / num_groups`` channels. The mean and standard-deviation are calculated
separately over the each group. :math:`\gamma` and :math:`\beta` are learnable
per-channel affine transform parameter vectors of size :attr:`num_channels` if
:attr:`affine` is ``True``.
The standard-deviation is calculated via the biased estimator, equivalent to
`torch.var(input, unbiased=False)`.
This layer uses statistics computed from input data in both training and
evaluation modes.
Args:
num_groups (int): number of groups to separate the channels into
num_channels (int): number of channels expected in input
eps: a value added to the denominator for numerical stability. Default: 1e-5
affine: a boolean value that when set to ``True``, this module
has learnable per-channel affine parameters initialized to ones (for weights)
and zeros (for biases). Default: ``True``.
Shape:
- Input: :math:`(N, C, *)` where :math:`C=\text{num\_channels}`
- Output: :math:`(N, C, *)` (same shape as input)
Examples::
>>> input = torch.randn(20, 6, 10, 10)
>>> # Separate 6 channels into 3 groups
>>> m = nn.GroupNorm(3, 6)
>>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm)
>>> m = nn.GroupNorm(6, 6)
>>> # Put all 6 channels into a single group (equivalent with LayerNorm)
>>> m = nn.GroupNorm(1, 6)
>>> # Activating the module
>>> output = m(input)
Ú
num_groupsÚ num_channelsr2 Úaffineçñh㈵øä>TN)rG rH r2 rI r c s„ ||dœ}t t| ƒjƒ || _|| _|| _|| _| jr`ttj |f|Žƒ| _
ttj |f|Žƒ| _ n| j dd ƒ | j dd ƒ | j
ƒ d S )N)r5 r6 r7 r8 )r rF r rG rH r2 rI r r= r> r7 r8 r? r@ )r rG rH r2 rI r5 r6 rA )r r r r õ s
zGroupNorm.__init__)r c C s" | j r tj| jƒ tj| jƒ d S )N)rI r rB r7 rC r8 )r r r r r@ s zGroupNorm.reset_parameters)r r c C s t j|| j| j| j| jƒS )N)r Z
group_normrG r7 r8 r2 )r r r r r r s zGroupNorm.forwardc C s dj f | jŽS )Nz8{num_groups}, {num_channels}, eps={eps}, affine={affine})r r! )r r r r r" s zGroupNorm.extra_repr)rJ TNN)r# r$ r% r& r' r( rG rH r) r2 rD rI r r@ r r r/ r" r* r r )r r rF Å s
) rF )r= r: Ztorch.nn.parameterr Úmoduler Z
_functionsr r- Ú r r r r r
Útypingr r r
r r( rE r0 rF r r r r Ú s 1o