/usr/local/lib64/python3.6/site-packages/torch/nn/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/nn/__pycache__/init.cpython-36.pyc (17551B)
3
Eg`K @ s d dl Z d dlZd dlmZ d dlZdd Zdd Zdd Zd d
Zd d Zd>d
dZ d?ee
e
edddZ d@ee
e
edddZ dBee
e
e
e
edddZ
ee
edd d Zeed d d Zeed d!d"Zd#d$ ZdCd&d'Zd(d) ZdDee
ed*d+d,ZdEee
ed*d-d.Zd/d0 ZdFd3d4ZdGd5d6ZdHd7d8ZdId:d;Zd
.norm_cdf zjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect.)
stacklevel g @)minmax) warningswarnr r r Zerfinv_mul_r r Zadd_Zclamp_)r r
r r r r lur r r
_no_grad_trunc_normal_ s
r c C s t j | j|S Q R X d S )N)r r Zfill_)r valr r r
_no_grad_fill_9 s
r! c C s t j
| j S Q R X d S )N)r r zero_)r r r r
_no_grad_zero_> s
r# c C s dddddddg}| |ks"| dkr&d S | d
kr2dS | d
krDt jdS | dkr|dkrZd}n4t|t rpt|tszt|tr|}ntdj|t jdd |d S | dkrdS tdj| dS )a Return the recommended gain value for the given nonlinearity function.
The values are as follows:
================= ====================================================
nonlinearity gain
================= ====================================================
Linear / Identity :math:`1`
Conv{1,2,3}D :math:`1`
Sigmoid :math:`1`
Tanh :math:`\frac{5}{3}`
ReLU :math:`\sqrt{2}`
Leaky Relu :math:`\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}`
SELU :math:`\frac{3}{4}`
================= ====================================================
.. warning::
In order to implement `Self-Normalizing Neural Networks`_ ,
you should use ``nonlinearity='linear'`` instead of ``nonlinearity='selu'``.
This gives the initial weights a variance of ``1 / N``,
which is necessary to induce a stable fixed point in the forward pass.
In contrast, the default gain for ``SELU`` sacrifices the normalisation
effect for more stable gradient flow in rectangular layers.
Args:
nonlinearity: the non-linear function (`nn.functional` name)
param: optional parameter for the non-linear function
Examples:
>>> gain = nn.init.calculate_gain('leaky_relu', 0.2) # leaky_relu with negative_slope=0.2
.. _Self-Normalizing Neural Networks: https://papers.nips.cc/paper/2017/hash/5d44ee6f2c3f71b73125876103c8f6c4-Abstract.html
ZlinearZconv1dZconv2dZconv3dZconv_transpose1dZconv_transpose2dZconv_transpose3dZsigmoidr tanhg @ Zrelug @
leaky_reluNg{Gz?z$negative_slope {} not a valid numberr Zselug @ zUnsupported nonlinearity {}g?g ?)r r
isinstanceboolintfloat
ValueErrorformat) nonlinearityparamZ
linear_fnsZnegative_sloper r r
calculate_gainC s" !
r0 ?)r r r returnc C s t | ||S )ad Fills the input Tensor with values drawn from the uniform
distribution :math:`\mathcal{U}(a, b)`.
Args:
tensor: an n-dimensional `torch.Tensor`
a: the lower bound of the uniform distribution
b: the upper bound of the uniform distribution
Examples:
>>> w = torch.empty(3, 5)
>>> nn.init.uniform_(w)
)r )r r r r r r
r z s
r )r r
r r3 c C s t | ||S )az Fills the input Tensor with values drawn from the normal
distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)`.
Args:
tensor: an n-dimensional `torch.Tensor`
mean: the mean of the normal distribution
std: the standard deviation of the normal distribution
Examples:
>>> w = torch.empty(3, 5)
>>> nn.init.normal_(w)
)r )r r
r r r r
r s
r @)r r
r r r r3 c C s t | ||||S )a Fills the input Tensor with values drawn from a truncated
normal distribution. The values are effectively drawn from the
normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)`
with values outside :math:`[a, b]` redrawn until they are within
the bounds. The method used for generating the random values works
best when :math:`a \leq \text{mean} \leq b`.
Args:
tensor: an n-dimensional `torch.Tensor`
mean: the mean of the normal distribution
std: the standard deviation of the normal distribution
a: the minimum cutoff value
b: the maximum cutoff value
Examples:
>>> w = torch.empty(3, 5)
>>> nn.init.trunc_normal_(w)
)r )r r
r r r r r r
trunc_normal_ s r5 )r r r3 c C s
t | |S )zFills the input Tensor with the value :math:`\text{val}`.
Args:
tensor: an n-dimensional `torch.Tensor`
val: the value to fill the tensor with
Examples:
>>> w = torch.empty(3, 5)
>>> nn.init.constant_(w, 0.3)
)r! )r r r r r
constant_ s r6 )r r3 c C s
t | dS )zFills the input Tensor with the scalar value `1`.
Args:
tensor: an n-dimensional `torch.Tensor`
Examples:
>>> w = torch.empty(3, 5)
>>> nn.init.ones_(w)
g ?)r! )r r r r
ones_ s
r7 c C s t | S )zFills the input Tensor with the scalar value `0`.
Args:
tensor: an n-dimensional `torch.Tensor`
Examples:
>>> w = torch.empty(3, 5)
>>> nn.init.zeros_(w)
)r# )r r r r
zeros_ s
r8 c C sB | j dkrtdtj tj| j| | jd W dQ R X | S )a= Fills the 2-dimensional input `Tensor` with the identity
matrix. Preserves the identity of the inputs in `Linear` layers, where as
many inputs are preserved as possible.
Args:
tensor: a 2-dimensional `torch.Tensor`
Examples:
>>> w = torch.empty(3, 5)
>>> nn.init.eye_(w)
r z,Only tensors with 2 dimensions are supported)out
requires_gradN)
ndimensionr, r r eyeshaper: )r r r r
eye_ s
r> r c C s. | j }|d
krtd| j }|d | dkr8td|d | }t||d }tj | j xt|D ]}xt|D ]}|dkrd| || | || jdd f< q||dkrd| || | || jdd | jdd f< q|d| || | || jdd | jdd | jdd f< q|W qnW W d Q R X | S ) aA Fills the {3, 4, 5}-dimensional input `Tensor` with the Dirac
delta function. Preserves the identity of the inputs in `Convolutional`
layers, where as many input channels are preserved as possible. In case
of groups>1, each group of channels preserves identity
Args:
tensor: a {3, 4, 5}-dimensional `torch.Tensor`
groups (optional): number of groups in the conv layer (default: 1)
Examples:
>>> w = torch.empty(3, 16, 5, 5)
>>> nn.init.dirac_(w)
>>> w = torch.empty(3, 24, 5, 5)
>>> nn.init.dirac_(w, 3)
r% r' z5Only tensors with 3, 4, or 5 dimensions are supportedr z!dim 0 must be divisible by groupsr r N)r% r' r? )r; r, sizer r r r" range)r groups
dimensionssizesZout_chans_per_grpZmin_dimgdr r r
dirac_ s$
".JrG c C st | j }|dk rtd| jd}| jd}d}| j dkr\x | jdd D ] }||9 }qLW || }|| }||fS )Nr zNFan in and fan out can not be computed for tensor with fewer than 2 dimensionsr r )Zdimr, r@ r= )r rC Znum_input_fmapsZnum_output_fmapsZreceptive_field_sizesfan_infan_outr r r
_calculate_fan_in_and_fan_out s
rK )r gainr3 c C sB t | \}}|tjdt|| }tjd| }t| | |S )a Fills the input `Tensor` with values according to the method
described in `Understanding the difficulty of training deep feedforward
neural networks` - Glorot, X. & Bengio, Y. (2010), using a uniform
distribution. The resulting tensor will have values sampled from
:math:`\mathcal{U}(-a, a)` where
.. math::
a = \text{gain} \times \sqrt{\frac{6}{\text{fan\_in} + \text{fan\_out}}}
Also known as Glorot initialization.
Args:
tensor: an n-dimensional `torch.Tensor`
gain: an optional scaling factor
Examples:
>>> w = torch.empty(3, 5)
>>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain('relu'))
g @g @)rK r r r+ r )r rL rI rJ r r r r r
xavier_uniform_) s rM c C s2 t | \}}|tjdt|| }t| d|S )a Fills the input `Tensor` with values according to the method
described in `Understanding the difficulty of training deep feedforward
neural networks` - Glorot, X. & Bengio, Y. (2010), using a normal
distribution. The resulting tensor will have values sampled from
:math:`\mathcal{N}(0, \text{std}^2)` where
.. math::
\text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan\_in} + \text{fan\_out}}}
Also known as Glorot initialization.
Args:
tensor: an n-dimensional `torch.Tensor`
gain: an optional scaling factor
Examples:
>>> w = torch.empty(3, 5)
>>> nn.init.xavier_normal_(w)
g @g )rK r r r+ r )r rL rI rJ r r r r
xavier_normal_D s rN c C sD |j }ddg}||kr(tdj||t| \}}|dkr@|S |S )NrI rJ z+Mode {} not supported, please use one of {})lowerr, r- rK )r modeZ valid_modesrI rJ r r r
_calculate_correct_fan^ s rQ rI r& c
C sj d| j krtjd | S t| |}t||}|tj| }tjd| }tj | j | |S Q R X dS )a Fills the input `Tensor` with values according to the method
described in `Delving deep into rectifiers: Surpassing human-level
performance on ImageNet classification` - He, K. et al. (2015), using a
uniform distribution. The resulting tensor will have values sampled from
:math:`\mathcal{U}(-\text{bound}, \text{bound})` where
.. math::
\text{bound} = \text{gain} \times \sqrt{\frac{3}{\text{fan\_mode}}}
Also known as He initialization.
Args:
tensor: an n-dimensional `torch.Tensor`
a: the negative slope of the rectifier used after this layer (only
used with ``'leaky_relu'``)
mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'``
preserves the magnitude of the variance of the weights in the
forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the
backwards pass.
nonlinearity: the non-linear function (`nn.functional` name),
recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default).
Examples:
>>> w = torch.empty(3, 5)
>>> nn.init.kaiming_uniform_(w, mode='fan_in', nonlinearity='relu')
r z,Initializing zero-element tensors is a no-opg @N)
r= r r rQ r0 r r r r r )r r rP r. fanrL r boundr r r
kaiming_uniform_h s
rT c
C sZ d| j krtjd | S t| |}t||}|tj| }tj | j d|S Q R X dS )a Fills the input `Tensor` with values according to the method
described in `Delving deep into rectifiers: Surpassing human-level
performance on ImageNet classification` - He, K. et al. (2015), using a
normal distribution. The resulting tensor will have values sampled from
:math:`\mathcal{N}(0, \text{std}^2)` where
.. math::
\text{std} = \frac{\text{gain}}{\sqrt{\text{fan\_mode}}}
Also known as He initialization.
Args:
tensor: an n-dimensional `torch.Tensor`
a: the negative slope of the rectifier used after this layer (only
used with ``'leaky_relu'``)
mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'``
preserves the magnitude of the variance of the weights in the
forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the
backwards pass.
nonlinearity: the non-linear function (`nn.functional` name),
recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default).
Examples:
>>> w = torch.empty(3, 5)
>>> nn.init.kaiming_normal_(w, mode='fan_out', nonlinearity='relu')
r z,Initializing zero-element tensors is a no-opN)
r= r r rQ r0 r r r r r )r r rP r. rR rL r r r r
kaiming_normal_ s
rU c C s | j dk rtd| jd}| j | }| j||jdd}||k rN|j tjj |\}}tj
|d}|j }||9 }||k r|j tj | j
|j| | j| W dQ R X | S )a! Fills the input `Tensor` with a (semi) orthogonal matrix, as
described in `Exact solutions to the nonlinear dynamics of learning in deep
linear neural networks` - Saxe, A. et al. (2013). The input tensor must have
at least 2 dimensions, and for tensors with more than 2 dimensions the
trailing dimensions are flattened.
Args:
tensor: an n-dimensional `torch.Tensor`, where :math:`n \geq 2`
gain: optional scaling factor
Examples:
>>> w = torch.empty(3, 5)
>>> nn.init.orthogonal_(w)
r z4Only tensors with 2 or more dimensions are supportedr r N)r; r, r@ Znumelnewr Zt_r ZlinalgZqrZdiagsignr Zview_asZcopy_r ) r rL rowscolsZ flattenedqrrF phr r r
orthogonal_ s"
r] {Gz?c C s | j dkrtd| j\}}ttj|| }tj F | jd| x2t |D ]&}tj
|}|d| }d| ||f< qPW W dQ R X | S )aN Fills the 2D input `Tensor` as a sparse matrix, where the
non-zero elements will be drawn from the normal distribution
:math:`\mathcal{N}(0, 0.01)`, as described in `Deep learning via
Hessian-free optimization` - Martens, J. (2010).
Args:
tensor: an n-dimensional `torch.Tensor`
sparsity: The fraction of elements in each column to be set to zero
std: the standard deviation of the normal distribution used to generate
the non-zero values
Examples:
>>> w = torch.empty(3, 5)
>>> nn.init.sparse_(w, sparsity=0.1)
r z,Only tensors with 2 dimensions are supportedr N) r; r, r= r* r ceilr r r rA Zrandperm) r Zsparsityr rX rY Z num_zerosZcol_idxZ row_indicesZ zero_indicesr r r
sparse_ s
r` c s< j d d fdd}djd|_|_ |S )Nr c s t jdjdd | |S )Nz4nn.init.{} is now deprecated in favor of nn.init.{}.r )r )r r r- )argskwargs)methnew_nameold_namer r
deprecated_init s z(_make_deprecate..deprecated_initz
{old_name}(...)
.. warning::
This method is now deprecated in favor of :func:`torch.nn.init.{new_name}`.
See :func:`~torch.nn.init.{new_name}` for details.)re rd )__name__r- __doc__)rc rf r )rc rd re r
_make_deprecate s rj )N)r1 r2 )r1 r2 )r1 r2 rk r4 )r )r2 )r2 )r rI r& )r rI r& )r )r^ )'r r r r r r r r! r# r0 r+ r r r5 r6 r7 r8 r> rG rK rM rN rQ rT rU r] r` rj uniformnormalZconstantr< ZdiracZxavier_uniformZ
xavier_normalZkaiming_uniformZkaiming_normalZ
orthogonalsparser r r r
sJ #
7
+
&
%
)