/usr/local/lib64/python3.6/site-packages/torch/nn/modules/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/nn/modules/__pycache__/activation.cpython-36.pyc (47635B)
3
Eg @ sD d dl Z d dlmZmZ d dlZd dlmZ ddlmZ d dlm Z m
Z
m Z d dl m
Z
ddlmZ d d
lmZ G d d d eZG d
d deZG dd deZG dd deZG dd deZG dd deZG dd deZG dd deZG dd d eZG d d d eZ G d d d eZ G d!d" d"eZ G d#d$ d$eZ G d%d& d&eZ G d'd( d(eZ!G d)d* d*eZ"G d+d, d,eZ#G d-d. d.eZ$G d/d0 d0eZ%G d1d2 d2eZ&G d3d4 d4eZ'G d5d6 d6eZ(G d7d8 d8eZ)G d9d: d:eZ*G d;d< d
d>eZ,G d?d@ d@eZ-G dAdB dBeZ.G dCdD dDeZ/dS )E N)OptionalTuple)Tensor ) NonDynamicallyQuantizableLinear) constant_xavier_normal_xavier_uniform_) Parameter)Module )
functionalc s^ e Zd ZU dZdddgZeee deeedd fdd
Z
e e d
d d Z d
d Z
ZS ) Thresholda Thresholds each element of the input Tensor.
Threshold is defined as:
.. math::
y =
\begin{cases}
x, &\text{ if } x > \text{threshold} \\
\text{value}, &\text{ otherwise }
\end{cases}
Args:
threshold: The value to threshold at
value: The value to replace with
inplace: can optionally do the operation in-place. Default: ``False``
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
Examples::
>>> m = nn.Threshold(0.1, 20)
>>> input = torch.randn(2)
>>> output = m(input)
thresholdvalueinplaceFN)r r r returnc s$ t t| j || _|| _|| _d S )N)superr __init__r r r )selfr r r ) __class__ G/usr/local/lib64/python3.6/site-packages/torch/nn/modules/activation.pyr . s zThreshold.__init__)inputr c C s t j|| j| j| jS )N)Fr r r )r r r r r forward5 s zThreshold.forwardc C s | j r
dnd}dj| j| j|S )Nz, inplace=True zthreshold={}, value={}{})r formatr r )r inplace_strr r r
extra_repr8 s zThreshold.extra_repr)F)__name__
__module__ __qualname____doc__
__constants__floatr r boolr r r r r
__classcell__r r )r r r
s
r c sR e Zd ZU dZdgZed
ed fdd
Zeeddd Z e
d
d d Z Z S )ReLUa Applies the rectified linear unit function element-wise:
:math:`\text{ReLU}(x) = (x)^+ = \max(0, x)`
Args:
inplace: can optionally do the operation in-place. Default: ``False``
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/ReLU.png
Examples::
>>> m = nn.ReLU()
>>> input = torch.randn(2)
>>> output = m(input)
An implementation of CReLU - https://arxiv.org/abs/1603.05201
>>> m = nn.ReLU()
>>> input = torch.randn(2).unsqueeze(0)
>>> output = torch.cat((m(input),m(-input)))
r F)r c s t t| j || _d S )N)r r( r r )r r )r r r r ] s z
ReLU.__init__)r r c C s t j|| jdS )N)r )r Zrelur )r r r r r r a s z ReLU.forward)r c C s | j r
dnd}|S )Nz inplace=Truer )r )r r r r r r d s zReLU.extra_repr)F)
r r! r" r# r$ r& r r r r strr r' r r )r r r( ? s
r( c s\ e Zd ZU dZdddgZeee deeed fd
d
Z
e e d d
dZ dd Z
ZS )RReLUa Applies the randomized leaky rectified liner unit function, element-wise,
as described in the paper:
`Empirical Evaluation of Rectified Activations in Convolutional Network`_.
The function is defined as:
.. math::
\text{RReLU}(x) =
\begin{cases}
x & \text{if } x \geq 0 \\
ax & \text{ otherwise }
\end{cases}
where :math:`a` is randomly sampled from uniform distribution
:math:`\mathcal{U}(\text{lower}, \text{upper})`.
See: https://arxiv.org/pdf/1505.00853.pdf
Args:
lower: lower bound of the uniform distribution. Default: :math:`\frac{1}{8}`
upper: upper bound of the uniform distribution. Default: :math:`\frac{1}{3}`
inplace: can optionally do the operation in-place. Default: ``False``
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
Examples::
>>> m = nn.RReLU(0.1, 0.3)
>>> input = torch.randn(2)
>>> output = m(input)
.. _`Empirical Evaluation of Rectified Activations in Convolutional Network`:
https://arxiv.org/abs/1505.00853
lowerupperr g ? F)r+ r, r c s$ t t| j || _|| _|| _d S )N)r r* r r+ r, r )r r+ r, r )r r r r s zRReLU.__init__)r r c C s t j|| j| j| j| jS )N)r Zrrelur+ r, trainingr )r r r r r r s z
RReLU.forwardc C s | j r
dnd}dj| j| j|S )Nz, inplace=Truer zlower={}, upper={}{})r r r+ r, )r r r r r r s zRReLU.extra_repr ?UUUUUU?)r0 r1 F)r r! r" r# r$ r% r+ r, r& r r r r r r' r r )r r r* i s
%
r* c sp e Zd ZU dZdddgZeee deeee
e e
e dd fd d
Z e e d d d
Z
edddZ ZS )Hardtanha/ Applies the HardTanh function element-wise
HardTanh is defined as:
.. math::
\text{HardTanh}(x) = \begin{cases}
1 & \text{ if } x > 1 \\
-1 & \text{ if } x < -1 \\
x & \text{ otherwise } \\
\end{cases}
The range of the linear region :math:`[-1, 1]` can be adjusted using
:attr:`min_val` and :attr:`max_val`.
Args:
min_val: minimum value of the linear region range. Default: -1
max_val: maximum value of the linear region range. Default: 1
inplace: can optionally do the operation in-place. Default: ``False``
Keyword arguments :attr:`min_value` and :attr:`max_value`
have been deprecated in favor of :attr:`min_val` and :attr:`max_val`.
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/Hardtanh.png
Examples::
>>> m = nn.Hardtanh(-2, 2)
>>> input = torch.randn(2)
>>> output = m(input)
min_valmax_valr ?FN)r3 r4 r min_value max_valuer c s` t t| j |d k r$tjd |}|d k r:tjd |}|| _|| _|| _| j| jks\td S )Nz>keyword argument min_value is deprecated and rename to min_valz>keyword argument max_value is deprecated and rename to max_val) r r2 r warningswarnr3 r4 r AssertionError)r r3 r4 r r6 r7 )r r r r s
zHardtanh.__init__)r r c C s t j|| j| j| jS )N)r Zhardtanhr3 r4 r )r r r r r r s zHardtanh.forward)r c C s | j r
dnd}dj| j| j|S )Nz, inplace=Truer zmin_val={}, max_val={}{})r r r3 r4 )r r r r r r s zHardtanh.extra_repr )r; r5 FNN)r r! r" r# r$ r% r3 r4 r& r r r r r r) r r' r r )r r r2 s
"
r2 c s6 e Zd ZdZd ed fdd
ZedddZ ZS )
ReLU6a Applies the element-wise function:
.. math::
\text{ReLU6}(x) = \min(\max(0,x), 6)
Args:
inplace: can optionally do the operation in-place. Default: ``False``
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/ReLU6.png
Examples::
>>> m = nn.ReLU6()
>>> input = torch.randn(2)
>>> output = m(input)
F)r c s t t| jdd| d S )Ng g @)r r< r )r r )r r r r s zReLU6.__init__)r c C s | j r
dnd}|S )Nz inplace=Truer )r )r r r r r r s zReLU6.extra_repr)F) r r! r" r# r& r r) r r' r r )r r r< s r< c @ s e Zd ZdZeedddZdS )Sigmoida Applies the element-wise function:
.. math::
\text{Sigmoid}(x) = \sigma(x) = \frac{1}{1 + \exp(-x)}
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/Sigmoid.png
Examples::
>>> m = nn.Sigmoid()
>>> input = torch.randn(2)
>>> output = m(input)
)r r c C s
t j|S )N)torchZsigmoid)r r r r r r " s zSigmoid.forwardN)r r! r" r# r r r r r r r= s r= c sF e Zd ZU dZdgZed edd fdd
Zeedd d
Z Z
S ) HardsigmoidaS Applies the element-wise function:
.. math::
\text{Hardsigmoid}(x) = \begin{cases}
0 & \text{if~} x \le -3, \\
1 & \text{if~} x \ge +3, \\
x / 6 + 1 / 2 & \text{otherwise}
\end{cases}
Args:
inplace: can optionally do the operation in-place. Default: ``False``
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
Examples::
>>> m = nn.Hardsigmoid()
>>> input = torch.randn(2)
>>> output = m(input)
r FN)r r c s t t| j || _d S )N)r r? r r )r r )r r r r A s zHardsigmoid.__init__)r r c C s t j|| jS )N)r Z hardsigmoidr )r r r r r r E s zHardsigmoid.forward)F) r r! r" r# r$ r& r r r r r' r r )r r r? & s
r? c @ s e Zd ZdZeedddZdS )Tanha Applies the element-wise function:
.. math::
\text{Tanh}(x) = \tanh(x) = \frac{\exp(x) - \exp(-x)} {\exp(x) + \exp(-x)}
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/Tanh.png
Examples::
>>> m = nn.Tanh()
>>> input = torch.randn(2)
>>> output = m(input)
)r r c C s
t j|S )N)r> tanh)r r r r r r \ s z Tanh.forwardN)r r! r" r# r r r r r r r@ I s r@ c sR e Zd ZU dZdgZed
ed fdd
Zeeddd Z e
d
d d Z Z S )SiLUa Applies the Sigmoid Linear Unit (SiLU) function, element-wise.
The SiLU function is also known as the swish function.
.. math::
\text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.}
.. note::
See `Gaussian Error Linear Units (GELUs) `_
where the SiLU (Sigmoid Linear Unit) was originally coined, and see
`Sigmoid-Weighted Linear Units for Neural Network Function Approximation
in Reinforcement Learning `_ and `Swish:
a Self-Gated Activation Function `_
where the SiLU was experimented with later.
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
Examples::
>>> m = nn.SiLU()
>>> input = torch.randn(2)
>>> output = m(input)
r F)r c s t t| j || _d S )N)r rB r r )r r )r r r r { s z
SiLU.__init__)r r c C s t j|| jdS )N)r )r Zsilur )r r r r r r s z SiLU.forward)r c C s | j r
dnd}|S )Nz inplace=Truer )r )r r r r r r s zSiLU.extra_repr)F)
r r! r" r# r$ r& r r r r r) r r' r r )r r rB _ s
rB c sR e Zd ZU dZdgZed
ed fdd
Zeeddd Z e
d
d d Z Z S )MishaA Applies the Mish function, element-wise.
Mish: A Self Regularized Non-Monotonic Neural Activation Function.
.. math::
\text{Mish}(x) = x * \text{Tanh}(\text{Softplus}(x))
.. note::
See `Mish: A Self Regularized Non-Monotonic Neural Activation Function `_
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
Examples::
>>> m = nn.Mish()
>>> input = torch.randn(2)
>>> output = m(input)
r F)r c s t t| j || _d S )N)r rC r r )r r )r r r r s z
Mish.__init__)r r c C s t j|| jdS )N)r )r Zmishr )r r r r r r s z Mish.forward)r c C s | j r
dnd}|S )Nz inplace=Truer )r )r r r r r r s zMish.extra_repr)F)
r r! r" r# r$ r& r r r r r) r r' r r )r r rC s
rC c sF e Zd ZU dZdgZed edd fdd
Zeedd d
Z Z
S ) Hardswisha Applies the hardswish function, element-wise, as described in the paper:
`Searching for MobileNetV3`_.
.. math::
\text{Hardswish}(x) = \begin{cases}
0 & \text{if~} x \le -3, \\
x & \text{if~} x \ge +3, \\
x \cdot (x + 3) /6 & \text{otherwise}
\end{cases}
Args:
inplace: can optionally do the operation in-place. Default: ``False``
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
Examples::
>>> m = nn.Hardswish()
>>> input = torch.randn(2)
>>> output = m(input)
.. _`Searching for MobileNetV3`:
https://arxiv.org/abs/1905.02244
r FN)r r c s t t| j || _d S )N)r rD r r )r r )r r r r s zHardswish.__init__)r r c C s t j|| jS )N)r Z hardswishr )r r r r r r s zHardswish.forward)F) r r! r" r# r$ r& r r r r r' r r )r r rD s
rD c s\ e Zd ZU dZddgZeedeedd fdd
Z e
e
d
d d Z e d
ddZ
ZS )ELUa Applies the element-wise function:
.. math::
\text{ELU}(x) = \begin{cases}
x, & \text{ if } x > 0\\
\alpha * (\exp(x) - 1), & \text{ if } x \leq 0
\end{cases}
Args:
alpha: the :math:`\alpha` value for the ELU formulation. Default: 1.0
inplace: can optionally do the operation in-place. Default: ``False``
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/ELU.png
Examples::
>>> m = nn.ELU()
>>> input = torch.randn(2)
>>> output = m(input)
alphar ?FN)rF r r c s t t| j || _|| _d S )N)r rE r rF r )r rF r )r r r r s z ELU.__init__)r r c C s t j|| j| jS )N)r ZelurF r )r r r r r r s z ELU.forward)r c C s | j r
dnd}dj| j|S )Nz, inplace=Truer z
alpha={}{})r r rF )r r r r r r s zELU.extra_repr)rG F)r r! r" r# r$ r% rF r& r r r r r) r r' r r )r r rE s
rE c s\ e Zd ZU dZddgZeedeedd fdd
Z e
e
d
d d Z e d
ddZ
ZS )CELUa. Applies the element-wise function:
.. math::
\text{CELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x/\alpha) - 1))
More details can be found in the paper `Continuously Differentiable Exponential Linear Units`_ .
Args:
alpha: the :math:`\alpha` value for the CELU formulation. Default: 1.0
inplace: can optionally do the operation in-place. Default: ``False``
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/CELU.png
Examples::
>>> m = nn.CELU()
>>> input = torch.randn(2)
>>> output = m(input)
.. _`Continuously Differentiable Exponential Linear Units`:
https://arxiv.org/abs/1704.07483
rF r ?FN)rF r r c s t t| j || _|| _d S )N)r rH r rF r )r rF r )r r r r s z
CELU.__init__)r r c C s t j|| j| jS )N)r ZcelurF r )r r r r r r s z CELU.forward)r c C s | j r
dnd}dj| j|S )Nz, inplace=Truer z
alpha={}{})r r rF )r r r r r r ! s zCELU.extra_repr)rI F)r r! r" r# r$ r% rF r& r r r r r) r r' r r )r r rH s
rH c sT e Zd ZU dZdgZededd fdd
Zeedd d
Z e
d d d
Z Z S )SELUay Applied element-wise, as:
.. math::
\text{SELU}(x) = \text{scale} * (\max(0,x) + \min(0, \alpha * (\exp(x) - 1)))
with :math:`\alpha = 1.6732632423543772848170429916717` and
:math:`\text{scale} = 1.0507009873554804934193349852946`.
.. warning::
When using ``kaiming_normal`` or ``kaiming_normal_`` for initialisation,
``nonlinearity='linear'`` should be used instead of ``nonlinearity='selu'``
in order to get `Self-Normalizing Neural Networks`_.
See :func:`torch.nn.init.calculate_gain` for more information.
More details can be found in the paper `Self-Normalizing Neural Networks`_ .
Args:
inplace (bool, optional): can optionally do the operation in-place. Default: ``False``
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/SELU.png
Examples::
>>> m = nn.SELU()
>>> input = torch.randn(2)
>>> output = m(input)
.. _Self-Normalizing Neural Networks: https://arxiv.org/abs/1706.02515
r FN)r r c s t t| j || _d S )N)r rJ r r )r r )r r r r K s z
SELU.__init__)r r c C s t j|| jS )N)r Zselur )r r r r r r O s z SELU.forward)r c C s | j r
dnd}|S )Nz inplace=Truer )r )r r r r r r R s zSELU.extra_repr)F)
r r! r" r# r$ r& r r r r r) r r' r r )r r rJ & s
!rJ c sT e Zd ZU dZdgZededd fdd
Zeedd d
Z e
d d d
Z Z S )GLUa3 Applies the gated linear unit function
:math:`{GLU}(a, b)= a \otimes \sigma(b)` where :math:`a` is the first half
of the input matrices and :math:`b` is the second half.
Args:
dim (int): the dimension on which to split the input. Default: -1
Shape:
- Input: :math:`(\ast_1, N, \ast_2)` where `*` means, any number of additional
dimensions
- Output: :math:`(\ast_1, M, \ast_2)` where :math:`M=N/2`
Examples::
>>> m = nn.GLU()
>>> input = torch.randn(4, 2)
>>> output = m(input)
dimr N)rL r c s t t| j || _d S )N)r rK r rL )r rL )r r r r m s z GLU.__init__)r r c C s t j|| jS )N)r ZglurL )r r r r r r q s z GLU.forward)r c C s dj | jS )Nzdim={})r rL )r r r r r t s zGLU.extra_repr)rM )
r r! r" r# r$ intrL r r r r) r r' r r )r r rK W s
rK c @ s e Zd ZdZeedddZdS )GELUa Applies the Gaussian Error Linear Units function:
.. math:: \text{GELU}(x) = x * \Phi(x)
where :math:`\Phi(x)` is the Cumulative Distribution Function for Gaussian Distribution.
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/GELU.png
Examples::
>>> m = nn.GELU()
>>> input = torch.randn(2)
>>> output = m(input)
)r r c C s
t j|S )N)r Zgelu)r r r r r r s z GELU.forwardN)r r! r" r# r r r r r r rO x s rO c sT e Zd ZU dZdgZededd fdd
Zeedd d
Z e
d d d
Z Z S )
Hardshrinka Applies the hard shrinkage function element-wise:
.. math::
\text{HardShrink}(x) =
\begin{cases}
x, & \text{ if } x > \lambda \\
x, & \text{ if } x < -\lambda \\
0, & \text{ otherwise }
\end{cases}
Args:
lambd: the :math:`\lambda` value for the Hardshrink formulation. Default: 0.5
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/Hardshrink.png
Examples::
>>> m = nn.Hardshrink()
>>> input = torch.randn(2)
>>> output = m(input)
lambd ?N)rQ r c s t t| j || _d S )N)r rP r rQ )r rQ )r r r r s zHardshrink.__init__)r r c C s t j|| jS )N)r Z
hardshrinkrQ )r r r r r r s zHardshrink.forward)r c C s dj | jS )Nz{})r rQ )r r r r r s zHardshrink.extra_repr)rR )
r r! r" r# r$ r% rQ r r r r) r r' r r )r r rP s
rP c s\ e Zd ZU dZddgZeedeedd fdd
Z e
e
d
d d Z e d
ddZ
ZS ) LeakyReLUa? Applies the element-wise function:
.. math::
\text{LeakyReLU}(x) = \max(0, x) + \text{negative\_slope} * \min(0, x)
or
.. math::
\text{LeakyRELU}(x) =
\begin{cases}
x, & \text{ if } x \geq 0 \\
\text{negative\_slope} \times x, & \text{ otherwise }
\end{cases}
Args:
negative_slope: Controls the angle of the negative slope. Default: 1e-2
inplace: can optionally do the operation in-place. Default: ``False``
Shape:
- Input: :math:`(*)` where `*` means, any number of additional
dimensions
- Output: :math:`(*)`, same shape as the input
.. image:: ../scripts/activation_images/LeakyReLU.png
Examples::
>>> m = nn.LeakyReLU(0.1)
>>> input = torch.randn(2)
>>> output = m(input)
r negative_slope{Gz?FN)rT r r c s t t| j || _|| _d S )N)r rS r rT r )r rT r )r r r r s zLeakyReLU.__init__)r r c C s t j|| j| jS )N)r Z
leaky_relurT r )r r r r r r s zLeakyReLU.forward)r c C s | j r
dnd}dj| j|S )Nz, inplace=Truer znegative_slope={}{})r r rT )r r r r r r s zLeakyReLU.extra_repr)rU F)r r! r" r# r$ r& r r% rT r r r r) r r' r r )r r rS s
rS c @ s e Zd ZdZeedddZdS )
LogSigmoida Applies the element-wise function:
.. math::
\text{LogSigmoid}(x) = \log\left(\frac{ 1 }{ 1 + \exp(-x)}\right)
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/LogSigmoid.png
Examples::
>>> m = nn.LogSigmoid()
>>> input = torch.randn(2)
>>> output = m(input)
)r r c C s
t j|S )N)r Z
logsigmoid)r r r r r r s zLogSigmoid.forwardN)r r! r" r# r r r r r r rV s rV c s\ e Zd ZU dZddgZeedeedd fdd
Ze e d
d d Z
e d
ddZ Z
S )Softplusap Applies the element-wise function:
.. math::
\text{Softplus}(x) = \frac{1}{\beta} * \log(1 + \exp(\beta * x))
SoftPlus is a smooth approximation to the ReLU function and can be used
to constrain the output of a machine to always be positive.
For numerical stability the implementation reverts to the linear function
when :math:`input \times \beta > threshold`.
Args:
beta: the :math:`\beta` value for the Softplus formulation. Default: 1
threshold: values above this revert to a linear function. Default: 20
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/Softplus.png
Examples::
>>> m = nn.Softplus()
>>> input = torch.randn(2)
>>> output = m(input)
betar r N)rX r r c s t t| j || _|| _d S )N)r rW r rX r )r rX r )r r r r s zSoftplus.__init__)r r c C s t j|| j| jS )N)r ZsoftplusrX r )r r r r r r % s zSoftplus.forward)r c C s dj | j| jS )Nzbeta={}, threshold={})r rX r )r r r r r ( s zSoftplus.extra_repr)r rY )r r! r" r# r$ rN rX r r r r r) r r' r r )r r rW s
rW c sT e Zd ZU dZdgZededd fdd
Zeedd d
Z e
d d d
Z Z S )
Softshrinka Applies the soft shrinkage function elementwise:
.. math::
\text{SoftShrinkage}(x) =
\begin{cases}
x - \lambda, & \text{ if } x > \lambda \\
x + \lambda, & \text{ if } x < -\lambda \\
0, & \text{ otherwise }
\end{cases}
Args:
lambd: the :math:`\lambda` (must be no less than zero) value for the Softshrink formulation. Default: 0.5
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/Softshrink.png
Examples::
>>> m = nn.Softshrink()
>>> input = torch.randn(2)
>>> output = m(input)
rQ ?N)rQ r c s t t| j || _d S )N)r rZ r rQ )r rQ )r r r r I s zSoftshrink.__init__)r r c C s t j|| jS )N)r Z
softshrinkrQ )r r r r r r M s zSoftshrink.forward)r c C s
t | jS )N)r) rQ )r r r r r P s zSoftshrink.extra_repr)r[ )
r r! r" r# r$ r% rQ r r r r) r r' r r )r r rZ , s
rZ c s e Zd ZU dZdgZeej eej ddd fdd
Z
d
d Z fd d
Z deeeee e
ee eeee f dddZ ZS )MultiheadAttentiona Allows the model to jointly attend to information
from different representation subspaces.
See `Attention Is All You Need `_.
.. math::
\text{MultiHead}(Q, K, V) = \text{Concat}(head_1,\dots,head_h)W^O
where :math:`head_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)`.
Args:
embed_dim: Total dimension of the model.
num_heads: Number of parallel attention heads. Note that ``embed_dim`` will be split
across ``num_heads`` (i.e. each head will have dimension ``embed_dim // num_heads``).
dropout: Dropout probability on ``attn_output_weights``. Default: ``0.0`` (no dropout).
bias: If specified, adds bias to input / output projection layers. Default: ``True``.
add_bias_kv: If specified, adds bias to the key and value sequences at dim=0. Default: ``False``.
add_zero_attn: If specified, adds a new batch of zeros to the key and value sequences at dim=1.
Default: ``False``.
kdim: Total number of features for keys. Default: ``None`` (uses ``kdim=embed_dim``).
vdim: Total number of features for values. Default: ``None`` (uses ``vdim=embed_dim``).
batch_first: If ``True``, then the input and output tensors are provided
as (batch, seq, feature). Default: ``False`` (seq, batch, feature).
Examples::
>>> multihead_attn = nn.MultiheadAttention(embed_dim, num_heads)
>>> attn_output, attn_output_weights = multihead_attn(query, key, value)
batch_first TFN)r c
s |
| d} t t| j || _|d k r*|n|| _|d k r<|n|| _| j|koT| j|k| _|| _|| _| | _ || | _
| j
| | jkst d| jdkrt t
j||ff| | _t t
j|| jff| | _t t
j|| jff| | _| jdd n@t t
jd| |ff| | _| jdd | jdd | jdd |rPt t
jd| f| | _n | jd d t||fd
|i| | _|rt t
jd d |ff| | _t t
jd d |ff| | _n d | _| _|| _| j d S ) N)devicedtypez(embed_dim must be divisible by num_headsFin_proj_weightr.
q_proj_weight
k_proj_weight
v_proj_weight in_proj_biasbiasr )r r\ r embed_dimkdimvdim_qkv_same_embed_dim num_headsdropoutr] Zhead_dimr: r
r> emptyrb rc rd Zregister_parameterra re r out_projbias_kbias_v
add_zero_attn_reset_parameters)
r rg rk rl rf Z add_bias_kvrq rh ri r] r_ r` factory_kwargs)r r r r u s<
zMultiheadAttention.__init__c C s | j rt| j n t| j t| j t| j | jd k rTt| jd t| jj d | j
d k rht | j
| j d k r|t | j d S )Ng )
rj r ra rb rc rd re r rn rf ro r rp )r r r r rr s
z$MultiheadAttention._reset_parametersc s$ d|krd|d< t t| j| d S )Nrj T)r r\ __setstate__)r state)r r r rt s z MultiheadAttention.__setstate__)querykeyr key_padding_mask need_weights attn_maskr c C s | j r dd |||fD \}}}| jstj|||| j| j| j| j| j| j | j
| j | j j
| j j| j|||d| j| j| jd\}}nJtj|||| j| j| j| j| j| j | j
| j | j j
| j j| j|||d\}}| j r|jdd|fS ||fS dS ) a\
Args:
query: Query embeddings of shape :math:`(L, N, E_q)` when ``batch_first=False`` or :math:`(N, L, E_q)`
when ``batch_first=True``, where :math:`L` is the target sequence length, :math:`N` is the batch size,
and :math:`E_q` is the query embedding dimension ``embed_dim``. Queries are compared against
key-value pairs to produce the output. See "Attention Is All You Need" for more details.
key: Key embeddings of shape :math:`(S, N, E_k)` when ``batch_first=False`` or :math:`(N, S, E_k)` when
``batch_first=True``, where :math:`S` is the source sequence length, :math:`N` is the batch size, and
:math:`E_k` is the key embedding dimension ``kdim``. See "Attention Is All You Need" for more details.
value: Value embeddings of shape :math:`(S, N, E_v)` when ``batch_first=False`` or :math:`(N, S, E_v)` when
``batch_first=True``, where :math:`S` is the source sequence length, :math:`N` is the batch size, and
:math:`E_v` is the value embedding dimension ``vdim``. See "Attention Is All You Need" for more details.
key_padding_mask: If specified, a mask of shape :math:`(N, S)` indicating which elements within ``key``
to ignore for the purpose of attention (i.e. treat as "padding"). Binary and byte masks are supported.
For a binary mask, a ``True`` value indicates that the corresponding ``key`` value will be ignored for
the purpose of attention. For a byte mask, a non-zero value indicates that the corresponding ``key``
value will be ignored.
need_weights: If specified, returns ``attn_output_weights`` in addition to ``attn_outputs``.
Default: ``True``.
attn_mask: If specified, a 2D or 3D mask preventing attention to certain positions. Must be of shape
:math:`(L, S)` or :math:`(N\cdot\text{num\_heads}, L, S)`, where :math:`N` is the batch size,
:math:`L` is the target sequence length, and :math:`S` is the source sequence length. A 2D mask will be
broadcasted across the batch while a 3D mask allows for a different mask for each entry in the batch.
Binary, byte, and float masks are supported. For a binary mask, a ``True`` value indicates that the
corresponding position is not allowed to attend. For a byte mask, a non-zero value indicates that the
corresponding position is not allowed to attend. For a float mask, the mask values will be added to
the attention weight.
Outputs:
- **attn_output** - Attention outputs of shape :math:`(L, N, E)` when ``batch_first=False`` or
:math:`(N, L, E)` when ``batch_first=True``, where :math:`L` is the target sequence length, :math:`N` is
the batch size, and :math:`E` is the embedding dimension ``embed_dim``.
- **attn_output_weights** - Attention output weights of shape :math:`(N, L, S)`, where :math:`N` is the batch
size, :math:`L` is the target sequence length, and :math:`S` is the source sequence length. Only returned
when ``need_weights=True``.
c S s g | ]}|j d dqS )r r ) transpose).0xr r r
s z.MultiheadAttention.forward..T)r/ rx ry rz Zuse_separate_proj_weightrb rc rd )r/ rx ry rz r r N)r] rj r Z multi_head_attention_forwardrg rk ra re ro rp rq rl rn weightrf r/ rb rc rd r{ ) r rv rw r rx ry rz Z attn_outputZattn_output_weightsr r r r s0 & zMultiheadAttention.forward) r^ TFFNNFNN)NTN)r r! r" r# r$ r r> r ro rp r rr rt r& r r r' r r )r r r\ T s
) r\ c sV e Zd ZU dZdgZedeedd fdd
Ze e d d
d Z
e d d
dZ Z
S )PReLUa Applies the element-wise function:
.. math::
\text{PReLU}(x) = \max(0,x) + a * \min(0,x)
or
.. math::
\text{PReLU}(x) =
\begin{cases}
x, & \text{ if } x \geq 0 \\
ax, & \text{ otherwise }
\end{cases}
Here :math:`a` is a learnable parameter. When called without arguments, `nn.PReLU()` uses a single
parameter :math:`a` across all input channels. If called with `nn.PReLU(nChannels)`,
a separate :math:`a` is used for each input channel.
.. note::
weight decay should not be used when learning :math:`a` for good performance.
.. note::
Channel dim is the 2nd dim of input. When input has dims < 2, then there is
no channel dim and the number of channels = 1.
Args:
num_parameters (int): number of :math:`a` to learn.
Although it takes an int as input, there is only two values are legitimate:
1, or the number of channels at input. Default: 1
init (float): the initial value of :math:`a`. Default: 0.25
Shape:
- Input: :math:`( *)` where `*` means, any number of additional
dimensions.
- Output: :math:`(*)`, same shape as the input.
Attributes:
weight (Tensor): the learnable weights of shape (:attr:`num_parameters`).
.. image:: ../scripts/activation_images/PReLU.png
Examples::
>>> m = nn.PReLU()
>>> input = torch.randn(2)
>>> output = m(input)
num_parametersr ?N)r initr c s< ||d}|| _ tt| j ttj|f|j|| _d S )N)r_ r` ) r r r r r
r> rm Zfill_r )r r r r_ r` rs )r r r r , s
zPReLU.__init__)r r c C s t j|| jS )N)r Zprelur )r r r r r r 3 s z
PReLU.forward)r c C s dj | jS )Nznum_parameters={})r r )r r r r r 6 s zPReLU.extra_repr)r r NN)r r! r" r# r$ rN r r% r r r r) r r' r r )r r r s
0 r c @ s e Zd ZdZeedddZdS )Softsigna Applies the element-wise function:
.. math::
\text{SoftSign}(x) = \frac{x}{ 1 + |x|}
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/Softsign.png
Examples::
>>> m = nn.Softsign()
>>> input = torch.randn(2)
>>> output = m(input)
)r r c C s
t j|S )N)r Zsoftsign)r r r r r r M s zSoftsign.forwardN)r r! r" r# r r r r r r r : s r c @ s e Zd ZdZeedddZdS )
Tanhshrinka Applies the element-wise function:
.. math::
\text{Tanhshrink}(x) = x - \tanh(x)
Shape:
- Input: :math:`(*)`, where :math:`*` means any number of dimensions.
- Output: :math:`(*)`, same shape as the input.
.. image:: ../scripts/activation_images/Tanhshrink.png
Examples::
>>> m = nn.Tanhshrink()
>>> input = torch.randn(2)
>>> output = m(input)
)r r c C s
t j|S )N)r Z
tanhshrink)r r r r r r d s zTanhshrink.forwardN)r r! r" r# r r r r r r r Q s r c s^ e Zd ZU dZdgZee dee dd fdd
Zdd Z e
e
d d
d Z d d
Z Z
S )Softmina4 Applies the Softmin function to an n-dimensional input Tensor
rescaling them so that the elements of the n-dimensional output Tensor
lie in the range `[0, 1]` and sum to 1.
Softmin is defined as:
.. math::
\text{Softmin}(x_{i}) = \frac{\exp(-x_i)}{\sum_j \exp(-x_j)}
Shape:
- Input: :math:`(*)` where `*` means, any number of additional
dimensions
- Output: :math:`(*)`, same shape as the input
Args:
dim (int): A dimension along which Softmin will be computed (so every slice
along dim will sum to 1).
Returns:
a Tensor of the same dimension and shape as the input, with
values in the range [0, 1]
Examples::
>>> m = nn.Softmin()
>>> input = torch.randn(2, 3)
>>> output = m(input)
rL N)rL r c s t t| j || _d S )N)r r r rL )r rL )r r r r s zSoftmin.__init__c C s | j j| t| ds d | _d S )NrL )__dict__updatehasattrrL )r ru r r r rt s
zSoftmin.__setstate__)r r c C s t j|| jddS )N ) _stacklevel)r ZsoftminrL )r r r r r r s zSoftmin.forwardc C s dj | jdS )Nz dim={dim})rL )r rL )r r r r r s zSoftmin.extra_repr)N)r r! r" r# r$ r rN rL r rt r r r r' r r )r r r h s
r c sd e Zd ZU dZdgZee dee dd fdd
Zdd Z e
e
d d
d Z e d d
dZ
ZS )Softmaxa Applies the Softmax function to an n-dimensional input Tensor
rescaling them so that the elements of the n-dimensional output Tensor
lie in the range [0,1] and sum to 1.
Softmax is defined as:
.. math::
\text{Softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_j \exp(x_j)}
When the input Tensor is a sparse tensor then the unspecifed
values are treated as ``-inf``.
Shape:
- Input: :math:`(*)` where `*` means, any number of additional
dimensions
- Output: :math:`(*)`, same shape as the input
Returns:
a Tensor of the same dimension and shape as the input with
values in the range [0, 1]
Args:
dim (int): A dimension along which Softmax will be computed (so every slice
along dim will sum to 1).
.. note::
This module doesn't work directly with NLLLoss,
which expects the Log to be computed between the Softmax and itself.
Use `LogSoftmax` instead (it's faster and has better numerical properties).
Examples::
>>> m = nn.Softmax(dim=1)
>>> input = torch.randn(2, 3)
>>> output = m(input)
rL N)rL r c s t t| j || _d S )N)r r r rL )r rL )r r r r s zSoftmax.__init__c C s | j j| t| ds d | _d S )NrL )r r r rL )r ru r r r rt s
zSoftmax.__setstate__)r r c C s t j|| jddS )Nr )r )r softmaxrL )r r r r r r s zSoftmax.forward)r c C s dj | jdS )Nz dim={dim})rL )r rL )r r r r r s zSoftmax.extra_repr)N)r r! r" r# r$ r rN rL r rt r r r) r r' r r )r r r s
%r c @ s e Zd ZdZeedddZdS ) Softmax2da| Applies SoftMax over features to each spatial location.
When given an image of ``Channels x Height x Width``, it will
apply `Softmax` to each location :math:`(Channels, h_i, w_j)`
Shape:
- Input: :math:`(N, C, H, W)` or :math:`(C, H, W)`.
- Output: :math:`(N, C, H, W)` or :math:`(C, H, W)` (same shape as input)
Returns:
a Tensor of the same dimension and shape as the input with
values in the range [0, 1]
Examples::
>>> m = nn.Softmax2d()
>>> # you softmax over the 2nd dimension
>>> input = torch.randn(2, 3, 12, 13)
>>> output = m(input)
)r r c C s0 |j dks |j dks tdtj|dddS )N r. z-Softmax2d requires a 3D or 4D tensor as inputr )r )rL r: r r )r r r r r r s zSoftmax2d.forwardN)r r! r" r# r r r r r r r s r c s^ e Zd ZU dZdgZee dee dd fdd
Zdd Z e
e
d d
d Z d d
Z Z
S )
LogSoftmaxa Applies the :math:`\log(\text{Softmax}(x))` function to an n-dimensional
input Tensor. The LogSoftmax formulation can be simplified as:
.. math::
\text{LogSoftmax}(x_{i}) = \log\left(\frac{\exp(x_i) }{ \sum_j \exp(x_j)} \right)
Shape:
- Input: :math:`(*)` where `*` means, any number of additional
dimensions
- Output: :math:`(*)`, same shape as the input
Args:
dim (int): A dimension along which LogSoftmax will be computed.
Returns:
a Tensor of the same dimension and shape as the input with
values in the range [-inf, 0)
Examples::
>>> m = nn.LogSoftmax()
>>> input = torch.randn(2, 3)
>>> output = m(input)
rL N)rL r c s t t| j || _d S )N)r r r rL )r rL )r r r r s zLogSoftmax.__init__c C s | j j| t| ds d | _d S )NrL )r r r rL )r ru r r r rt s
zLogSoftmax.__setstate__)r r c C s t j|| jddS )Nr )r )r Z log_softmaxrL )r r r r r r s zLogSoftmax.forwardc C s dj | jdS )Nz dim={dim})rL )r rL )r r r r r s zLogSoftmax.extra_repr)N)r r! r" r# r$ r rN rL r rt r r r r' r r )r r r s
r )0r8 typingr r r> r Zlinearr Z
torch.nn.initr r r Ztorch.nn.parameterr
moduler r r
r r r( r* r2 r< r= r? r@ rB rC rD rE rH rJ rK rO rP rS rV rW rZ r\ r r r r r r r r r r r sL 2*?H #'"(*,1!(2,( %B/9