/usr/local/lib64/python3.6/site-packages/torch/nn/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/nn/__pycache__/functional.cpython-36.pyc (161648B)
3
Eg# @ sd d Z ddlmZmZmZmZ ddlZddlZddlZddlm Z ddl
m Z m Z ddl
mZmZ ddlmZmZmZmZmZ dd lmZmZmZmZ d
d lm Z d
d lm Z d
d
l m Z d
dl!m"Z"m#Z#m$Z$m%Z% ej&Z&e ej'dj(f eed Z'e ej)dj(f eed Z)e ej*dj(f eed Z*e ej+dj(f eed Z+e ej,dj(f eed Z,e ej-dj(f eed Z-e ej.dZ.e ej/d Z/e ej0j1j2d Z2e ej0j1j3d Z3dse&ee4 eee4 eee5 e6ee& ee&e&f d d!d"Z7dte&ee4 eee4 eee5 e6ee& e&d d#d$Z8ed%d&d e7e8e9d'd(Z:due&ee4 eee4 eee5 e6ee& ee&e&f d d)d*Z;dve&ee4 eee4 eee5 e6ee& e&d d+d,Z
dxe&ee4 eee4 ee4 ee4 e6e6e&d.d1d2Z?ed%d3d e>e?e9d4d(Z@dye&ee4 eee4 ee4 ee4 e6e6ee&e&f d.d5d6ZAdze&ee4 eee4 ee4 ee4 e6e6e&d.d7d8ZBed%d3d eAeBe9d9d(ZCd{e&ee4 eee4 ee4 ee4 e6e6ee&e&f d.d:d;ZDd|e&ee4 eee4 ee4 ee4 e6e6e&d.dd(ZFe&ee4 ee4 ee4 eee4 ee4 d?d@dAZGd}e&e&ee4 eee4 ee4 eee4 e&dBdCdDZHd~e&e&ee4 eee4 ee4 eee4 e&dBdEdFZIde&e&ee4 eee4 ee4 eee4 e&dBdGdHZJde&e5e4eee4 e6e&dIdJdKZKde&e5e4eee4 e6e&dIdLdMZLde&ee4 e6ee&e&f dNdOdPZMde&ee4 e6e&dNdQdRZNed%dd eMeNe9dSd(ZOde&ee4 e6ee&e&f dNdTdUZPde&ee4 e6e&dNdVdWZQed%dd ePeQe9dXd(ZRde&ee4 e6ee&e&f dNdYdZZSde&ee4 e6e&dNd[d\ZTed%dd eSeTe9d]d(ZUe ejVd^ZVe&ee4 e&d_d`daZWe&ee4 e&d_dbdcZXde&e5e6e6e&dfdgdhZYde&e5e6e6e&dfdidjZZde&e5e6e6e&dfdkdlZ[de&e5e6e6e&dfdmdnZ\de&e5e6e6e&dfdodpZ]de&e5e5e6e&dqdrdsZ^e^Z_e e j`dtZ`de&e6e&dudvdwZae ejbdxZbde&e4e&dydzd{Zcde&e5e5e6e&d}d~dZde ej0j1jedZede&e6e&duddZfde&e5e6e&dddZge ej0j1jhdZhde&e6e&duddZie ejjdZjde&e5e6e&dddZke ejldZlde&e5e6e&dddZme ej0j1jndZne&e&e&dddZode&e5e5e6e6e&dddZpe ejqdZqe ej0j1jrdZsdd Ztde&e5e&dddZudd Zvdd Zwe ej0j1jxdZxeye4e4e4dddZzde&ee4 e4ee4 e&dddZ{de&ee4 e4ee4 e&dddZ|de&e5e6e5e4e&dddZ}de&ee4 e4ee4 e&dddZ~e ej0j1jdZdd Zdd Zde&e6e&duddZde&e&ee& e&dddZde&e&e&ee& e&dddZde&e6e&duddZde&e6e&duddĄZde&e6e&duddƄZe&e&e5e5e&dǜddɄZde&e&ee4 ee5 e5e6e6e&d˜dd̈́Zde&e&ee& ee5 e5e6eye6ee& e6ee4 e&dϜ ddфZej j(f ee_ ee4 ddҜddԄZde&ee& ee& ee& ee& e6e5e5e&dל ddلZee4 ddҜddۄZde&ee& ee& ee& ee& e6e5e5e&dܜ ddބZde&ee4 ee& ee& e5e&dߜddZde&e4ee& ee& e5e&dddZde&e4e5e5e5e&dddZde&e&e&e&e4eye6e&dddZej j(f ee_ de&e&ee& ee6 e4ee6 eye&dddZde&e&e6e6ee6 e5ee6 eye&d ddZde&e&e&e6e5eye&dddZde&e&ee6 ee6 eye6e&dddZde&e&ee& ee6 e4ee6 eye5e&d ddZde&e&ee& ee6 ee6 eye&d ddZde&e&ee& ee6 ee6 eyee& e&dddZde&e&ee6 ee6 eye5e&dddZde&e&eye5e&d d
d Zde&e&ee6 ee6 eye&d d
dZde&e&ee6 ee6 eye&d ddZde&e&e&e5ee6 ee6 eye&dddZde&e&e5ee6 ee6 eye&dddZde&e&ee6 ee6 eye&d ddZde&e&ee6 ee6 eye&d ddZde&e&ee& ee6 ee6 eye&d dd Zde&e&e&e5ee6 ee6 eye&dd d Zde&e&e4e5ee& ee6 ee6 eye&d d d!Ze ejd"Ze ejd#Ze ejd$Zede&ee4 ee5 eyee6 e&d&d'd(Zede&eee4 ee5 eyee6 e&d&d)d(ZdŐd*d(Zej j(f ee_ ede&ee4 eee5 eyee6 ee6 e&d+d,d-Zede&eee4 eee5 eyee6 ee6 e&d+d.d-Zede&ee4 ee5 eyee6 ee6 e&d+d/d-Zede&eee4 ee5 eyee6 ee6 e&d+d0d-Zde&ee4 eee5 eyee6 ee6 e&d+d1d-Zej j(f ee_ ede&ee4 ee5 e&d2d3d4Zede&eee4 ee5 e&d2d5d4Zd͐d6d4Zej j(f ee_ ede&ee4 ee5 e&d2d7d8Zede&eee4 ee5 e&d2d9d8Zede&ee4 eee5 e&d2d:d8Zede&eee4 eee5 e&d2d;d8ZdҐd<d8Zej j(f ee_ dd
dd=Zdd
dd>Zde&e&eyeyee6 e&d@dAdBZde&ee4 ee6 e&dCdDdEZde&ee4 eye5e&dGdHdIZeZde&e&e5e5e6e&dJdKdLZe ejdMZe ejdNZe ej0j1jdOZde&e&e&e5e5e5e6ee6 ee6 eye&dP dQdRZdd|d dΐdSe&e&e&eee&e&ge&f e5e6eye&dTdUdVZde&e5e4e5ee& e&dXdYdZZee4 eyeydd[d\d]Zde&ee4 ee4 ee4 ee4 e&d^d_d`Zde&ee4 ee4 ee4 ee4 ee4 e&dadbdcZe&ee4 e&dddedfZde&e&e&e&ee& ee& dgdhdiZde&e&e&e&e&e&ee& ee& ee& ee&e&e&f dj
dkdlZde&e&e&ee& e5ee&e&f dmdndoZde&e&e&e4e4e&ee& ee& ee& e6e5e&ee& e6ee& e6ee& e6ee& ee& ee& ee& ee& ee&ee& f dpdqdrZdS ( zFunctional interface )CallableListOptionalTupleN)_VF) _infer_size _add_docstr)reproducibility_notes
tf32_notes )boolean_dispatch _overloadBroadcastingList1BroadcastingList2BroadcastingList3)has_torch_functionhas_torch_function_unaryhas_torch_function_variadichandle_torch_function )
_reduction)grad)utils)_single_pair_triple_list_with_defaulta
conv1d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1) -> Tensor
Applies a 1D convolution over an input signal composed of several input
planes.
{tf32_note}
See :class:`~torch.nn.Conv1d` for details and output shape.
Note:
{cudnn_reproducibility_note}
ad
Args:
input: input tensor of shape :math:`(\text{minibatch} , \text{in\_channels} , iW)`
weight: filters of shape :math:`(\text{out\_channels} , \frac{\text{in\_channels}}{\text{groups}} , kW)`
bias: optional bias of shape :math:`(\text{out\_channels})`. Default: ``None``
stride: the stride of the convolving kernel. Can be a single number or
a one-element tuple `(sW,)`. Default: 1
padding: implicit paddings on both sides of the input. Can be a string {'valid', 'same'},
single number or a one-element tuple `(padW,)`. Default: 0
``padding='valid'`` is the same as no padding. ``padding='same'`` pads
the input so the output has the shape as the input. However, this mode
doesn't support any stride values other than 1.
.. warning::
For ``padding='same'``, if the ``weight`` is even-length and
``dilation`` is odd in any dimension, a full :func:`pad` operation
may be needed internally. Lowering performance.
dilation: the spacing between kernel elements. Can be a single number or
a one-element tuple `(dW,)`. Default: 1
groups: split input into groups, :math:`\text{in\_channels}` should be divisible by
the number of groups. Default: 1
Examples::
>>> inputs = torch.randn(33, 16, 30)
>>> filters = torch.randn(20, 16, 5)
>>> F.conv1d(inputs, filters)
a
conv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1) -> Tensor
Applies a 2D convolution over an input image composed of several input
planes.
{tf32_note}
See :class:`~torch.nn.Conv2d` for details and output shape.
Note:
{cudnn_reproducibility_note}
a
Args:
input: input tensor of shape :math:`(\text{minibatch} , \text{in\_channels} , iH , iW)`
weight: filters of shape :math:`(\text{out\_channels} , \frac{\text{in\_channels}}{\text{groups}} , kH , kW)`
bias: optional bias tensor of shape :math:`(\text{out\_channels})`. Default: ``None``
stride: the stride of the convolving kernel. Can be a single number or a
tuple `(sH, sW)`. Default: 1
padding: implicit paddings on both sides of the input. Can be a string {'valid', 'same'},
single number or a tuple `(padH, padW)`. Default: 0
``padding='valid'`` is the same as no padding. ``padding='same'`` pads
the input so the output has the shape as the input. However, this mode
doesn't support any stride values other than 1.
.. warning::
For ``padding='same'``, if the ``weight`` is even-length and
``dilation`` is odd in any dimension, a full :func:`pad` operation
may be needed internally. Lowering performance.
dilation: the spacing between kernel elements. Can be a single number or
a tuple `(dH, dW)`. Default: 1
groups: split input into groups, :math:`\text{in\_channels}` should be divisible by the
number of groups. Default: 1
Examples::
>>> # With square kernels and equal stride
>>> filters = torch.randn(8, 4, 3, 3)
>>> inputs = torch.randn(1, 4, 5, 5)
>>> F.conv2d(inputs, filters, padding=1)
a
conv3d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1) -> Tensor
Applies a 3D convolution over an input image composed of several input
planes.
{tf32_note}
See :class:`~torch.nn.Conv3d` for details and output shape.
Note:
{cudnn_reproducibility_note}
a
Args:
input: input tensor of shape :math:`(\text{minibatch} , \text{in\_channels} , iT , iH , iW)`
weight: filters of shape :math:`(\text{out\_channels} , \frac{\text{in\_channels}}{\text{groups}} , kT , kH , kW)`
bias: optional bias tensor of shape :math:`(\text{out\_channels})`. Default: None
stride: the stride of the convolving kernel. Can be a single number or a
tuple `(sT, sH, sW)`. Default: 1
padding: implicit paddings on both sides of the input. Can be a string {'valid', 'same'},
single number or a tuple `(padT, padH, padW)`. Default: 0
``padding='valid'`` is the same as no padding. ``padding='same'`` pads
the input so the output has the shape as the input. However, this mode
doesn't support any stride values other than 1.
.. warning::
For ``padding='same'``, if the ``weight`` is even-length and
``dilation`` is odd in any dimension, a full :func:`pad` operation
may be needed internally. Lowering performance.
dilation: the spacing between kernel elements. Can be a single number or
a tuple `(dT, dH, dW)`. Default: 1
groups: split input into groups, :math:`\text{in\_channels}` should be divisible by
the number of groups. Default: 1
Examples::
>>> filters = torch.randn(33, 16, 3, 3, 3)
>>> inputs = torch.randn(20, 16, 50, 10, 20)
>>> F.conv3d(inputs, filters)
az
conv_transpose1d(input, weight, bias=None, stride=1, padding=0, output_padding=0, groups=1, dilation=1) -> Tensor
Applies a 1D transposed convolution operator over an input signal
composed of several input planes, sometimes also called "deconvolution".
{tf32_note}
See :class:`~torch.nn.ConvTranspose1d` for details and output shape.
Note:
{cudnn_reproducibility_note}
al
Args:
input: input tensor of shape :math:`(\text{minibatch} , \text{in\_channels} , iW)`
weight: filters of shape :math:`(\text{in\_channels} , \frac{\text{out\_channels}}{\text{groups}} , kW)`
bias: optional bias of shape :math:`(\text{out\_channels})`. Default: None
stride: the stride of the convolving kernel. Can be a single number or a
tuple ``(sW,)``. Default: 1
padding: ``dilation * (kernel_size - 1) - padding`` zero-padding will be added to both
sides of each dimension in the input. Can be a single number or a tuple
``(padW,)``. Default: 0
output_padding: additional size added to one side of each dimension in the
output shape. Can be a single number or a tuple ``(out_padW)``. Default: 0
groups: split input into groups, :math:`\text{in\_channels}` should be divisible by the
number of groups. Default: 1
dilation: the spacing between kernel elements. Can be a single number or
a tuple ``(dW,)``. Default: 1
Examples::
>>> inputs = torch.randn(20, 16, 50)
>>> weights = torch.randn(16, 33, 5)
>>> F.conv_transpose1d(inputs, weights)
ay
conv_transpose2d(input, weight, bias=None, stride=1, padding=0, output_padding=0, groups=1, dilation=1) -> Tensor
Applies a 2D transposed convolution operator over an input image
composed of several input planes, sometimes also called "deconvolution".
{tf32_note}
See :class:`~torch.nn.ConvTranspose2d` for details and output shape.
Note:
{cudnn_reproducibility_note}
a
Args:
input: input tensor of shape :math:`(\text{minibatch} , \text{in\_channels} , iH , iW)`
weight: filters of shape :math:`(\text{in\_channels} , \frac{\text{out\_channels}}{\text{groups}} , kH , kW)`
bias: optional bias of shape :math:`(\text{out\_channels})`. Default: None
stride: the stride of the convolving kernel. Can be a single number or a
tuple ``(sH, sW)``. Default: 1
padding: ``dilation * (kernel_size - 1) - padding`` zero-padding will be added to both
sides of each dimension in the input. Can be a single number or a tuple
``(padH, padW)``. Default: 0
output_padding: additional size added to one side of each dimension in the
output shape. Can be a single number or a tuple ``(out_padH, out_padW)``.
Default: 0
groups: split input into groups, :math:`\text{in\_channels}` should be divisible by the
number of groups. Default: 1
dilation: the spacing between kernel elements. Can be a single number or
a tuple ``(dH, dW)``. Default: 1
Examples::
>>> # With square kernels and equal stride
>>> inputs = torch.randn(1, 4, 5, 5)
>>> weights = torch.randn(4, 8, 3, 3)
>>> F.conv_transpose2d(inputs, weights, padding=1)
ax
conv_transpose3d(input, weight, bias=None, stride=1, padding=0, output_padding=0, groups=1, dilation=1) -> Tensor
Applies a 3D transposed convolution operator over an input image
composed of several input planes, sometimes also called "deconvolution"
{tf32_note}
See :class:`~torch.nn.ConvTranspose3d` for details and output shape.
Note:
{cudnn_reproducibility_note}
a
Args:
input: input tensor of shape :math:`(\text{minibatch} , \text{in\_channels} , iT , iH , iW)`
weight: filters of shape :math:`(\text{in\_channels} , \frac{\text{out\_channels}}{\text{groups}} , kT , kH , kW)`
bias: optional bias of shape :math:`(\text{out\_channels})`. Default: None
stride: the stride of the convolving kernel. Can be a single number or a
tuple ``(sT, sH, sW)``. Default: 1
padding: ``dilation * (kernel_size - 1) - padding`` zero-padding will be added to both
sides of each dimension in the input. Can be a single number or a tuple
``(padT, padH, padW)``. Default: 0
output_padding: additional size added to one side of each dimension in the
output shape. Can be a single number or a tuple
``(out_padT, out_padH, out_padW)``. Default: 0
groups: split input into groups, :math:`\text{in\_channels}` should be divisible by the
number of groups. Default: 1
dilation: the spacing between kernel elements. Can be a single number or
a tuple `(dT, dH, dW)`. Default: 1
Examples::
>>> inputs = torch.randn(20, 16, 50, 10, 20)
>>> weights = torch.randn(16, 33, 3, 3, 3)
>>> F.conv_transpose3d(inputs, weights)
a
Applies a 1-dimensional sequence convolution over an input sequence.
Input and output dimensions are (Time, Batch, Channels) - hence TBC.
Args:
input: input tensor of shape :math:`(\text{sequence length} \times batch \times \text{in\_channels})`
weight: filter of shape (:math:`\text{kernel width} \times \text{in\_channels} \times \text{out\_channels}`)
bias: bias of shape (:math:`\text{out\_channels}`)
pad: number of timesteps to pad. Default: 0
az
avg_pool1d(input, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True) -> Tensor
Applies a 1D average pooling over an input signal composed of several
input planes.
See :class:`~torch.nn.AvgPool1d` for details and output shape.
Args:
input: input tensor of shape :math:`(\text{minibatch} , \text{in\_channels} , iW)`
kernel_size: the size of the window. Can be a single number or a
tuple `(kW,)`
stride: the stride of the window. Can be a single number or a tuple
`(sW,)`. Default: :attr:`kernel_size`
padding: implicit zero paddings on both sides of the input. Can be a
single number or a tuple `(padW,)`. Default: 0
ceil_mode: when True, will use `ceil` instead of `floor` to compute the
output shape. Default: ``False``
count_include_pad: when True, will include the zero-padding in the
averaging calculation. Default: ``True``
Examples::
>>> # pool of square window of size=3, stride=2
>>> input = torch.tensor([[[1, 2, 3, 4, 5, 6, 7]]], dtype=torch.float32)
>>> F.avg_pool1d(input, kernel_size=3, stride=2)
tensor([[[ 2., 4., 6.]]])
a
avg_pool2d(input, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True, divisor_override=None) -> Tensor
Applies 2D average-pooling operation in :math:`kH \times kW` regions by step size
:math:`sH \times sW` steps. The number of output features is equal to the number of
input planes.
See :class:`~torch.nn.AvgPool2d` for details and output shape.
Args:
input: input tensor :math:`(\text{minibatch} , \text{in\_channels} , iH , iW)`
kernel_size: size of the pooling region. Can be a single number or a
tuple `(kH, kW)`
stride: stride of the pooling operation. Can be a single number or a
tuple `(sH, sW)`. Default: :attr:`kernel_size`
padding: implicit zero paddings on both sides of the input. Can be a
single number or a tuple `(padH, padW)`. Default: 0
ceil_mode: when True, will use `ceil` instead of `floor` in the formula
to compute the output shape. Default: ``False``
count_include_pad: when True, will include the zero-padding in the
averaging calculation. Default: ``True``
divisor_override: if specified, it will be used as divisor, otherwise
size of the pooling region will be used. Default: None
a
avg_pool3d(input, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True, divisor_override=None) -> Tensor
Applies 3D average-pooling operation in :math:`kT \times kH \times kW` regions by step
size :math:`sT \times sH \times sW` steps. The number of output features is equal to
:math:`\lfloor\frac{\text{input planes}}{sT}\rfloor`.
See :class:`~torch.nn.AvgPool3d` for details and output shape.
Args:
input: input tensor :math:`(\text{minibatch} , \text{in\_channels} , iT \times iH , iW)`
kernel_size: size of the pooling region. Can be a single number or a
tuple `(kT, kH, kW)`
stride: stride of the pooling operation. Can be a single number or a
tuple `(sT, sH, sW)`. Default: :attr:`kernel_size`
padding: implicit zero paddings on both sides of the input. Can be a
single number or a tuple `(padT, padH, padW)`, Default: 0
ceil_mode: when True, will use `ceil` instead of `floor` in the formula
to compute the output shape
count_include_pad: when True, will include the zero-padding in the
averaging calculation
divisor_override: if specified, it will be used as divisor, otherwise
size of the pooling region will be used. Default: None
F)input kernel_size output_size output_ratioreturn_indices_random_samplesreturnc
C s t | |r&tt| |f| |||||dS |dkr>|dkr>td|dkr|dk sRtt|}t| jd |d t| jd
|d g}|dkr| j dkrdn| jd}t j
|| jd d| j | j d}t j
jj| |||S ) ab Applies 2D fractional max pooling over an input signal composed of several input planes.
Fractional MaxPooling is described in detail in the paper `Fractional MaxPooling`_ by Ben Graham
The max-pooling operation is applied in :math:`kH \times kW` regions by a stochastic
step size determined by the target output size.
The number of output features is equal to the number of input planes.
Args:
kernel_size: the size of the window to take a max over.
Can be a single number :math:`k` (for a square kernel of :math:`k \times k`)
or a tuple `(kH, kW)`
output_size: the target output size of the image of the form :math:`oH \times oW`.
Can be a tuple `(oH, oW)` or a single number :math:`oH` for a square image :math:`oH \times oH`
output_ratio: If one wants to have an output size as a ratio of the input size, this option can be given.
This has to be a number or tuple in the range (0, 1)
return_indices: if ``True``, will return the indices along with the outputs.
Useful to pass to :func:`~torch.nn.functional.max_unpool2d`.
Examples::
>>> input = torch.randn(20, 16, 50, 32)
>>> # pool of square window of size=3, and target output size 13x12
>>> F.fractional_max_pool2d(input, 3, output_size=(13, 12))
>>> # pool of square window and target output size being half of input image size
>>> F.fractional_max_pool2d(input, 3, output_ratio=(0.5, 0.5))
.. _Fractional MaxPooling:
http://arxiv.org/abs/1412.6071
)r r r! r" NzRfractional_max_pool2d requires specifying either an output_size or an output_ratior r r )dtypedevice)r r "fractional_max_pool2d_with_indices
ValueErrorAssertionErrorr intsizedimtorchrandr% r& _C_nnfractional_max_pool2d)r r r r r! r"
_output_ratioZn_batch r6 ?/usr/local/lib64/python3.6/site-packages/torch/nn/functional.pyr* s( $
, r* c
C s< t | |r&tt| |f| |||||dS t| |||||d S )N)r r r! r" r )r r r4 r* )r r r r r! r" r6 r6 r7 _fractional_max_pool2d s
r8 r! r4 )arg_nameZ arg_indexdefaultif_trueif_false module_name func_namec
C s t | |r&tt| |f| |||||dS |dkr>|dkr>td|dkr|dk sRtt|}t| jd|d t| jd|d t| jd|d g}|dkrtj | jd| jdd| j
| j d }tj j
j| |||S )
a Applies 3D fractional max pooling over an input signal composed of several input planes.
Fractional MaxPooling is described in detail in the paper `Fractional MaxPooling`_ by Ben Graham
The max-pooling operation is applied in :math:`kT \times kH \times kW` regions by a stochastic
step size determined by the target output size.
The number of output features is equal to the number of input planes.
Args:
kernel_size: the size of the window to take a max over.
Can be a single number :math:`k` (for a square kernel of :math:`k \times k \times k`)
or a tuple `(kT, kH, kW)`
output_size: the target output size of the form :math:`oT \times oH \times oW`.
Can be a tuple `(oT, oH, oW)` or a single number :math:`oH` for a cubic output
:math:`oH \times oH \times oH`
output_ratio: If one wants to have an output size as a ratio of the input size, this option can be given.
This has to be a number or tuple in the range (0, 1)
return_indices: if ``True``, will return the indices along with the outputs.
Useful to pass to :func:`~torch.nn.functional.max_unpool3d`.
Examples::
>>> input = torch.randn(20, 16, 50, 32, 16)
>>> # pool of cubic window of size=3, and target output size 13x12x11
>>> F.fractional_max_pool3d(input, 3, output_size=(13, 12, 11))
>>> # pool of cubic window and target output size being half of input size
>>> F.fractional_max_pool3d(input, 3, output_ratio=(0.5, 0.5, 0.5))
.. _Fractional MaxPooling:
http://arxiv.org/abs/1412.6071
)r r r! r" NzRfractional_max_pool3d requires specifying either an output_size or an output_ratior r r$ r r9 )r% r& )r r "fractional_max_pool3d_with_indicesr+ r, r r- r. r0 r1 r% r& r2 r3 fractional_max_pool3d)r r r r r! r" r5 r6 r6 r7 r@ s* %
$r@ c
C s< t | |r&tt| |f| |||||dS t| |||||d S )N)r r r! r" r )r r rA r@ )r r r r r! r" r6 r6 r7 _fractional_max_pool3d7 s
rB rA )r r stridepaddingdilation ceil_moder! r# c C sR t | r$tt| f| ||||||d S |dkr>tjjtt g }tj| |||||S )zApplies a 1D max pooling over an input signal composed of several input
planes.
See :class:`~torch.nn.MaxPool1d` for details.
)rC rD rE rF r! N)r r max_pool1d_with_indicesr0 jitannotater r- )r r rC rD rE rF r! r6 r6 r7 rG Y s
rG c C sR t | r$tt| f| ||||||d S |d kr>tjjtt g }tj| |||||S )N)rC rD rE rF r! )r r
max_pool1dr0 rH rI r r- )r r rC rD rE rF r! r6 r6 r7 _max_pool1dw s rK rJ c C sV t | r$tt| f| ||||||d S |dkr>tjjtt g }tjj j| |||||S )zApplies a 2D max pooling over an input signal composed of several input
planes.
See :class:`~torch.nn.MaxPool2d` for details.
)rC rD rE rF r! N)
r r max_pool2d_with_indicesr0 rH rI r r- r2 r3 )r r rC rD rE rF r! r6 r6 r7 rM s
rM c C sR t | r$tt| f| ||||||d S |d kr>tjjtt g }tj| |||||S )N)rC rD rE rF r! )r r
max_pool2dr0 rH rI r r- )r r rC rD rE rF r! r6 r6 r7 _max_pool2d s rO rN c C sV t | r$tt| f| ||||||d S |dkr>tjjtt g }tjj j| |||||S )zApplies a 3D max pooling over an input signal composed of several input
planes.
See :class:`~torch.nn.MaxPool3d` for details.
)rC rD rE rF r! N)
r r max_pool3d_with_indicesr0 rH rI r r- r2 r3 )r r rC rD rE rF r! r6 r6 r7 rP s
rP c C sR t | r$tt| f| ||||||d S |d kr>tjjtt g }tj| |||||S )N)rC rD rE rF r! )r r
max_pool3dr0 rH rI r r- )r r rC rD rE rF r! r6 r6 r7 _max_pool3d s rR rQ )r r rC rD r r# c C s8 | j }tjjtt g }xLtt|D ]<}|j|t| | d || || d|| q(W |d krv|}nt|t|d kr|dd }t|t|krt dj
t|t|d t|xftt|D ]V}|| || } || || }
| || k o|
k n st dj
||| |
qW |}|S )Nr r zZoutput_size should be a sequence containing {} or {} elements, but it has a length of '{}'z;invalid output_size "{}" (dim {} must be between {} and {})) r. r0 rH rI r r- rangelenappendr+ format) r r rC rD r Z
input_sizeZ default_sizedretZmin_sizemax_sizer6 r6 r7 _unpool_output_size s* < rZ )r indicesr rC rD r r# c
C s t | r"tt| f| |||||dS t|}|dk r Tensor
Applies a 1D adaptive average pooling over an input signal composed of
several input planes.
See :class:`~torch.nn.AdaptiveAvgPool1d` for details and output shape.
Args:
output_size: the target output size (single integer)
)r r r# c C s6 t | rtt| f| |S t|| j }tjjj| |S )a"
Applies a 2D adaptive average pooling over an input signal composed of
several input planes.
See :class:`~torch.nn.AdaptiveAvgPool2d` for details and output shape.
Args:
output_size: the target output size (single integer or
double-integer tuple)
)r r adaptive_avg_pool2dr r. r0 r2 r3 )r r _output_sizer6 r6 r7 ry ] s ry c C s6 t | rtt| f| |S t|| j }tjjj| |S )a"
Applies a 3D adaptive average pooling over an input signal composed of
several input planes.
See :class:`~torch.nn.AdaptiveAvgPool3d` for details and output shape.
Args:
output_size: the target output size (single integer or
triple-integer tuple)
)r r adaptive_avg_pool3dr r. r0 r2 r3 )r r rz r6 r6 r7 r{ n s r{ ?T)r ptraininginplacer# c C s\ t | r tt| f| |||dS |dk s.|dkr Tensor
In-place version of :func:`~threshold`.
)r r r# c C s8 t | rtt| f| |dS |r*tj| }n
tj| }|S )zrelu(input, inplace=False) -> Tensor
Applies the rectified linear unit function element-wise. See
:class:`~torch.nn.ReLU` for more details.
)r )r r ri r0 relu_)r r r r6 r6 r7 ri s
ri z<
relu_(input) -> Tensor
In-place version of :func:`~relu`.
)r r/ r# c C s> t | rtt| f| |dS | j dkr.tdtjjj| |S )a
glu(input, dim=-1) -> Tensor
The gated linear unit. Computes:
.. math ::
\text{GLU}(a, b) = a \otimes \sigma(b)
where `input` is split in half along `dim` to form `a` and `b`, :math:`\sigma`
is the sigmoid function and :math:`\otimes` is the element-wise product between matrices.
See `Language Modeling with Gated Convolutional Networks `_.
Args:
input (Tensor): input tensor
dim (int): dimension on which to split the input. Default: -1
)r/ r z>glu does not support scalars because halving size must be even)r r glur/ RuntimeErrorr0 r2 r3 )r r/ r6 r6 r7 r ! s
r ?)r min_valmax_valr r# c C sL t | r tt| f| |||dS |r6tjjj| ||}ntjjj| ||}|S )z
hardtanh(input, min_val=-1., max_val=1., inplace=False) -> Tensor
Applies the HardTanh function element-wise. See :class:`~torch.nn.Hardtanh` for more
details.
)r r r )r r hardtanhr0 r2 r3 hardtanh_)r r r r r r6 r6 r7 r : s r z]
hardtanh_(input, min_val=-1., max_val=1.) -> Tensor
In-place version of :func:`~hardtanh`.
c C s@ t | rtt| f| |dS |r.tjjj| }ntjjj| }|S )zrelu6(input, inplace=False) -> Tensor
Applies the element-wise function :math:`\text{ReLU6}(x) = \min(\max(0,x), 6)`.
See :class:`~torch.nn.ReLU6` for more details.
)r )r r relu6r0 r2 r3 Zrelu6_)r r r r6 r6 r7 r T s r )r alphar r# c C sF t | r tt| f| ||dS |r2tjjj| |}ntjjj| |}|S )zApplies element-wise,
:math:`\text{ELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x) - 1))`.
See :class:`~torch.nn.ELU` for more details.
)r r )r r elur0 r2 r3 elu_)r r r r r6 r6 r7 r d s r zD
elu_(input, alpha=1.) -> Tensor
In-place version of :func:`~elu`.
c C s8 t | rtt| f| |dS |r*tj| }n
tj| }|S )a> selu(input, inplace=False) -> Tensor
Applies element-wise,
:math:`\text{SELU}(x) = scale * (\max(0,x) + \min(0, \alpha * (\exp(x) - 1)))`,
with :math:`\alpha=1.6732632423543772848170429916717` and
:math:`scale=1.0507009873554804934193349852946`.
See :class:`~torch.nn.SELU` for more details.
)r )r r selur0 selu_)r r r r6 r6 r7 r } s
r z<
selu_(input) -> Tensor
In-place version of :func:`~selu`.
c C s> t | r tt| f| ||dS |r.tj| |}n tj| |}|S )zcelu(input, alpha=1., inplace=False) -> Tensor
Applies element-wise,
:math:`\text{CELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x/\alpha) - 1))`.
See :class:`~torch.nn.CELU` for more details.
)r r )r r celur0 celu_)r r r r r6 r6 r7 r s r zF
celu_(input, alpha=1.) -> Tensor
In-place version of :func:`~celu`.
{Gz?)r negative_sloper r# c C sF t | r tt| f| ||dS |r2tjjj| |}ntjjj| |}|S )z
leaky_relu(input, negative_slope=0.01, inplace=False) -> Tensor
Applies element-wise,
:math:`\text{LeakyReLU}(x) = \max(0, x) + \text{negative\_slope} * \min(0, x)`
See :class:`~torch.nn.LeakyReLU` for more details.
)r r )r r
leaky_relur0 r2 r3 leaky_relu_)r r r r r6 r6 r7 r s r z]
leaky_relu_(input, negative_slope=0.01) -> Tensor
In-place version of :func:`~leaky_relu`.
)r weightr# c C s$ t | rtt| f| |S tj| |S )zprelu(input, weight) -> Tensor
Applies element-wise the function
:math:`\text{PReLU}(x) = \max(0,x) + \text{weight} * \min(0,x)` where weight is a
learnable parameter.
See :class:`~torch.nn.PReLU` for more details.
)r r prelur0 )r r r6 r6 r7 r s r r$ )r lowerupperr~ r r# c C sJ t | r tt| f| ||||dS |r6tj| |||}ntj| |||}|S )zrrelu(input, lower=1./8, upper=1./3, training=False, inplace=False) -> Tensor
Randomized leaky ReLU.
See :class:`~torch.nn.RReLU` for more details.
)r r r~ r )r r rrelur0 rrelu_)r r r r~ r r r6 r6 r7 r s r zf
rrelu_(input, lower=1./8, upper=1./3, training=False) -> Tensor
In-place version of :func:`~rrelu`.
z
logsigmoid(input) -> Tensor
Applies element-wise :math:`\text{LogSigmoid}(x_i) = \log \left(\frac{1}{1 + \exp(-x_i)}\right)`
See :class:`~torch.nn.LogSigmoid` for more details.
c C s$ t | rtt| f| S tjjj| S )a gelu(input) -> Tensor
Applies element-wise the function
:math:`\text{GELU}(x) = x * \Phi(x)`
where :math:`\Phi(x)` is the Cumulative Distribution Function for Gaussian Distribution.
See `Gaussian Error Linear Units (GELUs) `_.
)r r gelur0 r2 r3 )r r6 r6 r7 r s
r )r lambdr# c C s& t | rtt| f| |dS tj| |S )z
hardshrink(input, lambd=0.5) -> Tensor
Applies the hard shrinkage function element-wise
See :class:`~torch.nn.Hardshrink` for more details.
)r )r r
hardshrinkr0 )r r r6 r6 r7 r s r c C s" t | rtt| f| S | | j S )ztanhshrink(input) -> Tensor
Applies element-wise, :math:`\text{Tanhshrink}(x) = x - \text{Tanh}(x)`
See :class:`~torch.nn.Tanhshrink` for more details.
)r r
tanhshrinktanh)r r6 r6 r7 r $ s r c C s& t | rtt| f| S | | j d S )zsoftsign(input) -> Tensor
Applies element-wise, the function :math:`\text{SoftSign}(x) = \frac{x}{1 + |x|}`
See :class:`~torch.nn.Softsign` for more details.
r )r r softsignrj )r r6 r6 r7 r 0 s r aJ
softplus(input, beta=1, threshold=20) -> Tensor
Applies element-wise, the function :math:`\text{Softplus}(x) = \frac{1}{\beta} * \log(1 + \exp(\beta * x))`.
For numerical stability the implementation reverts to the linear function
when :math:`input \times \beta > threshold`.
See :class:`~torch.nn.Softplus` for more details.
)namendim
stacklevelr# c C s: t jdj| |d |dks,|dks,|dkr2d}nd}|S )NzfImplicit dimension choice for {} has been deprecated. Change the call to include dim=X as an argument.)r r r r$ )warningswarnrV )r r r rX r6 r6 r7 _get_softmax_dimK s r )r r/ _stacklevelr% r# c C s` t | r tt| f| |||dS |dkr6td| j |}|dkrL| j|}n| j||d}|S )av Applies a softmin function.
Note that :math:`\text{Softmin}(x) = \text{Softmax}(-x)`. See softmax definition for mathematical formula.
See :class:`~torch.nn.Softmin` for more details.
Args:
input (Tensor): input
dim (int): A dimension along which softmin will be computed (so every slice
along dim will sum to 1).
dtype (:class:`torch.dtype`, optional): the desired data type of returned tensor.
If specified, the input tensor is casted to :attr:`dtype` before the operation
is performed. This is useful for preventing data type overflows. Default: None.
)r/ r r% Nsoftmin)r% )r r r r r/ softmax)r r/ r r% rX r6 r6 r7 r X s r c C s\ t | r tt| f| |||dS |dkr6td| j |}|dkrJ| j|}n| j||d}|S )a Applies a softmax function.
Softmax is defined as:
:math:`\text{Softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_j \exp(x_j)}`
It is applied to all slices along dim, and will re-scale them so that the elements
lie in the range `[0, 1]` and sum to 1.
See :class:`~torch.nn.Softmax` for more details.
Args:
input (Tensor): input
dim (int): A dimension along which softmax will be computed.
dtype (:class:`torch.dtype`, optional): the desired data type of returned tensor.
If specified, the input tensor is casted to :attr:`dtype` before the operation
is performed. This is useful for preventing data type overflows. Default: None.
.. note::
This function doesn't work directly with NLLLoss,
which expects the Log to be computed between the Softmax and itself.
Use log_softmax instead (it's faster and has better numerical properties).
)r/ r r% Nr )r% )r r r r r/ )r r/ r r% rX r6 r6 r7 r r s r 绽|=)logitstauhardepsr/ r# c
C s t | r tt| f| ||||dS |dkr2tjd tj| tjdj j }| | | }|j
|}|r|j |ddd }tj | tjdj
||d}||j | } n|} | S ) a
Samples from the Gumbel-Softmax distribution (`Link 1`_ `Link 2`_) and optionally discretizes.
Args:
logits: `[..., num_features]` unnormalized log probabilities
tau: non-negative scalar temperature
hard: if ``True``, the returned samples will be discretized as one-hot vectors,
but will be differentiated as if it is the soft sample in autograd
dim (int): A dimension along which softmax will be computed. Default: -1.
Returns:
Sampled tensor of same shape as `logits` from the Gumbel-Softmax distribution.
If ``hard=True``, the returned samples will be one-hot, otherwise they will
be probability distributions that sum to 1 across `dim`.
.. note::
This function is here for legacy reasons, may be removed from nn.Functional in the future.
.. note::
The main trick for `hard` is to do `y_hard - y_soft.detach() + y_soft`
It achieves two things:
- makes the output value exactly one-hot
(since we add then subtract y_soft value)
- makes the gradient equal to y_soft gradient
(since we strip all other gradients)
Examples::
>>> logits = torch.randn(20, 32)
>>> # Sample soft categorical using reparametrization trick:
>>> F.gumbel_softmax(logits, tau=1, hard=False)
>>> # Sample hard categorical using "Straight-through" trick:
>>> F.gumbel_softmax(logits, tau=1, hard=True)
.. _Link 1:
https://arxiv.org/abs/1611.00712
.. _Link 2:
https://arxiv.org/abs/1611.01144
)r r r r/ g|=z0`eps` parameter is deprecated and has no effect.)Z
memory_formatT)keepdimr g ?)r r gumbel_softmaxr r r0 Z
empty_likeZlegacy_contiguous_formatZ exponential_logr max
zeros_likeZscatter_detach)
r r r r r/ ZgumbelsZy_softindexZy_hardrX r6 r6 r7 r s (
r c C s\ t | r tt| f| |||dS |dkr6td| j |}|dkrJ| j|}n| j||d}|S )a Applies a softmax followed by a logarithm.
While mathematically equivalent to log(softmax(x)), doing these two
operations separately is slower and numerically unstable. This function
uses an alternative formulation to compute the output and gradient correctly.
See :class:`~torch.nn.LogSoftmax` for more details.
Args:
input (Tensor): input
dim (int): A dimension along which log_softmax will be computed.
dtype (:class:`torch.dtype`, optional): the desired data type of returned tensor.
If specified, the input tensor is cast to :attr:`dtype` before the operation
is performed. This is useful for preventing data type overflows. Default: None.
)r/ r r% N log_softmax)r% )r r r r r/ )r r/ r r% rX r6 r6 r7 r s r z
softshrink(input, lambd=0.5) -> Tensor
Applies the soft shrinkage function elementwise
See :class:`~torch.nn.Softshrink` for more details.
c C s t jd | j S )ztanh(input) -> Tensor
Applies element-wise,
:math:`\text{Tanh}(x) = \tanh(x) = \frac{\exp(x) - \exp(-x)}{\exp(x) + \exp(-x)}`
See :class:`~torch.nn.Tanh` for more details.
z9nn.functional.tanh is deprecated. Use torch.tanh instead.)r r r )r r6 r6 r7 r s
r c C s t jd | j S )zsigmoid(input) -> Tensor
Applies the element-wise function :math:`\text{Sigmoid}(x) = \frac{1}{1 + \exp(-x)}`
See :class:`~torch.nn.Sigmoid` for more details.
z?nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.)r r sigmoid)r r6 r6 r7 r s
r c C s: t | rtt| f| |dS |r,tjjj| S tjjj| S )a Applies the element-wise function
.. math::
\text{Hardsigmoid}(x) = \begin{cases}
0 & \text{if~} x \le -3, \\
1 & \text{if~} x \ge +3, \\
x / 6 + 1 / 2 & \text{otherwise}
\end{cases}
Args:
inplace: If set to ``True``, will do this operation in-place. Default: ``False``
See :class:`~torch.nn.Hardsigmoid` for more details.
)r )r r hardsigmoidr0 r2 r3 Z hardsigmoid_)r r r6 r6 r7 r s
r )r r biasr# c C s6 t | ||r$tt| ||f| ||dS tjjj| ||S )a
Applies a linear transformation to the incoming data: :math:`y = xA^T + b`.
This operator supports :ref:`TensorFloat32`.
Shape:
- Input: :math:`(N, *, in\_features)` N is the batch size, `*` means any number of
additional dimensions
- Weight: :math:`(out\_features, in\_features)`
- Bias: :math:`(out\_features)`
- Output: :math:`(N, *, out\_features)`
)r )r r linearr0 r2 r3 )r r r r6 r6 r7 r ( s r )input1input2r r r# c C s: t | |||r*tt| |||f| |||dS tj| |||S )a
Applies a bilinear transformation to the incoming data:
:math:`y = x_1^T A x_2 + b`
Shape:
- input1: :math:`(N, *, H_{in1})` where :math:`H_{in1}=\text{in1\_features}`
and :math:`*` means any number of additional dimensions.
All but the last dimension of the inputs should be the same.
- input2: :math:`(N, *, H_{in2})` where :math:`H_{in2}=\text{in2\_features}`
- weight: :math:`(\text{out\_features}, \text{in1\_features},
\text{in2\_features})`
- bias: :math:`(\text{out\_features})`
- output: :math:`(N, *, H_{out})` where :math:`H_{out}=\text{out\_features}`
and all but the last dimension are the same shape as the input.
)r )r r bilinearr0 )r r r r r6 r6 r7 r ; s
r c C s: t | rtt| f| |dS |r,tjjj| S tjjj| S )a Applies the Sigmoid Linear Unit (SiLU) function, element-wise.
The SiLU function is also known as the swish function.
.. math::
\text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.}
.. note::
See `Gaussian Error Linear Units (GELUs) `_
where the SiLU (Sigmoid Linear Unit) was originally coined, and see
`Sigmoid-Weighted Linear Units for Neural Network Function Approximation
in Reinforcement Learning `_ and `Swish:
a Self-Gated Activation Function `_
where the SiLU was experimented with later.
See :class:`~torch.nn.SiLU` for more details.
)r )r r silur0 r2 r3 Zsilu_)r r r6 r6 r7 r V s
r c C s: t | rtt| f| |dS |r,tjjj| S tjjj| S )av Applies the Mish function, element-wise.
Mish: A Self Regularized Non-Monotonic Neural Activation Function.
.. math::
\text{Mish}(x) = x * \text{Tanh}(\text{Softplus}(x))
.. note::
See `Mish: A Self Regularized Non-Monotonic Neural Activation Function `_
See :class:`~torch.nn.Mish` for more details.
)r )r r mishr0 r2 r3 Zmish_)r r r6 r6 r7 r n s
r c C s: t | rtt| f| |dS |r,tjjj| S tjjj| S )a Applies the hardswish function, element-wise, as described in the paper:
`Searching for MobileNetV3`_.
.. math::
\text{Hardswish}(x) = \begin{cases}
0 & \text{if~} x \le -3, \\
x & \text{if~} x \ge +3, \\
x \cdot (x + 3) /6 & \text{otherwise}
\end{cases}
See :class:`~torch.nn.Hardswish` for more details.
.. _`Searching for MobileNetV3`:
https://arxiv.org/abs/1905.02244
)r )r r hardswishr0 r2 r3 Z
hardswish_)r r r6 r6 r7 r s
r )r r max_normrd r# c C s( t j t j| ||| W d Q R X d S )N)r0 no_gradZembedding_renorm_)r r r rd r6 r6 r7 _no_grad_embedding_renorm_ s
r @)r r padding_idxr rd scale_grad_by_freqsparser# c
C s t | |r&tt| |f| |||||| S |dk r~|dkrN||jdk s|tdq|dk r||jd ksntd|jd| }nd}|dk r| j } t|| || tj|| |||S )as A simple lookup table that looks up embeddings in a fixed dictionary and size.
This module is often used to retrieve word embeddings using indices.
The input to the module is a list of indices, and the embedding matrix,
and the output is the corresponding word embeddings.
See :class:`torch.nn.Embedding` for more details.
Args:
input (LongTensor): Tensor containing indices into the embedding matrix
weight (Tensor): The embedding matrix with number of rows equal to the maximum possible index + 1,
and number of columns equal to the embedding size
padding_idx (int, optional): If specified, the entries at :attr:`padding_idx` do not contribute to the gradient;
therefore, the embedding vector at :attr:`padding_idx` is not updated during training,
i.e. it remains as a fixed "pad".
max_norm (float, optional): If given, each embedding vector with norm larger than :attr:`max_norm`
is renormalized to have norm :attr:`max_norm`.
Note: this will modify :attr:`weight` in-place.
norm_type (float, optional): The p of the p-norm to compute for the :attr:`max_norm` option. Default ``2``.
scale_grad_by_freq (boolean, optional): If given, this will scale gradients by the inverse of frequency of
the words in the mini-batch. Default ``False``.
sparse (bool, optional): If ``True``, gradient w.r.t. :attr:`weight` will be a sparse tensor. See Notes under
:class:`torch.nn.Embedding` for more details regarding sparse gradients.
Shape:
- Input: LongTensor of arbitrary shape containing the indices to extract
- Weight: Embedding matrix of floating point type with shape `(V, embedding_dim)`,
where V = maximum index + 1 and embedding_dim = the embedding size
- Output: `(*, embedding_dim)`, where `*` is the input shape
Examples::
>>> # a batch of 2 samples of 4 indices each
>>> input = torch.tensor([[1,2,4,5],[4,3,2,9]])
>>> # an embedding matrix containing 10 tensors of size 3
>>> embedding_matrix = torch.rand(10, 3)
>>> F.embedding(input, embedding_matrix)
tensor([[[ 0.8490, 0.9625, 0.6753],
[ 0.9666, 0.7761, 0.6108],
[ 0.6246, 0.9751, 0.3618],
[ 0.4161, 0.2419, 0.7383]],
[[ 0.6246, 0.9751, 0.3618],
[ 0.0237, 0.7794, 0.0528],
[ 0.9666, 0.7761, 0.6108],
[ 0.3385, 0.8612, 0.1867]]])
>>> # example with padding_idx
>>> weights = torch.rand(10, 3)
>>> weights[0, :].zero_()
>>> embedding_matrix = weights
>>> input = torch.tensor([[0,2,0,5]])
>>> F.embedding(input, embedding_matrix, padding_idx=0)
tensor([[[ 0.0000, 0.0000, 0.0000],
[ 0.5609, 0.5384, 0.8720],
[ 0.0000, 0.0000, 0.0000],
[ 0.6262, 0.2438, 0.7471]]])
Nr z)Padding_idx must be within num_embeddingsr r( )r r embeddingr. r,
contiguousr r0 )r r r r rd r r r6 r6 r7 r s D
r mean) r r offsetsr rd r moder per_sample_weightsinclude_last_offsetr r# c C s
t | |||r8tt| |||f| ||||||||| |
d
S |jtjkr`| j r`tjd | | }} |dk r| j |j krt
dj |j | j | j
dkr |dk rd} tjj stt|} t
dj | tjd| j | j d | j| jd
}| jd} |dk rV|jd}nJ| j
d krD|dkr,t
d |j
d krVt
d nt
d
j | j
|dkrfd} nD|dkrvd } n4|dkrd} |rt
d|rt
dnt
d|dk rt|| || |dk r|dkrtdj |tj|| ||| ||| |
\}
}}}|
S )a Computes sums, means or maxes of `bags` of embeddings, without instantiating the
intermediate embeddings.
See :class:`torch.nn.EmbeddingBag` for more details.
Note:
{backward_reproducibility_note}
Args:
input (LongTensor): Tensor containing bags of indices into the embedding matrix
weight (Tensor): The embedding matrix with number of rows equal to the maximum possible index + 1,
and number of columns equal to the embedding size
offsets (LongTensor, optional): Only used when :attr:`input` is 1D. :attr:`offsets` determines
the starting index position of each bag (sequence) in :attr:`input`.
max_norm (float, optional): If given, each embedding vector with norm larger than :attr:`max_norm`
is renormalized to have norm :attr:`max_norm`.
Note: this will modify :attr:`weight` in-place.
norm_type (float, optional): The ``p`` in the ``p``-norm to compute for the :attr:`max_norm` option.
Default ``2``.
scale_grad_by_freq (boolean, optional): if given, this will scale gradients by the inverse of frequency of
the words in the mini-batch. Default ``False``.
Note: this option is not supported when ``mode="max"``.
mode (string, optional): ``"sum"``, ``"mean"`` or ``"max"``. Specifies the way to reduce the bag.
Default: ``"mean"``
sparse (bool, optional): if ``True``, gradient w.r.t. :attr:`weight` will be a sparse tensor. See Notes under
:class:`torch.nn.Embedding` for more details regarding sparse gradients.
Note: this option is not supported when ``mode="max"``.
per_sample_weights (Tensor, optional): a tensor of float / double weights, or None
to indicate all weights should be taken to be 1. If specified, :attr:`per_sample_weights`
must have exactly the same shape as input and is treated as having the same
:attr:`offsets`, if those are not None.
include_last_offset (bool, optional): if ``True``, the size of offsets is equal to the number of bags + 1.
The last element is the size of the input, or the ending index position of the last bag (sequence).
padding_idx (int, optional): If specified, the entries at :attr:`padding_idx` do not contribute to the
gradient; therefore, the embedding vector at :attr:`padding_idx` is not updated
during training, i.e. it remains as a fixed "pad". Note that the embedding
vector at :attr:`padding_idx` is excluded from the reduction.
Shape:
- :attr:`input` (LongTensor) and :attr:`offsets` (LongTensor, optional)
- If :attr:`input` is 2D of shape `(B, N)`, it will be treated as ``B`` bags (sequences)
each of fixed length ``N``, and this will return ``B`` values aggregated in a way
depending on the :attr:`mode`. :attr:`offsets` is ignored and required to be ``None`` in this case.
- If :attr:`input` is 1D of shape `(N)`, it will be treated as a concatenation of
multiple bags (sequences). :attr:`offsets` is required to be a 1D tensor containing
the starting index positions of each bag in :attr:`input`. Therefore, for :attr:`offsets`
of shape `(B)`, :attr:`input` will be viewed as having ``B`` bags.
Empty bags (i.e., having 0-length) will have returned vectors filled by zeros.
- :attr:`weight` (Tensor): the learnable weights of the module of shape `(num_embeddings, embedding_dim)`
- :attr:`per_sample_weights` (Tensor, optional). Has the same shape as :attr:`input`.
- :attr:`output`: aggregated embedding values of shape `(B, embedding_dim)`
Examples::
>>> # an Embedding module containing 10 tensors of size 3
>>> embedding_matrix = torch.rand(10, 3)
>>> # a batch of 2 samples of 4 indices each
>>> input = torch.tensor([1,2,4,5,4,3,2,9])
>>> offsets = torch.tensor([0,4])
>>> F.embedding_bag(input, embedding_matrix, offsets)
tensor([[ 0.3397, 0.3552, 0.5545],
[ 0.5893, 0.4386, 0.5882]])
>>> # example with padding_idx
>>> embedding_matrix = torch.rand(10, 3)
>>> input = torch.tensor([2, 2, 2, 2, 4, 3, 2, 9])
>>> offsets = torch.tensor([0,4])
>>> F.embedding_bag(input, embedding_matrix, offsets, padding_idx=2, mode='sum')
tensor([[ 0.0000, 0.0000, 0.0000],
[-0.7082, 3.2145, -2.6251]])
) r r rd r r r r r r zArgument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.Nziembedding_bag: If per_sample_weights ({}) is not None, then it must have the same shape as the input ({})r z zif input is 2D, then offsets has to be None, as input is treated is a mini-batch of fixed length sequences. However, found offsets of type {}r r )r% r& z*offsets has to be a 1D Tensor but got Nonez offsets has to be a 1D Tensorz?input has to be 1D or 2D Tensor, but got Tensor of dimension {}sumr r z?max mode does not support scaling the gradient by the frequencyz(max mode does not support sparse weightsz&mode has to be one of sum, mean or maxzembedding_bag: per_sample_weights was not None. per_sample_weights is only supported for mode='sum' (got mode='{}'). Please open a feature request on GitHub.r( r( )r r
embedding_bagr% r0 longis_floating_pointr r r. r+ rV shaper/ rH is_scriptingstrtypeZarangenumelr& reshaper NotImplementedError)r r r r rd r r r r r r Ztype_str mode_enumrX _r6 r6 r7 r sz [
"
r )r. r# c C sL | d }x(t t| d D ]}|| |d 9 }qW |dkrHtdj| d S )Nr r r zGExpected more than 1 value per channel when training, got input size {})rS rT r+ rV )r.
size_prodsir6 r6 r7 _verify_batch_size s
r 皙?h㈵>) r running_mean running_varr r r~ momentumr r# c C sf t | ||||r6tt| ||||f| |||||||d
S |rFt| j tj| |||||||tjjj S )zApplies Batch Normalization for each channel across a batch of data.
See :class:`~torch.nn.BatchNorm1d`, :class:`~torch.nn.BatchNorm2d`,
:class:`~torch.nn.BatchNorm3d` for details.
)r r r~ r r ) r r
batch_normr r. r0 backendscudnnenabled)r r r r r r~ r r r6 r6 r7 r s r c C sB d}x"t dt| D ]}|| | 9 }qW |dkr>tdj| d S )Nr r zEExpected more than 1 spatial element when training, got input size {})rS rT r+ rV )r. r r r6 r6 r7 _verify_spatial_size s
r ) r r r r r use_input_statsr r r# c C sf t | ||||r6tt| ||||f| |||||||d
S |rFt| j tj| |||||||tjjj S )zApplies Instance Normalization for each channel in each data sample in a
batch.
See :class:`~torch.nn.InstanceNorm1d`, :class:`~torch.nn.InstanceNorm2d`,
:class:`~torch.nn.InstanceNorm3d` for details.
)r r r r r r r ) r r
instance_normr r. r0 r r r )r r r r r r r r r6 r6 r7 r s r )r normalized_shaper r r r# c C sB t | ||r(tt| ||f| ||||dS tj| ||||tjjjS )zzApplies Layer Normalization for last certain number of dimensions.
See :class:`~torch.nn.LayerNorm` for details.
)r r r )r r
layer_normr0 r r r )r r r r r r6 r6 r7 r s r )r
num_groupsr r r r# c C sv t | ||r(tt| ||f| ||||dS t| jd| jd | |gt| j dd tj| ||||tjjj S )zzApplies Group Normalization for last certain number of dimensions.
See :class:`~torch.nn.GroupNorm` for details.
)r r r r r r N)
r r
group_normr r. r^ r0 r r r )r r r r r r6 r6 r7 r . s 4r -C 6? ?)r r. r betakr# c C s( t | r tt| f| ||||dS | j }|dk r>tdj|| j dkrN| S | j| jd}|dkrt |dd|d |d d f}t
||dfddj d}nl| j }|j
|d d|d |d d}t |dddd|d |d d f}t||ddfddj d}|j
|}|j|j|j|}| | S ) zApplies local response normalization over an input signal composed of
several input planes, where channels occupy the second dimension.
Applies normalization across channels.
See :class:`~torch.nn.LocalResponseNorm` for details.
)r r r r$ zWExpected 3D or higher dimensionality input (got {} dimensions)r r r )rC r( )r r local_response_normr/ r+ rV r rk r` padrf ra r. view
avg_pool3daddrg )r r. r r r r/ divsizesr6 r6 r7 r ; s( "
r ) log_probstargets
input_lengthstarget_lengthsblank reduction
zero_infinityr# c C sL t | |||r0tt| |||f| ||||||d S tj| ||||tj||S )a The Connectionist Temporal Classification loss.
See :class:`~torch.nn.CTCLoss` for details.
Note:
{cudnn_reproducibility_note}
Note:
{backward_reproducibility_note}
Args:
log_probs: :math:`(T, N, C)` where `C = number of characters in alphabet including blank`,
`T = input length`, and `N = batch size`.
The logarithmized probabilities of the outputs
(e.g. obtained with :func:`torch.nn.functional.log_softmax`).
targets: :math:`(N, S)` or `(sum(target_lengths))`.
Targets cannot be blank. In the second form, the targets are assumed to be concatenated.
input_lengths: :math:`(N)`.
Lengths of the inputs (must each be :math:`\leq T`)
target_lengths: :math:`(N)`.
Lengths of the targets
blank (int, optional):
Blank label. Default :math:`0`.
reduction (string, optional): Specifies the reduction to apply to the output:
``'none'`` | ``'mean'`` | ``'sum'``. ``'none'``: no reduction will be applied,
``'mean'``: the output losses will be divided by the target lengths and
then the mean over the batch is taken, ``'sum'``: the output will be
summed. Default: ``'mean'``
zero_infinity (bool, optional):
Whether to zero infinite losses and the associated gradients.
Default: ``False``
Infinite losses mainly occur when the inputs are too short
to be aligned to the targets.
Example::
>>> log_probs = torch.randn(50, 16, 20).log_softmax(2).detach().requires_grad_()
>>> targets = torch.randint(1, 20, (16, 30), dtype=torch.long)
>>> input_lengths = torch.full((16,), 50, dtype=torch.long)
>>> target_lengths = torch.randint(10,30,(16,), dtype=torch.long)
>>> loss = F.ctc_loss(log_probs, targets, input_lengths, target_lengths)
>>> loss.backward()
)r
r r )r r ctc_lossr0
_Reductionget_enum)r r r r r
r r r6 r6 r7 r
a s 4
r
d )r targetr size_average ignore_indexreducer r# c C sd t | ||r,tt| ||f| ||||||d S |dk s<|dk rHtj||}tjjj| ||tj ||S )a
The negative log likelihood loss.
See :class:`~torch.nn.NLLLoss` for details.
Args:
input: :math:`(N, C)` where `C = number of classes` or :math:`(N, C, H, W)`
in case of 2D Loss, or :math:`(N, C, d_1, d_2, ..., d_K)` where :math:`K \geq 1`
in the case of K-dimensional loss. `input` is expected to be log-probabilities.
target: :math:`(N)` where each value is :math:`0 \leq \text{targets}[i] \leq C-1`,
or :math:`(N, d_1, d_2, ..., d_K)` where :math:`K \geq 1` for
K-dimensional loss.
weight (Tensor, optional): a manual rescaling weight given to each
class. If given, has to be a Tensor of size `C`
size_average (bool, optional): Deprecated (see :attr:`reduction`). By default,
the losses are averaged over each loss element in the batch. Note that for
some losses, there multiple elements per sample. If the field :attr:`size_average`
is set to ``False``, the losses are instead summed for each minibatch. Ignored
when reduce is ``False``. Default: ``True``
ignore_index (int, optional): Specifies a target value that is ignored
and does not contribute to the input gradient. When :attr:`size_average` is
``True``, the loss is averaged over non-ignored targets. Default: -100
reduce (bool, optional): Deprecated (see :attr:`reduction`). By default, the
losses are averaged or summed over observations for each minibatch depending
on :attr:`size_average`. When :attr:`reduce` is ``False``, returns a loss per
batch element instead and ignores :attr:`size_average`. Default: ``True``
reduction (string, optional): Specifies the reduction to apply to the output:
``'none'`` | ``'mean'`` | ``'sum'``. ``'none'``: no reduction will be applied,
``'mean'``: the sum of the output will be divided by the number of
elements in the output, ``'sum'``: the output will be summed. Note: :attr:`size_average`
and :attr:`reduce` are in the process of being deprecated, and in the meantime,
specifying either of those two args will override :attr:`reduction`. Default: ``'mean'``
Example::
>>> # input is of size N x C = 3 x 5
>>> input = torch.randn(3, 5, requires_grad=True)
>>> # each element in target has to have 0 <= value < C
>>> target = torch.tensor([1, 0, 4])
>>> output = F.nll_loss(F.log_softmax(input), target)
>>> output.backward()
)r r r r r N)
r r nll_lossr legacy_get_stringr0 r2 r3 Z nll_loss_ndr )r r r r r r r r6 r6 r7 r s 2 r :0yE>) r r log_inputfullr r r r r# c C s t | |r*tt| |f| |||||||d
S |dk s:|dk rFtj||}|dkrn|dkrn|dkrn| }t|d tj| ||||tj|}|S )a Poisson negative log likelihood loss.
See :class:`~torch.nn.PoissonNLLLoss` for details.
Args:
input: expectation of underlying Poisson distribution.
target: random sample :math:`target \sim \text{Poisson}(input)`.
log_input: if ``True`` the loss is computed as
:math:`\exp(\text{input}) - \text{target} * \text{input}`, if ``False`` then loss is
:math:`\text{input} - \text{target} * \log(\text{input}+\text{eps})`. Default: ``True``
full: whether to compute full loss, i. e. to add the Stirling
approximation term. Default: ``False``
:math:`\text{target} * \log(\text{target}) - \text{target} + 0.5 * \log(2 * \pi * \text{target})`.
size_average (bool, optional): Deprecated (see :attr:`reduction`). By default,
the losses are averaged over each loss element in the batch. Note that for
some losses, there multiple elements per sample. If the field :attr:`size_average`
is set to ``False``, the losses are instead summed for each minibatch. Ignored
when reduce is ``False``. Default: ``True``
eps (float, optional): Small value to avoid evaluation of :math:`\log(0)` when
:attr:`log_input`\ =\ ``False``. Default: 1e-8
reduce (bool, optional): Deprecated (see :attr:`reduction`). By default, the
losses are averaged or summed over observations for each minibatch depending
on :attr:`size_average`. When :attr:`reduce` is ``False``, returns a loss per
batch element instead and ignores :attr:`size_average`. Default: ``True``
reduction (string, optional): Specifies the reduction to apply to the output:
``'none'`` | ``'mean'`` | ``'sum'``. ``'none'``: no reduction will be applied,
``'mean'``: the sum of the output will be divided by the number of
elements in the output, ``'sum'``: the output will be summed. Note: :attr:`size_average`
and :attr:`reduce` are in the process of being deprecated, and in the meantime,
specifying either of those two args will override :attr:`reduction`. Default: ``'mean'``
)r r r r r r Nnoner r z
is not valid)r r poisson_nll_lossr r r+ r0 r ) r r r r r r r r rX r6 r6 r7 r s& *
r ư>)r r varr r r r# c C sb t | ||r*tt| ||f| |||||dS |j | j kr| j dd |j kr`tj|d}n8| j dd |j dd kr|jddkrntd|dkr|dkr|dkrt|d tj|d k rtd
|j }tj |j
|d W dQ R X d tj || | d
| }|r6|d t j d
t j
7 }|dkrH|j S |dkrZ|j S |S dS )a Gaussian negative log likelihood loss.
See :class:`~torch.nn.GaussianNLLLoss` for details.
Args:
input: expectation of the Gaussian distribution.
target: sample from the Gaussian distribution.
var: tensor of positive variance(s), one for each of the expectations
in the input (heteroscedastic), or a single one (homoscedastic).
full (bool, optional): include the constant term in the loss calculation. Default: ``False``.
eps (float, optional): value added to var, for stability. Default: 1e-6.
reduction (string, optional): specifies the reduction to apply to the output:
``'none'`` | ``'mean'`` | ``'sum'``. ``'none'``: no reduction will be applied,
``'mean'``: the output is the average of all batch member losses,
``'sum'``: the output is the sum of all batch member losses.
Default: ``'mean'``.
)r r r Nr zvar is of incorrect sizer r r z
is not validr z var has negative entry/entries)ming ?r r( r( r( r( r( )r r gaussian_nll_lossr. r0 r` r+ anycloner Zclamp_r mathpir r )r r r r r r lossr6 r6 r7 r (
s> .
r )r r r r r
log_targetr# c
C s t | |r&tt| |f| |||||dS |dk s6|dk rDtj||}n0|dkrVtjd |dkrjtjd}n
tj|}tj| |||d}|dkr| j dkr|| j
d }|S ) a0 The `Kullback-Leibler divergence Loss
`__
See :class:`~torch.nn.KLDivLoss` for details.
Args:
input: Tensor of arbitrary shape in log-probabilities.
target: Tensor of the same shape as input. See :attr:`log_target` for
the target's interpretation.
size_average (bool, optional): Deprecated (see :attr:`reduction`). By default,
the losses are averaged over each loss element in the batch. Note that for
some losses, there multiple elements per sample. If the field :attr:`size_average`
is set to ``False``, the losses are instead summed for each minibatch. Ignored
when reduce is ``False``. Default: ``True``
reduce (bool, optional): Deprecated (see :attr:`reduction`). By default, the
losses are averaged or summed over observations for each minibatch depending
on :attr:`size_average`. When :attr:`reduce` is ``False``, returns a loss per
batch element instead and ignores :attr:`size_average`. Default: ``True``
reduction (string, optional): Specifies the reduction to apply to the output:
``'none'`` | ``'batchmean'`` | ``'sum'`` | ``'mean'``.
``'none'``: no reduction will be applied
``'batchmean'``: the sum of the output will be divided by the batchsize
``'sum'``: the output will be summed
``'mean'``: the output will be divided by the number of elements in the output
Default: ``'mean'``
log_target (bool): A flag indicating whether ``target`` is passed in the log space.
It is recommended to pass certain distributions (like ``softmax``)
in the log space to avoid numerical issues caused by explicit ``log``.
Default: ``False``
.. note::
:attr:`size_average` and :attr:`reduce` are in the process of being deprecated,
and in the meantime, specifying either of those two args will override :attr:`reduction`.
.. note::
:attr:`reduction` = ``'mean'`` doesn't return the true kl divergence value, please use
:attr:`reduction` = ``'batchmean'`` which aligns with KL math definition.
In the next major release, ``'mean'`` will be changed to be the same as 'batchmean'.
)r r r r% Nr zreduction: 'mean' divides the total loss by both the batch size and the support size.'batchmean' divides only by the batch size, and aligns with the KL div math definition.'mean' will be changed to behave the same as 'batchmean' in the next major release.Z batchmeanr )r% r ) r r kl_divr legacy_get_enumr r r r0 r/ r. )r r r r r r% reduction_enumZreducedr6 r6 r7 r& }
s, /
r& ) r r r r r r r label_smoothingr# c C sh t | ||r.tt| ||f| |||||||d
S |dk s>|dk rJtj||}tjjj| ||tj |||S )a
This criterion computes the cross entropy loss between input and target.
See :class:`~torch.nn.CrossEntropyLoss` for details.
Args:
input (Tensor) : :math:`(N, C)` where `C = number of classes` or :math:`(N, C, H, W)`
in case of 2D Loss, or :math:`(N, C, d_1, d_2, ..., d_K)` where :math:`K \geq 1`
in the case of K-dimensional loss. `input` is expected to contain unnormalized scores
(often referred to as logits).
target (Tensor) : If containing class indices, shape :math:`(N)` where each value is
:math:`0 \leq \text{targets}[i] \leq C-1`, or :math:`(N, d_1, d_2, ..., d_K)` with
:math:`K \geq 1` in the case of K-dimensional loss. If containing class probabilities,
same shape as the input.
weight (Tensor, optional): a manual rescaling weight given to each
class. If given, has to be a Tensor of size `C`
size_average (bool, optional): Deprecated (see :attr:`reduction`). By default,
the losses are averaged over each loss element in the batch. Note that for
some losses, there multiple elements per sample. If the field :attr:`size_average`
is set to ``False``, the losses are instead summed for each minibatch. Ignored
when reduce is ``False``. Default: ``True``
ignore_index (int, optional): Specifies a target value that is ignored
and does not contribute to the input gradient. When :attr:`size_average` is
``True``, the loss is averaged over non-ignored targets. Note that
:attr:`ignore_index` is only applicable when the target contains class indices.
Default: -100
reduce (bool, optional): Deprecated (see :attr:`reduction`). By default, the
losses are averaged or summed over observations for each minibatch depending
on :attr:`size_average`. When :attr:`reduce` is ``False``, returns a loss per
batch element instead and ignores :attr:`size_average`. Default: ``True``
reduction (string, optional): Specifies the reduction to apply to the output:
``'none'`` | ``'mean'`` | ``'sum'``. ``'none'``: no reduction will be applied,
``'mean'``: the sum of the output will be divided by the number of
elements in the output, ``'sum'``: the output will be summed. Note: :attr:`size_average`
and :attr:`reduce` are in the process of being deprecated, and in the meantime,
specifying either of those two args will override :attr:`reduction`. Default: ``'mean'``
label_smoothing (float, optional): A float in [0.0, 1.0]. Specifies the amount
of smoothing when computing the loss, where 0.0 means no smoothing. The targets
become a mixture of the original ground truth and a uniform distribution as described in
`Rethinking the Inception Architecture for Computer Vision `__. Default: :math:`0.0`.
Examples::
>>> # Example of target with class indices
>>> input = torch.randn(3, 5, requires_grad=True)
>>> target = torch.randint(5, (3,), dtype=torch.int64)
>>> loss = F.cross_entropy(input, target)
>>> loss.backward()
>>>
>>> # Example of target with class probabilities
>>> input = torch.randn(3, 5, requires_grad=True)
>>> target = torch.randn(3, 5).softmax(dim=1)
>>> loss = F.cross_entropy(input, target)
>>> loss.backward()
)r r r r r r* N)
r r
cross_entropyr r r0 r2 r3 Zcross_entropy_lossr )r r r r r r r r* r6 r6 r7 r+
s @ r+ )r r r r r r r# c
C s t | ||r*tt| ||f| |||||dS |dk s:|dk rHtj||}n
tj|}|j | j krztdj|j | j |dk rt |j |j }|j
|}t j j
j| |||S )a Function that measures the Binary Cross Entropy between the target and input
probabilities.
See :class:`~torch.nn.BCELoss` for details.
Args:
input: Tensor of arbitrary shape as probabilities.
target: Tensor of the same shape as input with values between 0 and 1.
weight (Tensor, optional): a manual rescaling weight
if provided it's repeated to match input tensor shape
size_average (bool, optional): Deprecated (see :attr:`reduction`). By default,
the losses are averaged over each loss element in the batch. Note that for
some losses, there multiple elements per sample. If the field :attr:`size_average`
is set to ``False``, the losses are instead summed for each minibatch. Ignored
when reduce is ``False``. Default: ``True``
reduce (bool, optional): Deprecated (see :attr:`reduction`). By default, the
losses are averaged or summed over observations for each minibatch depending
on :attr:`size_average`. When :attr:`reduce` is ``False``, returns a loss per
batch element instead and ignores :attr:`size_average`. Default: ``True``
reduction (string, optional): Specifies the reduction to apply to the output:
``'none'`` | ``'mean'`` | ``'sum'``. ``'none'``: no reduction will be applied,
``'mean'``: the sum of the output will be divided by the number of
elements in the output, ``'sum'``: the output will be summed. Note: :attr:`size_average`
and :attr:`reduce` are in the process of being deprecated, and in the meantime,
specifying either of those two args will override :attr:`reduction`. Default: ``'mean'``
Examples::
>>> input = torch.randn((3, 2), requires_grad=True)
>>> target = torch.rand((3, 2), requires_grad=False)
>>> loss = F.binary_cross_entropy(F.sigmoid(input), target)
>>> loss.backward()
)r r r r NzwUsing a target size ({}) that is different to the input size ({}) is deprecated. Please ensure they have the same size.)r r binary_cross_entropyr r' r r. r+ rV r expandr0 r2 r3 )r r r r r r r( Znew_sizer6 r6 r7 r, ! s* )
r, )r r r r r r
pos_weightr# c C s t | |||r0tt| |||f| ||||||d S |dk s@|dk rNtj||}n
tj|}|j | j kstdj|j | j t j| ||||S )aZ Function that measures Binary Cross Entropy between target and input
logits.
See :class:`~torch.nn.BCEWithLogitsLoss` for details.
Args:
input: Tensor of arbitrary shape as unnormalized scores (often referred to as logits).
target: Tensor of the same shape as input with values between 0 and 1
weight (Tensor, optional): a manual rescaling weight
if provided it's repeated to match input tensor shape
size_average (bool, optional): Deprecated (see :attr:`reduction`). By default,
the losses are averaged over each loss element in the batch. Note that for
some losses, there multiple elements per sample. If the field :attr:`size_average`
is set to ``False``, the losses are instead summed for each minibatch. Ignored
when reduce is ``False``. Default: ``True``
reduce (bool, optional): Deprecated (see :attr:`reduction`). By default, the
losses are averaged or summed over observations for each minibatch depending
on :attr:`size_average`. When :attr:`reduce` is ``False``, returns a loss per
batch element instead and ignores :attr:`size_average`. Default: ``True``
reduction (string, optional): Specifies the reduction to apply to the output:
``'none'`` | ``'mean'`` | ``'sum'``. ``'none'``: no reduction will be applied,
``'mean'``: the sum of the output will be divided by the number of
elements in the output, ``'sum'``: the output will be summed. Note: :attr:`size_average`
and :attr:`reduce` are in the process of being deprecated, and in the meantime,
specifying either of those two args will override :attr:`reduction`. Default: ``'mean'``
pos_weight (Tensor, optional): a weight of positive examples.
Must be a vector with length equal to the number of classes.
Examples::
>>> input = torch.randn(3, requires_grad=True)
>>> target = torch.empty(3).random_(2)
>>> loss = F.binary_cross_entropy_with_logits(input, target)
>>> loss.backward()
)r r r r r. Nz4Target size ({}) must be the same as input size ({}))
r r binary_cross_entropy_with_logitsr r' r r. r+ rV r0 )r r r r r r r. r( r6 r6 r7 r/ f s" ,
r/ )r r r r r r r# c
C s t | |r&tt| |f| |||||dS |j | j ksTtjdj|j | j dd |dk sd|dk rptj||}t j
| |\}}t j j j||tj
||S )zFunction that uses a squared term if the absolute
element-wise error falls below beta and an L1 term otherwise.
See :class:`~torch.nn.SmoothL1Loss` for details.
)r r r r zUsing a target size ({}) that is different to the input size ({}). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.r )r N)r r smooth_l1_lossr. r r rV r r r0 broadcast_tensorsr2 r3 r )r r r r r r expanded_inputexpanded_targetr6 r6 r7 r0 s&
r0 )r r r deltar# c C sz t | |r"tt| |f| |||dS |j | j ksPtjdj|j | j dd tj| |\}}tj j
j||t j ||S )zFunction that uses a squared term if the absolute
element-wise error falls below delta and a delta-scaled L1 term otherwise.
See :class:`~torch.nn.HuberLoss` for details.
)r r4 zUsing a target size ({}) that is different to the input size ({}). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.r )r )
r r
huber_lossr. r r rV r0 r1 r2 r3 r r )r r r r4 r2 r3 r6 r6 r7 r5 s
r5 )r r r r r r# c C s t | |r$tt| |f| ||||dS |j | j ksRtjdj|j | j dd |dk sb|dk rntj||}t j
| |\}}t j j j||tj
|S )zl1_loss(input, target, size_average=None, reduce=None, reduction='mean') -> Tensor
Function that takes the mean element-wise absolute value difference.
See :class:`~torch.nn.L1Loss` for details.
)r r r zUsing a target size ({}) that is different to the input size ({}). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.r )r N)r r l1_lossr. r r rV r r r0 r1 r2 r3 r )r r r r r r2 r3 r6 r6 r7 r6 s
r6 c C s t | |r$tt| |f| ||||dS |j | j ksRtjdj|j | j dd |dk sb|dk rntj||}t j
| |\}}t j j j||tj
|S )zmse_loss(input, target, size_average=None, reduce=None, reduction='mean') -> Tensor
Measures the element-wise mean squared error.
See :class:`~torch.nn.MSELoss` for details.
)r r r zUsing a target size ({}) that is different to the input size ({}). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.r )r N)r r mse_lossr. r r rV r r r0 r1 r2 r3 r )r r r r r r2 r3 r6 r6 r7 r7 s
r7 )r r r marginr r r r# c C s t | ||r,tt| ||f| ||||||d S |dk s<|dk rJtj||}n
tj|}| j dksx|j dksx|j dkrtdj| j |j |j t
j| ||||S )zmargin_ranking_loss(input1, input2, target, margin=0, size_average=None, reduce=None, reduction='mean') -> Tensor
See :class:`~torch.nn.MarginRankingLoss` for details.
)r8 r r r Nr z\margin_ranking_loss does not support scalars, got sizes: input1: {}, input2: {}, target: {} ) r r margin_ranking_lossr r' r r/ r rV r. r0 )r r r r8 r r r r( r6 r6 r7 r9 + s&
$r9 )r r r8 r r r r# c
C s^ t | |r&tt| |f| |||||dS |dk s6|dk rDtj||}n
tj|}tj| |||S )zhinge_embedding_loss(input, target, margin=1.0, size_average=None, reduce=None, reduction='mean') -> Tensor
See :class:`~torch.nn.HingeEmbeddingLoss` for details.
)r8 r r r N)r r hinge_embedding_lossr r' r r0 )r r r8 r r r r( r6 r6 r7 r: R s
r: c C s^ t | |r$tt| |f| ||||dS |dk s4|dk rBtj||}n
tj|}tjjj| ||S )zmultilabel_margin_loss(input, target, size_average=None, reduce=None, reduction='mean') -> Tensor
See :class:`~torch.nn.MultiLabelMarginLoss` for details.
)r r r N) r r multilabel_margin_lossr r' r r0 r2 r3 )r r r r r r( r6 r6 r7 r; p s
r; c C s^ t | |r$tt| |f| ||||dS |dk s4|dk rBtj||}n
tj|}tjjj| ||S )zsoft_margin_loss(input, target, size_average=None, reduce=None, reduction='mean') -> Tensor
See :class:`~torch.nn.SoftMarginLoss` for details.
)r r r N) r r soft_margin_lossr r' r r0 r2 r3 )r r r r r r( r6 r6 r7 r< s
r< c
C s t | ||r*tt| ||f| |||||dS |dk s:|dk rFtj||}|t| d| t| }|dk rv|| }|jdd| jd }|dkr|}n4|dkr|j }n"|dkr|j }n| }t |d |S ) zmultilabel_soft_margin_loss(input, target, weight=None, size_average=None) -> Tensor
See :class:`~torch.nn.MultiLabelSoftMarginLoss` for details.
)r r r r Nr )r/ r r r z
is not valid)
r r multilabel_soft_margin_lossr r
logsigmoidr r. r r+ )r r r r r r r$ rX r6 r6 r7 r= s2
r= c C sf t | ||r,tt| ||f| ||||||d S |dk s<|dk rJtj||}n
tj|}tj| ||||S )zcosine_embedding_loss(input1, input2, target, margin=0, size_average=None, reduce=None, reduction='mean') -> Tensor
See :class:`~torch.nn.CosineEmbeddingLoss` for details.
)r8 r r r N)r r cosine_embedding_lossr r' r r0 )r r r r8 r r r r( r6 r6 r7 r? s
r? ) r r r} r8 r r r r r# c C s t | ||r.tt| ||f| |||||||d
S |dk s>|dk rLtj||}n
tj|}|dkrn|dkrntd|dk r|j dkrtdtj j
j| |||||S )zmulti_margin_loss(input, target, p=1, margin=1, weight=None, size_average=None,
reduce=None, reduction='mean') -> Tensor
See :class:`~torch.nn.MultiMarginLoss` for details.
)r} r8 r r r r Nr r z only p == 1 and p == 2 supportedz weight must be one-dimensional) r r multi_margin_lossr r' r r+ r/ r0 r2 r3 ) r r r} r8 r r r r r( r6 r6 r7 r@ s*
r@ a-
pixel_shuffle(input, upscale_factor) -> Tensor
Rearranges elements in a tensor of shape :math:`(*, C \times r^2, H, W)` to a
tensor of shape :math:`(*, C, H \times r, W \times r)`, where r is the :attr:`upscale_factor`.
See :class:`~torch.nn.PixelShuffle` for details.
Args:
input (Tensor): the input tensor
upscale_factor (int): factor to increase spatial resolution by
Examples::
>>> input = torch.randn(1, 9, 4, 4)
>>> output = torch.nn.functional.pixel_shuffle(input, 3)
>>> print(output.size())
torch.Size([1, 1, 12, 12])
at
pixel_unshuffle(input, downscale_factor) -> Tensor
Reverses the :class:`~torch.nn.PixelShuffle` operation by rearranging elements in a
tensor of shape :math:`(*, C, H \times r, W \times r)` to a tensor of shape
:math:`(*, C \times r^2, H, W)`, where r is the :attr:`downscale_factor`.
See :class:`~torch.nn.PixelUnshuffle` for details.
Args:
input (Tensor): the input tensor
downscale_factor (int): factor to increase spatial resolution by
Examples::
>>> input = torch.randn(1, 1, 12, 12)
>>> output = torch.nn.functional.pixel_unshuffle(input, 3)
>>> print(output.size())
torch.Size([1, 9, 4, 4])
a5
channel_shuffle(input, groups) -> Tensor
Divide the channels in a tensor of shape :math:`(*, C , H, W)`
into g groups and rearrange them as :math:`(*, C \frac g, g, H, W)`,
while keeping the original tensor shape.
See :class:`~torch.nn.ChannelShuffle` for details.
Args:
input (Tensor): the input tensor
groups (int): number of groups to divide channels in and rearrange.
Examples::
>>> input = torch.randn(1, 4, 2, 2)
>>> print(input)
[[[[1, 2],
[3, 4]],
[[5, 6],
[7, 8]],
[[9, 10],
[11, 12]],
[[13, 14],
[15, 16]],
]]
>>> output = torch.nn.functional.channel_shuffle(input, 2)
>>> print(output)
[[[[1, 2],
[3, 4]],
[[9, 10],
[11, 12]],
[[5, 6],
[7, 8]],
[[13, 14],
[15, 16]],
]]
nearest)r r. scale_factorr
align_cornersr# c C s d S )Nr6 )r r. rB r rC r6 r6 r7 upsampler
s rD c C s d S )Nr6 )r r. rB r rC r6 r6 r7 rD w
s c C s t jd t| ||||S )a% Upsamples the input to either the given :attr:`size` or the given
:attr:`scale_factor`
.. warning::
This function is deprecated in favor of :func:`torch.nn.functional.interpolate`.
This is equivalent with ``nn.functional.interpolate(...)``.
Note:
{backward_reproducibility_note}
The algorithm used for upsampling is determined by :attr:`mode`.
Currently temporal, spatial and volumetric upsampling are supported, i.e.
expected inputs are 3-D, 4-D or 5-D in shape.
The input dimensions are interpreted in the form:
`mini-batch x channels x [optional depth] x [optional height] x width`.
The modes available for upsampling are: `nearest`, `linear` (3D-only),
`bilinear`, `bicubic` (4D-only), `trilinear` (5D-only)
Args:
input (Tensor): the input tensor
size (int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int]):
output spatial size.
scale_factor (float or Tuple[float]): multiplier for spatial size. Has to match input size if it is a tuple.
mode (string): algorithm used for upsampling:
``'nearest'`` | ``'linear'`` | ``'bilinear'`` | ``'bicubic'`` |
``'trilinear'``. Default: ``'nearest'``
align_corners (bool, optional): Geometrically, we consider the pixels of the
input and output as squares rather than points.
If set to ``True``, the input and output tensors are aligned by the
center points of their corner pixels, preserving the values at the corner pixels.
If set to ``False``, the input and output tensors are aligned by the corner
points of their corner pixels, and the interpolation uses edge value padding
for out-of-boundary values, making this operation *independent* of input size
when :attr:`scale_factor` is kept the same. This only has an effect when :attr:`mode`
is ``'linear'``, ``'bilinear'``, ``'bicubic'`` or ``'trilinear'``.
Default: ``False``
.. note::
With ``mode='bicubic'``, it's possible to cause overshoot, in other words it can produce
negative values or values greater than 255 for images.
Explicitly call ``result.clamp(min=0, max=255)`` if you want to reduce the overshoot
when displaying the image.
.. warning::
With ``align_corners = True``, the linearly interpolating modes
(`linear`, `bilinear`, and `trilinear`) don't proportionally align the
output and input pixels, and thus the output values can depend on the
input size. This was the default behavior for these modes up to version
0.3.1. Since then, the default behavior is ``align_corners = False``.
See :class:`~torch.nn.Upsample` for concrete examples on how this
affects the outputs.
zLnn.functional.upsample is deprecated. Use nn.functional.interpolate instead.)r r interpolate)r r. rB r rC r6 r6 r7 rD |
s 9
)r r. rB r rC recompute_scale_factorr# c C s d S )Nr6 )r r. rB r rC rF r6 r6 r7 rE
s rE c C s d S )Nr6 )r r. rB r rC rF r6 r6 r7 rE
s c C s d S )Nr6 )r r. rB r rC rF r6 r6 r7 rE
s c C s d S )Nr6 )r r. rB r rC rF r6 r6 r7 rE
s c
s t r"tt f |||dS |d#kr<|dk rXtdn |dkrXtjdj| d} j d }dk r~dk r~td nֈdk rdkstdt t
t frt |krtd
j|t }nfd d t
|D }nrdk rLdkstd}t t
t fr4t |kr.td
j|t nfdd t
|D ntd|dkrdk rxDD ]"}tj||krntjd P qnW n|rdk rtd|dkr|dkrd}|dk r4|r4tjj r
tjj r
fdd t
|D }n&dk st fdd t
|D }d j dkr^|dkr^tjjj |S j dkr|dkrtjjj |S j dkr|dkrtjjj |S j dkr|dkr|dk stt |S j dkr|dkr|dk stt |S j dkrB|dkrB|dk s8tt |S j dkr||dkr||dk shttjjj ||S j dkr|dkr|dk sttjjj ||S j dkr|dkr|dk sttjjj ||S j dkr*|dkr*|dk sttjjj ||S j dkrJ|dkrJt d j dkrj|dkrjt d j dkr|dkrt d j dkr|dkrt d j dkr|dkrt d j dkr|dkrt d!t d"j j |dS )$a Down/up samples the input to either the given :attr:`size` or the given
:attr:`scale_factor`
The algorithm used for interpolation is determined by :attr:`mode`.
Currently temporal, spatial and volumetric sampling are supported, i.e.
expected inputs are 3-D, 4-D or 5-D in shape.
The input dimensions are interpreted in the form:
`mini-batch x channels x [optional depth] x [optional height] x width`.
The modes available for resizing are: `nearest`, `linear` (3D-only),
`bilinear`, `bicubic` (4D-only), `trilinear` (5D-only), `area`
Args:
input (Tensor): the input tensor
size (int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int]):
output spatial size.
scale_factor (float or Tuple[float]): multiplier for spatial size. If `scale_factor` is a tuple,
its length has to match `input.dim()`.
mode (str): algorithm used for upsampling:
``'nearest'`` | ``'linear'`` | ``'bilinear'`` | ``'bicubic'`` |
``'trilinear'`` | ``'area'``. Default: ``'nearest'``
align_corners (bool, optional): Geometrically, we consider the pixels of the
input and output as squares rather than points.
If set to ``True``, the input and output tensors are aligned by the
center points of their corner pixels, preserving the values at the corner pixels.
If set to ``False``, the input and output tensors are aligned by the corner
points of their corner pixels, and the interpolation uses edge value padding
for out-of-boundary values, making this operation *independent* of input size
when :attr:`scale_factor` is kept the same. This only has an effect when :attr:`mode`
is ``'linear'``, ``'bilinear'``, ``'bicubic'`` or ``'trilinear'``.
Default: ``False``
recompute_scale_factor (bool, optional): recompute the scale_factor for use in the
interpolation calculation. If `recompute_scale_factor` is ``True``, then
`scale_factor` must be passed in and `scale_factor` is used to compute the
output `size`. The computed output `size` will be used to infer new scales for
the interpolation. Note that when `scale_factor` is floating-point, it may differ
from the recomputed `scale_factor` due to rounding and precision issues.
If `recomputed_scale_factor` is ``False``, then `size` or `scale_factor` will
be used directly for interpolation.
.. note::
With ``mode='bicubic'``, it's possible to cause overshoot, in other words it can produce
negative values or values greater than 255 for images.
Explicitly call ``result.clamp(min=0, max=255)`` if you want to reduce the overshoot
when displaying the image.
.. warning::
With ``align_corners = True``, the linearly interpolating modes
(`linear`, `bilinear`, and `trilinear`) don't proportionally align the
output and input pixels, and thus the output values can depend on the
input size. This was the default behavior for these modes up to version
0.3.1. Since then, the default behavior is ``align_corners = False``.
See :class:`~torch.nn.Upsample` for concrete examples on how this
affects the outputs.
.. warning::
When scale_factor is specified, if recompute_scale_factor=True,
scale_factor is used to compute the output_size which will then
be used to infer new scales for the interpolation.
The default behavior for recompute_scale_factor changed to False
in 1.6.0, and scale_factor is used in the interpolation
calculation.
Note:
{backward_reproducibility_note}
)r. rB r rC rF rA areaNzjalign_corners option can only be set with the interpolating modes: linear | bilinear | bicubic | trilinearzDefault upsampling behavior when mode={} is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.Fr z2only one of size or scale_factor should be definedz;size shape must match input shape. Input is {}D, size is {}c s g | ]} qS r6 r6 ).0r )r. r6 r7
I s z interpolate..zKscale_factor shape must match input shape. Input is {}D, scale_factor is {}c s g | ]} qS r6 r6 )rH r )rB r6 r7 rI U s z-either size or scale_factor should be definedaX The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details. z?recompute_scale_factor is not meaningful with an explicit size.Tc s< g | ]4}t j j|d j t j| t jd j qS )r )r% )r0 floorr. floatZtensorZfloat32)rH r )r
scale_factorsr6 r7 rI u s c s0 g | ](}t tjt j|d | qS )r )r- r" rJ rK r. )rH r )r rL r6 r7 rI z s r$ r9 r r Z trilinearbicubicz.Got 3D input, but bilinear mode needs 4D inputz/Got 3D input, but trilinear mode needs 5D inputz,Got 4D input, but linear mode needs 3D inputz/Got 4D input, but trilinear mode needs 5D inputz,Got 5D input, but linear mode needs 3D inputz.Got 5D input, but bilinear mode needs 4D inputzInput Error: Only 3D, 4D and 5D input Tensors supported (got {}D) for the modes: nearest | linear | bilinear | bicubic | trilinear (got {}))rA rG )!r r rE r+ r r rV r/ r, r] r^ tuplerT rS r" rJ r0 rH r r2 Z_get_tracing_stater3 Zupsample_nearest1dZupsample_nearest2dZupsample_nearest3dadaptive_avg_pool1dry r{ Zupsample_linear1dZupsample_bilinear2dZupsample_trilinear3dZupsample_bicubic2dr ) r r. rB r rC rF r/ r Zscaler6 )r rB rL r. r7 rE
s E
)r r. rB r# c C s d S )Nr6 )r r. rB r6 r6 r7 upsample_nearest s rQ c C s d S )Nr6 )r r. rB r6 r6 r7 rQ s c C s t jd t| ||ddS )a| Upsamples the input, using nearest neighbours' pixel values.
.. warning::
This function is deprecated in favor of :func:`torch.nn.functional.interpolate`.
This is equivalent with ``nn.functional.interpolate(..., mode='nearest')``.
Currently spatial and volumetric upsampling are supported (i.e. expected
inputs are 4 or 5 dimensional).
Args:
input (Tensor): input
size (int or Tuple[int, int] or Tuple[int, int, int]): output spatia
size.
scale_factor (int): multiplier for spatial size. Has to be an integer.
Note:
{backward_reproducibility_note}
zTnn.functional.upsample_nearest is deprecated. Use nn.functional.interpolate instead.rA )r )r r rE )r r. rB r6 r6 r7 rQ s
c C s d S )Nr6 )r r. rB r6 r6 r7 upsample_bilinear s rR c C s d S )Nr6 )r r. rB r6 r6 r7 rR s c C s d S )Nr6 )r r. rB r6 r6 r7 rR s c C s d S )Nr6 )r r. rB r6 r6 r7 rR s c C s t jd t| ||dddS )ah Upsamples the input, using bilinear upsampling.
.. warning::
This function is deprecated in favor of :func:`torch.nn.functional.interpolate`.
This is equivalent with
``nn.functional.interpolate(..., mode='bilinear', align_corners=True)``.
Expected inputs are spatial (4 dimensional). Use `upsample_trilinear` fo
volumetric (5 dimensional) inputs.
Args:
input (Tensor): input
size (int or Tuple[int, int]): output spatial size.
scale_factor (int or Tuple[int, int]): multiplier for spatial size
Note:
{backward_reproducibility_note}
zUnn.functional.upsample_bilinear is deprecated. Use nn.functional.interpolate instead.r T)r rC )r r rE )r r. rB r6 r6 r7 rR s
)r rA rN )zerosborder
reflectionrS )r gridr padding_moderC r# c C s t | |r$tt| |f| ||||dS |dkrJ|dkrJ|dkrJtdj||dkrp|dkrp|dkrptd j||dkr~d
}n|dkrd }nd }|dkrd
}n|dkrd }nd }|d
krtjd d}tj| ||||S )a Given an :attr:`input` and a flow-field :attr:`grid`, computes the
``output`` using :attr:`input` values and pixel locations from :attr:`grid`.
Currently, only spatial (4-D) and volumetric (5-D) :attr:`input` are
supported.
In the spatial (4-D) case, for :attr:`input` with shape
:math:`(N, C, H_\text{in}, W_\text{in})` and :attr:`grid` with shape
:math:`(N, H_\text{out}, W_\text{out}, 2)`, the output will have shape
:math:`(N, C, H_\text{out}, W_\text{out})`.
For each output location ``output[n, :, h, w]``, the size-2 vector
``grid[n, h, w]`` specifies :attr:`input` pixel locations ``x`` and ``y``,
which are used to interpolate the output value ``output[n, :, h, w]``.
In the case of 5D inputs, ``grid[n, d, h, w]`` specifies the
``x``, ``y``, ``z`` pixel locations for interpolating
``output[n, :, d, h, w]``. :attr:`mode` argument specifies ``nearest`` or
``bilinear`` interpolation method to sample the input pixels.
:attr:`grid` specifies the sampling pixel locations normalized by the
:attr:`input` spatial dimensions. Therefore, it should have most values in
the range of ``[-1, 1]``. For example, values ``x = -1, y = -1`` is the
left-top pixel of :attr:`input`, and values ``x = 1, y = 1`` is the
right-bottom pixel of :attr:`input`.
If :attr:`grid` has values outside the range of ``[-1, 1]``, the corresponding
outputs are handled as defined by :attr:`padding_mode`. Options are
* ``padding_mode="zeros"``: use ``0`` for out-of-bound grid locations,
* ``padding_mode="border"``: use border values for out-of-bound grid locations,
* ``padding_mode="reflection"``: use values at locations reflected by
the border for out-of-bound grid locations. For location far away
from the border, it will keep being reflected until becoming in bound,
e.g., (normalized) pixel location ``x = -3.5`` reflects by border ``-1``
and becomes ``x' = 1.5``, then reflects by border ``1`` and becomes
``x'' = -0.5``.
Note:
This function is often used in conjunction with :func:`affine_grid`
to build `Spatial Transformer Networks`_ .
Note:
When using the CUDA backend, this operation may induce nondeterministic
behaviour in its backward pass that is not easily switched off.
Please see the notes on :doc:`/notes/randomness` for background.
Note:
NaN values in :attr:`grid` would be interpreted as ``-1``.
Args:
input (Tensor): input of shape :math:`(N, C, H_\text{in}, W_\text{in})` (4-D case)
or :math:`(N, C, D_\text{in}, H_\text{in}, W_\text{in})` (5-D case)
grid (Tensor): flow-field of shape :math:`(N, H_\text{out}, W_\text{out}, 2)` (4-D case)
or :math:`(N, D_\text{out}, H_\text{out}, W_\text{out}, 3)` (5-D case)
mode (str): interpolation mode to calculate output values
``'bilinear'`` | ``'nearest'`` | ``'bicubic'``. Default: ``'bilinear'``
Note: ``mode='bicubic'`` supports only 4-D input.
When ``mode='bilinear'`` and the input is 5-D, the interpolation mode
used internally will actually be trilinear. However, when the input is 4-D,
the interpolation mode will legitimately be bilinear.
padding_mode (str): padding mode for outside grid values
``'zeros'`` | ``'border'`` | ``'reflection'``. Default: ``'zeros'``
align_corners (bool, optional): Geometrically, we consider the pixels of the
input as squares rather than points.
If set to ``True``, the extrema (``-1`` and ``1``) are considered as referring
to the center points of the input's corner pixels. If set to ``False``, they
are instead considered as referring to the corner points of the input's corner
pixels, making the sampling more resolution agnostic.
This option parallels the ``align_corners`` option in
:func:`interpolate`, and so whichever option is used here
should also be used there to resize the input image before grid sampling.
Default: ``False``
Returns:
output (Tensor): output Tensor
.. _`Spatial Transformer Networks`:
https://arxiv.org/abs/1506.02025
.. warning::
When ``align_corners = True``, the grid positions depend on the pixel
size relative to the input image size, and so the locations sampled by
:func:`grid_sample` will differ for the same input given at different
resolutions (that is, after being upsampled or downsampled).
The default behavior up to version 1.2.0 was ``align_corners = True``.
Since then, the default behavior has been changed to ``align_corners = False``,
in order to bring it in line with the default for :func:`interpolate`.
.. note::
``mode='bicubic'`` is implemented using the `cubic convolution algorithm`_ with :math:`\alpha=-0.75`.
The constant :math:`\alpha` might be different from packages to packages.
For example, `PIL`_ and `OpenCV`_ use -0.5 and -0.75 respectively.
This algorithm may "overshoot" the range of values it's interpolating.
For example, it may produce negative values or values greater than 255 when interpolating input in [0, 255].
Clamp the results with :func: `torch.clamp` to ensure they are within the valid range.
.. _`cubic convolution algorithm`: https://en.wikipedia.org/wiki/Bicubic_interpolation
.. _`PIL`: https://github.com/python-pillow/Pillow/blob/4634eafe3c695a014267eefdce830b4a825beed7/src/libImaging/Resample.c#L51
.. _`OpenCV`: https://github.com/opencv/opencv/blob/f345ed564a06178670750bad59526cfa4033be55/modules/imgproc/src/resize.cpp#L908
)r rW rC r rA rN zbnn.functional.grid_sample(): expected mode to be 'bilinear', 'nearest' or 'bicubic', but got: '{}'rS rT rU zjnn.functional.grid_sample(): expected padding_mode to be 'zeros', 'border', or 'reflection', but got: '{}'r r r NzDefault grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.F) r r grid_sampler+ rV r r r0 Z grid_sampler)r rV r rW rC r Zpadding_mode_enumr6 r6 r7 rX s4 j
rX )thetar. rC r# c C sL t | r tt| f| ||dS |dkr2tjd d}| j sJtdj| jt |dkr| j
dks~| j d dks~| j d dkrtd
j|| j |dd }ndt |d k r| j
dks| j d dks| j d dkrtd j|| j |dd }nt d
j||r"t
|d kr"tjd n t
|dkr>tdj|tj| ||S )a} Generates a 2D or 3D flow field (sampling grid), given a batch of
affine matrices :attr:`theta`.
.. note::
This function is often used in conjunction with :func:`grid_sample`
to build `Spatial Transformer Networks`_ .
Args:
theta (Tensor): input batch of affine matrices with shape
(:math:`N \times 2 \times 3`) for 2D or
(:math:`N \times 3 \times 4`) for 3D
size (torch.Size): the target output image size.
(:math:`N \times C \times H \times W` for 2D or
:math:`N \times C \times D \times H \times W` for 3D)
Example: torch.Size((32, 3, 24, 24))
align_corners (bool, optional): if ``True``, consider ``-1`` and ``1``
to refer to the centers of the corner pixels rather than the image corners.
Refer to :func:`grid_sample` for a more complete description.
A grid generated by :func:`affine_grid` should be passed to :func:`grid_sample`
with the same setting for this option.
Default: ``False``
Returns:
output (Tensor): output Tensor of size (:math:`N \times H \times W \times 2`)
.. _`Spatial Transformer Networks`:
https://arxiv.org/abs/1506.02025
.. warning::
When ``align_corners = True``, the grid positions depend on the pixel
size relative to the input image size, and so the locations sampled by
:func:`grid_sample` will differ for the same input given at different
resolutions (that is, after being upsampled or downsampled).
The default behavior up to version 1.2.0 was ``align_corners = True``.
Since then, the default behavior has been changed to ``align_corners = False``,
in order to bring it in line with the default for :func:`interpolate`.
.. warning::
When ``align_corners = True``, 2D affine transforms on 1D data and
3D affine transforms on 2D data (that is, when one of the spatial
dimensions has unit size) are ill-defined, and not an intended use case.
This is not a problem when ``align_corners = False``.
Up to version 1.2.0, all grid points along a unit dimension were
considered arbitrarily to be at ``-1``.
From version 1.3.0, under ``align_corners = True`` all grid points
along a unit dimension are considered to be at ``0``
(the center of the input image).
)rC NzDefault grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.Fz6Expected theta to have floating point type, but got {}r9 r$ r r zJExpected a batch of 2D affine matrices of shape Nx2x3 for size {}. Got {}.rM zJExpected a batch of 3D affine matrices of shape Nx3x4 for size {}. Got {}.zfaffine_grid only supports 4D and 5D sizes, for 2D and 3D affine transforms, respectively. Got size {}.zSince version 1.3.0, affine_grid behavior has changed for unit-size grids when align_corners=True. This is not an intended use case of affine_grid. See the documentation of affine_grid for details.r z/Expected non-zero, positive output size. Got {}r' r( r' r' r( r) )r r affine_gridr r r r+ rV r% rT r/ r r r r0 Zaffine_grid_generator)rY r. rC Z spatial_sizer6 r6 r7 rZ s: 0 ( ( rZ constant)r r r r r# c C s t | r tt| f| |||dS t|d dks6tdt|d | j ksRtd|dkrhtj| ||S |dks~tdj|t|dkr| j dks| j d kr|d
krt j
j j | |S |d krt j
j j
| |S |d krt| |S t nt|d
krd| j d ks| j d
krd|d
kr0t j
j j| |S |d krJt j
j j| |S |d kr^t| |S tnt|dkr| j d
ks| j dkr|d
krt j
j j| |S |d krt j
j j| |S |d krt| |S tntddS )a`
Pads tensor.
Padding size:
The padding size by which to pad some dimensions of :attr:`input`
are described starting from the last dimension and moving forward.
:math:`\left\lfloor\frac{\text{len(pad)}}{2}\right\rfloor` dimensions
of ``input`` will be padded.
For example, to pad only the last dimension of the input tensor, then
:attr:`pad` has the form
:math:`(\text{padding\_left}, \text{padding\_right})`;
to pad the last 2 dimensions of the input tensor, then use
:math:`(\text{padding\_left}, \text{padding\_right},`
:math:`\text{padding\_top}, \text{padding\_bottom})`;
to pad the last 3 dimensions, use
:math:`(\text{padding\_left}, \text{padding\_right},`
:math:`\text{padding\_top}, \text{padding\_bottom}`
:math:`\text{padding\_front}, \text{padding\_back})`.
Padding mode:
See :class:`torch.nn.ConstantPad2d`, :class:`torch.nn.ReflectionPad2d`, and
:class:`torch.nn.ReplicationPad2d` for concrete examples on how each of the
padding modes works. Constant padding is implemented for arbitrary dimensions.
Replicate and reflection padding is implemented for padding the last 3
dimensions of 5D input tensor, or the last 2 dimensions of 4D input
tensor, or the last dimension of 3D input tensor.
Note:
When using the CUDA backend, this operation may induce nondeterministic
behaviour in its backward pass that is not easily switched off.
Please see the notes on :doc:`/notes/randomness` for background.
Args:
input (Tensor): N-dimensional tensor
pad (tuple): m-elements tuple, where
:math:`\frac{m}{2} \leq` input dimensions and :math:`m` is even.
mode: ``'constant'``, ``'reflect'``, ``'replicate'`` or ``'circular'``.
Default: ``'constant'``
value: fill value for ``'constant'`` padding. Default: ``0``
Examples::
>>> t4d = torch.empty(3, 3, 4, 2)
>>> p1d = (1, 1) # pad last dim by 1 on each side
>>> out = F.pad(t4d, p1d, "constant", 0) # effectively zero padding
>>> print(out.size())
torch.Size([3, 3, 4, 4])
>>> p2d = (1, 1, 2, 2) # pad last dim by (1, 1) and 2nd to last by (2, 2)
>>> out = F.pad(t4d, p2d, "constant", 0)
>>> print(out.size())
torch.Size([3, 3, 8, 4])
>>> t4d = torch.empty(3, 3, 4, 2)
>>> p3d = (0, 1, 2, 1, 3, 3) # pad by (0, 1), (2, 1), and (3, 3)
>>> out = F.pad(t4d, p3d, "constant", 0)
>>> print(out.size())
torch.Size([3, 9, 7, 3])
)r r r r z%Padding length must be divisible by 2zPadding length too larger[ g z1Padding mode "{}"" doesn't take in value argumentr$ ZreflectZ replicateZcircularr9 rL rM zKOnly 2D, 3D, 4D, 5D padding with non-constant padding are supported for nowN)r r _padrT r, r/ r Zconstant_pad_ndrV r0 r2 r3 Zreflection_pad1dZreplication_pad1d
_pad_circularr Zreflection_pad2dZreplication_pad2dZreflection_pad3dZreplication_pad3d)r r r r r6 r6 r7 r\ s@ : $
*
*
r\ )x1x2r} r r r# c C s6 t | |r$tt| |f| ||||dS tj| ||||S )z<
See :class:`torch.nn.PairwiseDistance` for details
)r} r r )r r pairwise_distancer0 )r^ r_ r} r r r6 r6 r7 r` z s
r` a
pdist(input, p=2) -> Tensor
Computes the p-norm distance between every pair of row vectors in the input.
This is identical to the upper triangular portion, excluding the diagonal, of
`torch.norm(input[:, None] - input, dim=2, p=p)`. This function will be faster
if the rows are contiguous.
If input has shape :math:`N \times M` then the output will have shape
:math:`\frac{1}{2} N (N - 1)`.
This function is equivalent to `scipy.spatial.distance.pdist(input,
'minkowski', p=p)` if :math:`p \in (0, \infty)`. When :math:`p = 0` it is
equivalent to `scipy.spatial.distance.pdist(input, 'hamming') * M`.
When :math:`p = \infty`, the closest scipy function is
`scipy.spatial.distance.pdist(xn, lambda x, y: np.abs(x - y).max())`.
Args:
input: input tensor of shape :math:`N \times M`.
p: p value for the p-norm distance to calculate between each vector pair
:math:`\in [0, \infty]`.
a
cosine_similarity(x1, x2, dim=1, eps=1e-8) -> Tensor
Returns cosine similarity between ``x1`` and ``x2``, computed along dim. ``x1`` and ``x2`` must be broadcastable
to a common shape. ``dim`` refers to the dimension in this common shape. Dimension ``dim`` of the output is
squeezed (see :func:`torch.squeeze`), resulting in the
output tensor having 1 fewer dimension.
.. math ::
\text{similarity} = \dfrac{x_1 \cdot x_2}{\max(\Vert x_1 \Vert _2 \cdot \Vert x_2 \Vert _2, \epsilon)}
Supports :ref:`type promotion `.
Args:
x1 (Tensor): First input.
x2 (Tensor): Second input.
dim (int, optional): Dimension along which cosine similarity is computed. Default: 1
eps (float, optional): Small value to avoid division by zero.
Default: 1e-8
Example::
>>> input1 = torch.randn(100, 128)
>>> input2 = torch.randn(100, 128)
>>> output = F.cosine_similarity(input1, input2)
>>> print(output)
aa
one_hot(tensor, num_classes=-1) -> LongTensor
Takes LongTensor with index values of shape ``(*)`` and returns a tensor
of shape ``(*, num_classes)`` that have zeros everywhere except where the
index of last dimension matches the corresponding value of the input tensor,
in which case it will be 1.
See also `One-hot on Wikipedia`_ .
.. _One-hot on Wikipedia:
https://en.wikipedia.org/wiki/One-hot
Arguments:
tensor (LongTensor): class values of any shape.
num_classes (int): Total number of classes. If set to -1, the number
of classes will be inferred as one greater than the largest class
value in the input tensor.
Returns:
LongTensor that has one more dimension with 1 values at the
index of last dimension indicated by the input, and 0 everywhere
else.
Examples:
>>> F.one_hot(torch.arange(0, 5) % 3)
tensor([[1, 0, 0],
[0, 1, 0],
[0, 0, 1],
[1, 0, 0],
[0, 1, 0]])
>>> F.one_hot(torch.arange(0, 5) % 3, num_classes=5)
tensor([[1, 0, 0, 0, 0],
[0, 1, 0, 0, 0],
[0, 0, 1, 0, 0],
[1, 0, 0, 0, 0],
[0, 1, 0, 0, 0]])
>>> F.one_hot(torch.arange(0, 6).view(3,2) % 3)
tensor([[[1, 0, 0],
[0, 1, 0]],
[[0, 0, 1],
[1, 0, 0]],
[[0, 1, 0],
[0, 0, 1]]])
) anchorpositivenegativer8 r} r swapr r r r# c
C sr t | ||r2tt| ||f| ||||||||| d S |dk sB|dk rPtj||}
n
tj| }
tj| |||||||
S )z>
See :class:`~torch.nn.TripletMarginLoss` for details
)r8 r} r rd r r r N)r r triplet_margin_lossr r' r r0 ) ra rb rc r8 r} r rd r r r r( r6 r6 r7 re s$
re )distance_functionr8 rd r )ra rb rc rf r8 rd r r# c C s t jj rtdt| ||r>tt| ||f| ||||||d S |dk rJ|nt}|| |}|| |}|r||||} t j|| }t j || | dd}
t
j |} | dkr|
j S | dkr|
j
S |
S dS )zK
See :class:`~torch.nn.TripletMarginWithDistanceLoss` for details.
zuF.triplet_margin_with_distance_loss does not support JIT scripting: functions requiring Callables cannot be scripted.)rf r8 rd r Ng )r r r )r0 rH r r r r !triplet_margin_with_distance_lossr` r clampr r r r ) ra rb rc rf r8 rd r Z
positive_distZ
negative_distZ swap_distoutputr( r6 r6 r7 rg s6
rg -q=)r r} r/ r rm r# c C s t | |r$tt| |f| ||||dS |dkrP| j||ddj|j| }| | S | j||ddj|j| }tj| ||dS dS )aL Performs :math:`L_p` normalization of inputs over specified dimension.
For a tensor :attr:`input` of sizes :math:`(n_0, ..., n_{dim}, ..., n_k)`, each
:math:`n_{dim}` -element vector :math:`v` along dimension :attr:`dim` is transformed as
.. math::
v = \frac{v}{\max(\lVert v \rVert_p, \epsilon)}.
With the default arguments it uses the Euclidean norm over vectors along dimension :math:`1` for normalization.
Args:
input: input tensor of any shape
p (float): the exponent value in the norm formulation. Default: 2
dim (int): the dimension to reduce. Default: 1
eps (float): small value to avoid division by zero. Default: 1e-12
out (Tensor, optional): the output tensor. If :attr:`out` is used, this
operation won't be differentiable.
)r} r/ r rm NT)r )rm ) r r normalizeZnormZ clamp_minZ expand_asZ
clamp_min_r0 r )r r} r/ r rm Zdenomr6 r6 r7 rk I s
rk )argr: messager# c C s( t | ts$t| dks$t|j|d S )Nr )r] r- rT r, rV )rl r: rm r6 r6 r7 assert_int_or_pairf s rn )r r rE rD rC r# c C s t | r tt| f| ||||dS | j dkrd}t|d| t|d| t|d| t|d| tjjj| t |t |t |t |S t
dj | j d S )
a Extracts sliding local blocks from a batched input tensor.
.. warning::
Currently, only 4-D input tensors (batched image-like tensors) are
supported.
.. warning::
More than one element of the unfolded tensor may refer to a single
memory location. As a result, in-place operations (especially ones that
are vectorized) may result in incorrect behavior. If you need to write
to the tensor, please clone it first.
See :class:`torch.nn.Unfold` for details
)rE rD rC r9 z&{} must be int or 2-tuple for 4D inputr rE rD rC z:Input Error: Only 4D input Tensors are supported (got {}D)N) r r unfoldr/ rn r0 r2 r3 Zim2colr r rV )r r rE rD rC msgr6 r6 r7 ro j s &ro )r r r rE rD rC r# c
C s t | r"tt| f| |||||dS | j dkrd}t|d| t|d| t|d| t|d| t|d| tjjj| t |t |t |t |t |S t
d j | j d
S ) zCombines an array of sliding local blocks into a large containing
tensor.
.. warning::
Currently, only 3-D output tensors (unfolded batched image-like tensors) are
supported.
See :class:`torch.nn.Fold` for details
)rE rD rC r$ z&{} must be int or 2-tuple for 3D inputr r rE rD rC z:Input Error: Only 3D input Tensors are supported (got {}D)N) r r foldr/ rn r0 r2 r3 Zcol2imr r rV )r r r rE rD rC rp r6 r6 r7 rq s $rq )r rD r# c C s | j }|dd }t|}xt|D ]x\}}||d d |ksJtd||d d |kshtd||d d ||d d | dks$tdq$W |dd }xBt|D ]6\}}||||d d ||d d f7 }qW tj|| j| j| jd}|dkr~t |d
d} |d t |d d }
t |d d} |d t |d d } | d| | f |d| |
f< n|dkrLt |d d} |d t |d d }
t |d d}
|d
t |d d }t |d d} |d t |d d } t |d d}|d
t |d d }| d| | ||f |d| |
|
|f< n"|d
krnt |d d} |d t |d d }
t |d d}
|d
t |d d }t |d d}|d t |d d }t |d d} |d t |d d } t |d! d}|d
t |d" d }t |d# d}|d t |d$ d }| d| | ||||f |d| |
|
|||f< |d% dkr|d |d& t |d' d }|d t |d( d }d}|d) }|dddd||f |dddd||f< |d* dkrft |d+ d}t |d, d|d- }|d |d. }|d }|dddd||f |dddd||f< t|dkr|d/ dkr|d
|d0 t |d1 d }|d
t |d2 d }d}|d3 }|dddddd||f |dddddd||f< |d4 dkrt |d5 d}t |d6 d|d7 }|d
|d8 }|d
}|dddddd||f |dddddd||f< t|d kr|d9 dkr(|d |d: t |d; d }|d t |d< d }d}|d= }|dddddddd||f |dddddddd||f< |d> dkrt |d? d}t |d@ d|dA }|d |dB }|d }|dddddddd||f |dddddddd||f< |S )Ca Circularly pads tensor.
Tensor values at the beginning are used to pad the end, and values at the
end are used to pad the beginning. For example, consider a single dimension
with values [0, 1, 2, 3]. With circular padding of (1, 1) it would be
padded to [3, 0, 1, 2, 3, 0], and with padding (1, 2) it would be padded to
[3, 0, 1, 2, 3, 0, 1]. If negative padding is applied then the ends of the
tensor get removed. With circular padding of (-1, -1) the previous example
would become [1, 2]. Circular padding of (-1, 1) would produce
[1, 2, 3, 1].
The first and second dimensions of the tensor are not padded.
Args:
input: Tensor with shape :math:`(N, C, D[, H, W])`.
padding: Tuple containing the number of elements to pad each side of
the tensor. The length of padding must be twice the number of
paddable dimensions. For example, the length of padding should be 4
for a tensor of shape :math:`(N, C, H, W)`, and the length should
be 6 for a tensor of shape :math:`(N, C, D, H, W)`.
Examples::
>>> x = torch.tensor([[[[0, 1, 2], [3, 4, 5]]]]) # Create tensor
>>> # Example 1
>>> padding = (1, 1, 1, 1)
>>> y = F.pad(x, padding, mode='circular')
>>> print(y)
tensor([[[[5, 3, 4, 5, 3],
[2, 0, 1, 2, 0],
[5, 3, 4, 5, 3],
[2, 0, 1, 2, 0]]]])
>>> print(y.shape)
torch.Size([1, 1, 4, 5])
>>> # Example 2
>>> padding = (1, 1, 2, 2)
>>> z = F.pad(x, padding, mode='circular')
>>> print(z)
tensor([[[[2, 0, 1, 2, 0],
[5, 3, 4, 5, 3],
[2, 0, 1, 2, 0],
[5, 3, 4, 5, 3],
[2, 0, 1, 2, 0],
[5, 3, 4, 5, 3]]]])
>>> print(z.shape)
torch.Size([1, 1, 6, 5])
r Nr z4Padding value causes wrapping around more than once.r z:Negative padding value is resulting in an empty dimension.)r% layoutr& .r9 r$ rL rM r' r( r' r( r' r( r) r' r( rs r) r' r( rs r) ir' r( rs r) irt r' r' r( r( r' r( r' r' r( r( rs rs r) r) rs r) rs rs r) r) iirt rt irt iirt rt )
r rT enumerater, r0 emptyr% rr r& r )r rD Zin_shapeZpaddable_shaper idxr. Z out_shaperm Zout_d0Zout_d1Zin_d0Zin_d1Zout_h0Zout_h1Zin_h0Zin_h1Zout_w0Zout_w1Zin_w0Zin_w1Zi0i1Zo0Zo1r6 r6 r7 r] s 0 .
2
,
4
00 << HHr] )qr vwbr# c C s | j d}||kr| |kr0t| ||jdddS |j||d g\}}|dkrXd }} n|j||d g\}} t| ||ft||| jddd S nX|jd\}}
} |dkrd } } }
n|jd\}} }
t| ||t||
| t|| |
fS dS ) aK
Performs the in-projection step of the attention operation, using packed weights.
Output is a triple containing projection tensors for query, key and value.
Args:
q, k, v: query, key and value tensors to be projected. For self-attention,
these are typically the same tensor; for encoder-decoder attention,
k and v are typically the same tensor. (We take advantage of these
identities for performance if they are present.) Regardless, q, k and v
must share a common embedding dimension; otherwise their shapes may vary.
w: projection weights for q, k and v, packed into a single tensor. Weights
are packed along dimension 0, in q, k, v order.
b: optional projection biases for q, k and v, packed into a single tensor
in q, k, v order.
Shape:
Inputs:
- q: :math:`(..., E)` where E is the embedding dimension
- k: :math:`(..., E)` where E is the embedding dimension
- v: :math:`(..., E)` where E is the embedding dimension
- w: :math:`(E * 3, E)` where E is the embedding dimension
- b: :math:`E * 3` where E is the embedding dimension
Output:
- in output list :math:`[q', k', v']`, each output tensor will have the
same shape as the corresponding input tensor.
r r$ )r/ r Nr( r( r( )r. r chunksplit)ry r rz r{ r| Ew_qZw_kvb_qZb_kvw_kw_vb_kb_vr6 r6 r7 _in_projection_packed\ s "
&r )
ry r rz r r r r r r r# c C sD | j d
|j d |j d } }
} |j| | fksJtd| | f d|j |j| |
fksrtd| |
f d|j |j| | fkstd| | f d|j |dks|j| fkstd| f d|j |dks|j| fkstd| f d|j |dks"|j| fks"td | f d|j t| ||t|||t|||fS )
a
Performs the in-projection step of the attention operation. This is simply
a triple of linear projections, with shape constraints on the weights which
ensure embedding dimension uniformity in the projected outputs.
Output is a triple containing projection tensors for query, key and value.
Args:
q, k, v: query, key and value tensors to be projected.
w_q, w_k, w_v: weights for q, k and v, respectively.
b_q, b_k, b_v: optional biases for q, k and v, respectively.
Shape:
Inputs:
- q: :math:`(Qdims..., Eq)` where Eq is the query embedding dimension and Qdims are any
number of leading dimensions.
- k: :math:`(Kdims..., Ek)` where Ek is the key embedding dimension and Kdims are any
number of leading dimensions.
- v: :math:`(Vdims..., Ev)` where Ev is the value embedding dimension and Vdims are any
number of leading dimensions.
- w_q: :math:`(Eq, Eq)`
- w_k: :math:`(Eq, Ek)`
- w_v: :math:`(Eq, Ev)`
- b_q: :math:`(Eq)`
- b_k: :math:`(Eq)`
- b_v: :math:`(Eq)`
Output: in output triple :math:`(q', k', v')`,
- q': :math:`[Qdims..., Eq]`
- k': :math:`[Kdims..., Eq]`
- v': :math:`[Vdims..., Eq]`
r z!expecting query weights shape of z
, but got z expecting key weights shape of z!expecting value weights shape of Nz expecting query bias shape of z expecting key bias shape of z expecting value bias shape of r( r( r( )r. r r, r ) ry r rz r r r r r r EqZEkZEvr6 r6 r7 _in_projection s +"(((,,0r )ry r rz attn_mask dropout_pr# c
C sr | j \}}}| tj| } tj| |jdd}|dk r>||7 }t|d d}|dkr^t||d}tj||} | |fS )
a
Computes scaled dot product attention on query, key and value tensors, using
an optional attention mask if passed, and applying dropout if a probability
greater than 0.0 is specified.
Returns a tensor pair containing attended values and attention weights.
Args:
q, k, v: query, key and value tensors. See Shape section for shape details.
attn_mask: optional tensor containing mask values to be added to calculated
attention. May be 2D or 3D; see Shape section for details.
dropout_p: dropout probability. If greater than 0.0, dropout is applied.
Shape:
- q: :math:`(B, Nt, E)` where B is batch size, Nt is the target sequence length,
and E is embedding dimension.
- key: :math:`(B, Ns, E)` where B is batch size, Ns is the source sequence length,
and E is embedding dimension.
- value: :math:`(B, Ns, E)` where B is batch size, Ns is the source sequence length,
and E is embedding dimension.
- attn_mask: either a 3D tensor of shape :math:`(B, Nt, Ns)` or a 2D tensor of
shape :math:`(Nt, Ns)`.
- Output: attention values have shape :math:`(B, Nt, E)`; attention weights
have shape :math:`(B, Nt, Ns)`
r r N)r/ g )r} r' r( r( )r r" sqrtr0 Zbmm transposer r )
ry r rz r r BZNtr Zattnri r6 r6 r7 _scaled_dot_product_attention s r )querykeyr embed_dim_to_check num_headsin_proj_weight in_proj_biasbias_kbias_v
add_zero_attnr out_proj_weight
out_proj_biasr~ key_padding_mask need_weightsr use_separate_proj_weight
q_proj_weight
k_proj_weight
v_proj_weightstatic_kstatic_vr# c * C s | ||||||| | f }t |rXtt|| ||||||||| |
| | |
|||||||||dS | j\}}}|j\}} } ||kstd| d| t|tjr|j|dd} n|| } | | |kstd| d| |r |jdd |jdd ksDtd
|jdd d |jdd n&|j|jksDtd |j d
|j |sbt | ||||\} } } n||dk sttd|dk std|dk std|dkrd }! }"}#n|j
d\}!}"}#t | ||||||!|"|# \} } } |dk r|j tj
krtjd |jtj}n(|j s6|j tjks6td|j |j d kr|||f}$|j|$krptd|j d|$ d|jd}nV|j dkr|| ||f}%|j|%krtd|j d|% dntd|j d|dk r |j tj
kr tjd |jtj}|dk r|dk r|dks&td |dks8td tj| |jd |d g} tj| |jd |d g} |dk r|t|d)}|dk rt|d*}n |dkst|dkst| j j||| | jdd } |dkr| j j| jd || | jdd } n`|j d|| ks.td || d|j d |j d | ksXtd | d|j d |} |dkr| j j| jd || | jdd } n`|j d|| kstd!|| d|j d |j d | kstd"| d|j d |} | rl|| d | f}&tj| tj |&| j | j d#gd d$} tj| tj |&| j | j d#gd d$} |dk rXt|d+}|dk rlt|d,}| j d }|dk r|j||fkstd%||f d|j |j|d d |j d-|d.d/j || d |}|dkr|}n*|j tjkr|j!|}n|j"|t#d&}|dk rJ|j tjkrJtj$|tj#d'}'|'j%|t#d& |'}|
sTd(}
t&| | | ||
\}(})|(jdd j j|||}(t'|(| | }(|r|)j||||})|(|)j(d d$| fS |(dfS dS )0a?
Args:
query, key, value: map a query and a set of key-value pairs to an output.
See "Attention Is All You Need" for more details.
embed_dim_to_check: total dimension of the model.
num_heads: parallel attention heads.
in_proj_weight, in_proj_bias: input projection weight and bias.
bias_k, bias_v: bias of the key and value sequences to be added at dim=0.
add_zero_attn: add a new batch of zeros to the key and
value sequences at dim=1.
dropout_p: probability of an element to be zeroed.
out_proj_weight, out_proj_bias: the output projection weight and bias.
training: apply dropout if is ``True``.
key_padding_mask: if provided, specified padding elements in the key will
be ignored by the attention. This is an binary mask. When the value is True,
the corresponding value on the attention layer will be filled with -inf.
need_weights: output attn_output_weights.
attn_mask: 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all
the batches while a 3D mask allows to specify a different mask for the entries of each batch.
use_separate_proj_weight: the function accept the proj. weights for query, key,
and value in different forms. If false, in_proj_weight will be used, which is
a combination of q_proj_weight, k_proj_weight, v_proj_weight.
q_proj_weight, k_proj_weight, v_proj_weight, in_proj_bias: input projection weight and bias.
static_k, static_v: static key and value used for attention operators.
Shape:
Inputs:
- query: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is
the embedding dimension.
- key: :math:`(S, N, E)`, where S is the source sequence length, N is the batch size, E is
the embedding dimension.
- value: :math:`(S, N, E)` where S is the source sequence length, N is the batch size, E is
the embedding dimension.
- key_padding_mask: :math:`(N, S)` where N is the batch size, S is the source sequence length.
If a ByteTensor is provided, the non-zero positions will be ignored while the zero positions
will be unchanged. If a BoolTensor is provided, the positions with the
value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged.
- attn_mask: 2D mask :math:`(L, S)` where L is the target sequence length, S is the source sequence length.
3D mask :math:`(N*num_heads, L, S)` where N is the batch size, L is the target sequence length,
S is the source sequence length. attn_mask ensures that position i is allowed to attend the unmasked
positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend
while the zero positions will be unchanged. If a BoolTensor is provided, positions with ``True``
are not allowed to attend while ``False`` values will be unchanged. If a FloatTensor
is provided, it will be added to the attention weight.
- static_k: :math:`(N*num_heads, S, E/num_heads)`, where S is the source sequence length,
N is the batch size, E is the embedding dimension. E/num_heads is the head dimension.
- static_v: :math:`(N*num_heads, S, E/num_heads)`, where S is the source sequence length,
N is the batch size, E is the embedding dimension. E/num_heads is the head dimension.
Outputs:
- attn_output: :math:`(L, N, E)` where L is the target sequence length, N is the batch size,
E is the embedding dimension.
- attn_output_weights: :math:`(N, L, S)` where N is the batch size,
L is the target sequence length, S is the source sequence length.
)
r~ r r r r r r r r r z%was expecting embedding dimension of z
, but got trunc)Z
rounding_modez
embed_dim z not divisible by num_heads Nr z key's sequence and batch dims z do not match value's z
key shape z does not match value shape z:use_separate_proj_weight is True but q_proj_weight is Nonez:use_separate_proj_weight is True but k_proj_weight is Nonez:use_separate_proj_weight is True but v_proj_weight is Noner$ zZByte tensor for attn_mask in nn.MultiheadAttention is deprecated. Use bool tensor instead.zBOnly float, byte, and bool types are supported for attn_mask, not z!The shape of the 2D attn_mask is z, but should be .r z!The shape of the 3D attn_mask is zattn_mask's dimension z is not supportedzaByte tensor for key_padding_mask in nn.MultiheadAttention is deprecated. Use bool tensor instead.z#bias cannot be added to static key.z%bias cannot be added to static value.r z expecting static_k.size(0) of z expecting static_k.size(2) of z expecting static_v.size(0) of z expecting static_v.size(2) of )r% r& )r/ z$expecting key_padding_mask shape of z-inf)r% g )r r )r r )r r )r r r( r( r( ))r r multi_head_attention_forwardr r, r] r0 Tensorr r r} r r% Zuint8r r Ztoboolr r/ r r` catrepeatr r r r r. rS r& r- r
logical_orZ masked_fillrK r Z masked_fill_r r r )*r r r r r r r r r r r r r r~ r r r r r r r r r Ztens_opsZtgt_lenZbszZ embed_dimZsrc_lenr Zhead_dimry r rz r r r Zcorrect_2d_sizeZcorrect_3d_sizeZzero_attn_shapeZ
new_attn_maskZ attn_outputZattn_output_weightsr6 r6 r7 r s Q
(&
&
& ""
r )NNFN)NNFN)NNFN)NNFN)Nr r FF)Nr r FF)Nr r FF)Nr r FF)Nr r FF)Nr r FF)Nr N)Nr N)Nr N)NF)NF)F)F)F)F)F)F)r| TF)r| FF)r| TF)r| TF)r| FF)F)Fr( )r( )r r F)F)r F)F)r F)r F ?UUUUUU?)r r FF)r| )Nr$ N)Nr$ Nr( )r Fr r( )Nr$ N)F)N)N)F)F)F)NNr FF) NNr Fr FNFN)NNFr r )NNNNTr r )NNr )NNr )r r r )r r F)NNr Nr )TFNr Nr )Fr r )NNr F)NNr Nr r) )NNNr )NNNr N)NNr r )r r )NNr )NNr )r NNr )r NNr )NNr )NNr )NNNr )r NNr )r r NNNr )NNrA N)NNrA N)NNrA N)NNrA NN)NNrA NN)NNrA NN)NNrA NN)NNrA NN)NN)NN)NN)NN)NN)NN)NN)NN)r rS N)N)r[ r) )r r F)r r r FNNr )r r rj N)r r r )r r r )N)NNN)Nr) )
TNTNFNNNNN)__doc__typingr r r r r" r r0 r Ztorch._Cr r Ztorch._torch_docsr r
Z
_jit_internalr r
r r r Z overridesr r r r r r r modulesr Z
modules.utilsr r r r r Zconv1drV Zconv2dZconv3dZconv_transpose1dZconv_transpose2dZconv_transpose3dZconv_tbcro r2 r3 rf r r- rK r r* r8 __name__r4 r@ rB rA rG rK rJ rM rO rN rP rR rQ rZ r\ r_ rc re rn rp rr rq rs ru rt rv rx rw rP ry r{ r r r r r r r r ri r r r r r r r r r r r r r r r r Z log_sigmoidr> r r r r Zsoftplusr r r r r r Z
softshrinkr r r r r r r r r r r r r r r r r r r
r r r r& r+ r, r/ r0 r5 r6 r7 r9 r: r; r< r= r? r@ Z
pixel_shuffleZpixel_unshuffleZchannel_shufflerD rE rQ rR Z GRID_SAMPLE_INTERPOLATION_MODESZGRID_SAMPLE_PADDING_MODESrX rZ r\ r r` ZpdistZcosine_similarityZone_hotre rg rk rn ro rq r] r r r r r6 r6 r6 r7 s0
(8 ( (< ( & & & & & & " (! ( (
""$ >" \ * - $ $
+ 8 > < Q N L A "> " " * &04=:>6 $8 Z(,
&a$f" - &+& $ $ /9 " / 6 ^