/usr/local/lib64/python3.6/site-packages/torch/nn/modules/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/nn/modules/__pycache__/rnn.cpython-36.pyc (44653B)
3
Eg @ s0 d dl Z d dlZd dlZd dlmZmZmZmZmZm Z d dl
Z
d dl
m Z ddl m
Z
ddlmZ ddlmZ dd lmZ d
d lmZ ejejd Zd e e ee d
ddZG dd de
ZG dd deZG dd deZ G dd deZ G dd de
Z G dd de Z G d d d e Z G d d d e Z!dS )! N)ListTupleOptionaloverloadUnioncast)Tensor )Module ) Parameter)PackedSequence)init )_VF)RNN_TANHRNN_RELU)tensor permutationdimreturnc C s | j ||S )N)Z index_select)r r r r @/usr/local/lib64/python3.6/site-packages/torch/nn/modules/rnn.pyapply_permutation s r c s e Zd ZU ddddddddd g Zd
gZeee e
e e
ee ed7eeeee e ee edd
fdd
Z fddZddddZ fddZdddd Zeee dd d d Zeee eeeef d d d!Zd8eeeeef edd#d$d%Z eeee d&d'd(Z eee d)d*d+Z d9e ee f ee ee ee f ef d,d-d.Z!edd/d0Z" fd1d2Z#e$e%e%e& dd3d4Z' fd5d6Z( Z)S ):RNNBasemode
input_size hidden_size
num_layersbias batch_firstdropout
bidirectional proj_size all_weightsr TF r N)
r r r r r r r! r" r# r c s |
| d} t tj |_|_|_|_|_|_t |_
|_ | _ |rZdnd}
t
|tj sd| ko~dkn st
|trtd|dkr|dkrtjdj|| | dk rtd| |krtd|d krd
| }n>|d krd | }n,|d
kr|}n |dkr |}n td| g _g _xt|D ] xt|
D ]v}| dkrh| n|} dkrz|n||
}ttj||ff| }ttj||ff| }ttj|f| }ttj|f| }f }j dkr |r||||f}n||f}n6ttj| |ff| }|r,|||||f}n
|||f}|dkrDdndddg}|rb|ddg7 }j dkrx|dg7 } fdd|D }x$t||D ]\}}t || qW jj | jj | qTW qBW fddjD _ j j! d S )N)devicedtyper r r zbdropout should be a number in range [0, 1] representing the probability of an element being zeroedzdropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout={} and num_layers={}zEproj_size should be a positive integer or zero to disable projectionsz,proj_size has to be smaller than hidden_sizeLSTM GRUr r r zUnrecognized RNN mode: _reverse zweight_ih_l{}{}zweight_hh_l{}{}z
bias_ih_l{}{}z
bias_hh_l{}{}zweight_hr_l{}{}c s g | ]}|j qS r )format).0x)layersuffixr r
r s z$RNNBase.__init__..c s g | ]} fd d|qS )c s t | rt | S d S )N)hasattrgetattr)wn)selfr r y s z-RNNBase.__init__...r )r. r5 )r6 r r r2 y s )"superr __init__r r r r r r floatr! r" r#
isinstancenumbersNumberbool
ValueErrorwarningswarnr- _flat_weights_names _all_weightsranger torchemptyzipsetattrextendappend
_flat_weightsflatten_parametersreset_parameters)r6 r r r r r r r! r" r# r& r' factory_kwargsnum_directionsZ gate_size directionreal_hidden_sizeZlayer_input_sizeZw_ihZw_hhZb_ihZb_hhZ layer_paramsZw_hrZ param_namesnameparam) __class__)r0 r6 r1 r r9 ' s
(
zRNNBase.__init__c s@ t | dr*|| jkr*| jj|}|| j|< tt| j|| d S )NrB )r3 rB indexrK r8 r __setattr__)r6 attrvalueidx)rT r r rV ~ s
zRNNBase.__setattr__)r c C sZ t | jt | jkrdS x | jD ]}t|ts dS q W | jd }|j}xJ| jD ]@}t|jt s|jj|k s|jj stj j
j |j rNdS qNW t dd | jD }t |t | jkrdS tj
j| ddlj j
j}tj f tj rB| jrdnd}| jdkr|d7 }tj| j|| j|j| j| j| j| j| jt | j W dQ R X W dQ R X dS )zResets parameter data pointer so that they can use faster code paths.
Right now, this works only if the module is on the GPU and cuDNN is enabled.
Otherwise, it's a no-op.
Nr c s s | ]}|j V qd S )N)Zdata_ptr)r. pr r r s z-RNNBase.flatten_parameters..r) r r ) lenrK rB r; r r' dataZis_cudarE backendsZcudnnZ
is_acceptablesetZcudaZ device_ofZtorch.backends.cudnn.rnnrnnZno_gradZ _use_cudnn_rnn_flatten_weightr r# Z_cudnn_rnn_flatten_weightr Zget_cudnn_moder r r r r> r" )r6 wZfirst_fwr' fwZunique_data_ptrsr` Z num_weightsr r r rL s6
zRNNBase.flatten_parametersc s2 t t j|} fdd jD _ j |S )Nc s g | ]} fd d|qS )c s t | rt | S d S )N)r3 r4 )r5 )r6 r r r7 s z+RNNBase._apply...r )r. r5 )r6 r r r2 s z"RNNBase._apply..)r8 r _applyrB rK rL )r6 fnret)rT )r6 r rc s zRNNBase._applyc C s6 dt j| j }x | j D ]}tj|| | qW d S )Ng ?)mathsqrtr
parametersr uniform_)r6 stdvweightr r r rM s zRNNBase.reset_parameters)input batch_sizesr c C s\ |d k r dnd}|j |kr0tdj||j | j|jdkrXtdj| j|jdd S )Nr r z%input must have {} dimensions, got {}r z?input.size(-1) must be equal to input_size. Expected {}, got {}rn )r RuntimeErrorr- r size)r6 rl rm Zexpected_input_dimr r r check_input s zRNNBase.check_inputc C sr |d k rt |d }n| jr&|jdn|jd}| jr:dnd}| jdkr\| j| || jf}n| j| || jf}|S )Nr r r )intr rp r" r# r r )r6 rl rm
mini_batchrO expected_hidden_sizer r r get_expected_hidden_size s
z RNNBase.get_expected_hidden_size Expected hidden size {}, got {})hxrt msgr c C s( |j |kr$t|j|t|j d S )N)rp ro r- list)r6 rw rt rx r r r check_hidden_size s zRNNBase.check_hidden_size)rl hiddenrm c C s( | j || | j||}| j|| d S )N)rq ru rz )r6 rl r{ rm rt r r r check_forward_args s zRNNBase.check_forward_args)rw r c C s |d kr |S t ||S )N)r )r6 rw r r r r permute_hidden s zRNNBase.permute_hidden)rl rw r c
C s^ t |t}|r(|\}}}}t|d }n0tt|}d }| jrF|jdn|jd}d }d }|d krtt|}| jrtdnd}tj | j
| || j |j |j
d}n | j||}|d k sttt|}| j||| t| j } |d kr| ||| j| j| j
| j| j| j| j }
n$| |||| j| j| j
| j| j| j }
|
d } |
d } |rNt| |||} | | j| |fS )Nr r r )r' r& )r; r
rr r r r rp r" rE zerosr r r' r& r} AssertionErrorr|
_rnn_implsr rK r r! training)
r6 rl rw Z is_packedrm sorted_indicesunsorted_indicesmax_batch_sizerO _implresultoutputr{ r r r forward s<
zRNNBase.forwardc C s~ d}| j dkr|d7 }| jdkr(|d7 }| jdk r:|d7 }| jdk rL|d 7 }| jdkr^|d
7 }| jdk rp|d 7 }|jf | jS ) Nz{input_size}, {hidden_size}r z, proj_size={proj_size}r z, num_layers={num_layers}Tz
, bias={bias}Fz, batch_first={batch_first}z, dropout={dropout}z , bidirectional={bidirectional})r# r r r r! r" r- __dict__)r6 sr r r
extra_repr s
zRNNBase.extra_reprc s t tj| d|kr"|d _d|kr0d_tjd d trHd S j}jrXdnd}g _ g _x:t
|D ], x$t
|D ]}|dkrdnddd d
d d g} fd
d|D }j r jdkr j|g7 _j j | n, j|d d g7 _j j |d d qjdkrr j|d d g|dd g 7 _j j |d d |dd g q j|d d g7 _j j |d d qW qtW fddj D _
d S )Nr$ r# r r r r+ r, zweight_ih_l{}{}zweight_hh_l{}{}z
bias_ih_l{}{}z
bias_hh_l{}{}zweight_hr_l{}{}c s g | ]}|j qS r )r- )r. r/ )r0 r1 r r r2 ? s z(RNNBase.__setstate__..r) c s g | ]} fd d|qS )c s t | rt | S d S )N)r3 r4 )r5 )r6 r r r7 N s z1RNNBase.__setstate__...r )r. r5 )r6 r r r2 N s rn rn )r8 r __setstate__rC r# r; strr r" rB rD r rI rK )r6 dr rO rP weights)rT )r0 r6 r1 r r * s:
&$ zRNNBase.__setstate__c s fdd j D S )Nc s g | ]} fd d|D qS )c s g | ]}t |qS r )r4 )r. rk )r6 r r r2 R s z2RNNBase.all_weights...r )r. r )r6 r r r2 R s z'RNNBase.all_weights..)rC )r6 r )r6 r r$ P s zRNNBase.all_weightsc s2 t t| j }|jd d |_|jd d |_|S )N)r8 r _replicate_for_data_parallelrK rB )r6 Zreplica)rT r r r T s z$RNNBase._replicate_for_data_parallel)r TFr% Fr NN)rv )N)*__name__
__module__ __qualname__
__constants__Z__jit_unused_properties__r r rr r r r r> r r r: r! r" r# r9 rV rL rc rM r r rq r ru rz r| r} r r
r r r propertyr r r$ r
__classcell__r r )rT r r s<
&T 0 0) &r c s e Zd ZdZ fddZ ZS )RNNaF Applies a multi-layer Elman RNN with :math:`\tanh` or :math:`\text{ReLU}` non-linearity to an
input sequence.
For each element in the input sequence, each layer computes the following
function:
.. math::
h_t = \tanh(W_{ih} x_t + b_{ih} + W_{hh} h_{(t-1)} + b_{hh})
where :math:`h_t` is the hidden state at time `t`, :math:`x_t` is
the input at time `t`, and :math:`h_{(t-1)}` is the hidden state of the
previous layer at time `t-1` or the initial hidden state at time `0`.
If :attr:`nonlinearity` is ``'relu'``, then :math:`\text{ReLU}` is used instead of :math:`\tanh`.
Args:
input_size: The number of expected features in the input `x`
hidden_size: The number of features in the hidden state `h`
num_layers: Number of recurrent layers. E.g., setting ``num_layers=2``
would mean stacking two RNNs together to form a `stacked RNN`,
with the second RNN taking in outputs of the first RNN and
computing the final results. Default: 1
nonlinearity: The non-linearity to use. Can be either ``'tanh'`` or ``'relu'``. Default: ``'tanh'``
bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`.
Default: ``True``
batch_first: If ``True``, then the input and output tensors are provided
as `(batch, seq, feature)` instead of `(seq, batch, feature)`.
Note that this does not apply to hidden or cell states. See the
Inputs/Outputs sections below for details. Default: ``False``
dropout: If non-zero, introduces a `Dropout` layer on the outputs of each
RNN layer except the last layer, with dropout probability equal to
:attr:`dropout`. Default: 0
bidirectional: If ``True``, becomes a bidirectional RNN. Default: ``False``
Inputs: input, h_0
* **input**: tensor of shape :math:`(L, N, H_{in})` when ``batch_first=False`` or
:math:`(N, L, H_{in})` when ``batch_first=True`` containing the features of
the input sequence. The input can also be a packed variable length sequence.
See :func:`torch.nn.utils.rnn.pack_padded_sequence` or
:func:`torch.nn.utils.rnn.pack_sequence` for details.
* **h_0**: tensor of shape :math:`(D * \text{num\_layers}, N, H_{out})` containing the initial hidden
state for each element in the batch. Defaults to zeros if not provided.
where:
.. math::
\begin{aligned}
N ={} & \text{batch size} \\
L ={} & \text{sequence length} \\
D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\
H_{in} ={} & \text{input\_size} \\
H_{out} ={} & \text{hidden\_size}
\end{aligned}
Outputs: output, h_n
* **output**: tensor of shape :math:`(L, N, D * H_{out})` when ``batch_first=False`` or
:math:`(N, L, D * H_{out})` when ``batch_first=True`` containing the output features
`(h_t)` from the last layer of the RNN, for each `t`. If a
:class:`torch.nn.utils.rnn.PackedSequence` has been given as the input, the output
will also be a packed sequence.
* **h_n**: tensor of shape :math:`(D * \text{num\_layers}, N, H_{out})` containing the final hidden state
for each element in the batch.
Attributes:
weight_ih_l[k]: the learnable input-hidden weights of the k-th layer,
of shape `(hidden_size, input_size)` for `k = 0`. Otherwise, the shape is
`(hidden_size, num_directions * hidden_size)`
weight_hh_l[k]: the learnable hidden-hidden weights of the k-th layer,
of shape `(hidden_size, hidden_size)`
bias_ih_l[k]: the learnable input-hidden bias of the k-th layer,
of shape `(hidden_size)`
bias_hh_l[k]: the learnable hidden-hidden bias of the k-th layer,
of shape `(hidden_size)`
.. note::
All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})`
where :math:`k = \frac{1}{\text{hidden\_size}}`
.. note::
For bidirectional RNNs, forward and backward are directions 0 and 1 respectively.
Example of splitting the output layers when ``batch_first=False``:
``output.view(seq_len, batch, num_directions, hidden_size)``.
.. include:: ../cudnn_rnn_determinism.rst
.. include:: ../cudnn_persistent_rnn.rst
Examples::
>>> rnn = nn.RNN(10, 20, 2)
>>> input = torch.randn(5, 3, 10)
>>> h0 = torch.randn(2, 3, 20)
>>> output, hn = rnn(input, h0)
c sj d|krt d|jdd| _| jdkr.d}n | jdkr>d}nt dj| jtt| j|f|| d S ) Nr# z=proj_size argument is only supported for LSTM, not RNN or GRU nonlinearitytanhr relur zUnknown nonlinearity '{}')r? popr r- r8 r r9 )r6 argskwargsr )rT r r r9 s
z RNN.__init__)r r r __doc__r9 r r r )rT r r ] s ^r c
s e Zd ZdZ fddZeee eeeef dddZ eeeef ee ddd Z
eeef ee eeef d
d d Z e e
jjdeeeeef eeeeef f dddZe e
jjdeeeeef eeeeef f dddZdddZ ZS )r( a Applies a multi-layer long short-term memory (LSTM) RNN to an input
sequence.
For each element in the input sequence, each layer computes the following
function:
.. math::
\begin{array}{ll} \\
i_t = \sigma(W_{ii} x_t + b_{ii} + W_{hi} h_{t-1} + b_{hi}) \\
f_t = \sigma(W_{if} x_t + b_{if} + W_{hf} h_{t-1} + b_{hf}) \\
g_t = \tanh(W_{ig} x_t + b_{ig} + W_{hg} h_{t-1} + b_{hg}) \\
o_t = \sigma(W_{io} x_t + b_{io} + W_{ho} h_{t-1} + b_{ho}) \\
c_t = f_t \odot c_{t-1} + i_t \odot g_t \\
h_t = o_t \odot \tanh(c_t) \\
\end{array}
where :math:`h_t` is the hidden state at time `t`, :math:`c_t` is the cell
state at time `t`, :math:`x_t` is the input at time `t`, :math:`h_{t-1}`
is the hidden state of the layer at time `t-1` or the initial hidden
state at time `0`, and :math:`i_t`, :math:`f_t`, :math:`g_t`,
:math:`o_t` are the input, forget, cell, and output gates, respectively.
:math:`\sigma` is the sigmoid function, and :math:`\odot` is the Hadamard product.
In a multilayer LSTM, the input :math:`x^{(l)}_t` of the :math:`l` -th layer
(:math:`l >= 2`) is the hidden state :math:`h^{(l-1)}_t` of the previous layer multiplied by
dropout :math:`\delta^{(l-1)}_t` where each :math:`\delta^{(l-1)}_t` is a Bernoulli random
variable which is :math:`0` with probability :attr:`dropout`.
If ``proj_size > 0`` is specified, LSTM with projections will be used. This changes
the LSTM cell in the following way. First, the dimension of :math:`h_t` will be changed from
``hidden_size`` to ``proj_size`` (dimensions of :math:`W_{hi}` will be changed accordingly).
Second, the output hidden state of each layer will be multiplied by a learnable projection
matrix: :math:`h_t = W_{hr}h_t`. Note that as a consequence of this, the output
of LSTM network will be of different shape as well. See Inputs/Outputs sections below for exact
dimensions of all variables. You can find more details in https://arxiv.org/abs/1402.1128.
Args:
input_size: The number of expected features in the input `x`
hidden_size: The number of features in the hidden state `h`
num_layers: Number of recurrent layers. E.g., setting ``num_layers=2``
would mean stacking two LSTMs together to form a `stacked LSTM`,
with the second LSTM taking in outputs of the first LSTM and
computing the final results. Default: 1
bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`.
Default: ``True``
batch_first: If ``True``, then the input and output tensors are provided
as `(batch, seq, feature)` instead of `(seq, batch, feature)`.
Note that this does not apply to hidden or cell states. See the
Inputs/Outputs sections below for details. Default: ``False``
dropout: If non-zero, introduces a `Dropout` layer on the outputs of each
LSTM layer except the last layer, with dropout probability equal to
:attr:`dropout`. Default: 0
bidirectional: If ``True``, becomes a bidirectional LSTM. Default: ``False``
proj_size: If ``> 0``, will use LSTM with projections of corresponding size. Default: 0
Inputs: input, (h_0, c_0)
* **input**: tensor of shape :math:`(L, N, H_{in})` when ``batch_first=False`` or
:math:`(N, L, H_{in})` when ``batch_first=True`` containing the features of
the input sequence. The input can also be a packed variable length sequence.
See :func:`torch.nn.utils.rnn.pack_padded_sequence` or
:func:`torch.nn.utils.rnn.pack_sequence` for details.
* **h_0**: tensor of shape :math:`(D * \text{num\_layers}, N, H_{out})` containing the
initial hidden state for each element in the batch.
Defaults to zeros if (h_0, c_0) is not provided.
* **c_0**: tensor of shape :math:`(D * \text{num\_layers}, N, H_{cell})` containing the
initial cell state for each element in the batch.
Defaults to zeros if (h_0, c_0) is not provided.
where:
.. math::
\begin{aligned}
N ={} & \text{batch size} \\
L ={} & \text{sequence length} \\
D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\
H_{in} ={} & \text{input\_size} \\
H_{cell} ={} & \text{hidden\_size} \\
H_{out} ={} & \text{proj\_size if } \text{proj\_size}>0 \text{ otherwise hidden\_size} \\
\end{aligned}
Outputs: output, (h_n, c_n)
* **output**: tensor of shape :math:`(L, N, D * H_{out})` when ``batch_first=False`` or
:math:`(N, L, D * H_{out})` when ``batch_first=True`` containing the output features
`(h_t)` from the last layer of the LSTM, for each `t`. If a
:class:`torch.nn.utils.rnn.PackedSequence` has been given as the input, the output
will also be a packed sequence.
* **h_n**: tensor of shape :math:`(D * \text{num\_layers}, N, H_{out})` containing the
final hidden state for each element in the batch.
* **c_n**: tensor of shape :math:`(D * \text{num\_layers}, N, H_{cell})` containing the
final cell state for each element in the batch.
Attributes:
weight_ih_l[k] : the learnable input-hidden weights of the :math:`\text{k}^{th}` layer
`(W_ii|W_if|W_ig|W_io)`, of shape `(4*hidden_size, input_size)` for `k = 0`.
Otherwise, the shape is `(4*hidden_size, num_directions * hidden_size)`. If
``proj_size > 0`` was specified, the shape will be
`(4*hidden_size, num_directions * proj_size)` for `k > 0`
weight_hh_l[k] : the learnable hidden-hidden weights of the :math:`\text{k}^{th}` layer
`(W_hi|W_hf|W_hg|W_ho)`, of shape `(4*hidden_size, hidden_size)`. If ``proj_size > 0``
was specified, the shape will be `(4*hidden_size, proj_size)`.
bias_ih_l[k] : the learnable input-hidden bias of the :math:`\text{k}^{th}` layer
`(b_ii|b_if|b_ig|b_io)`, of shape `(4*hidden_size)`
bias_hh_l[k] : the learnable hidden-hidden bias of the :math:`\text{k}^{th}` layer
`(b_hi|b_hf|b_hg|b_ho)`, of shape `(4*hidden_size)`
weight_hr_l[k] : the learnable projection weights of the :math:`\text{k}^{th}` layer
of shape `(proj_size, hidden_size)`. Only present when ``proj_size > 0`` was
specified.
weight_ih_l[k]_reverse: Analogous to `weight_ih_l[k]` for the reverse direction.
Only present when ``bidirectional=True``.
weight_hh_l[k]_reverse: Analogous to `weight_hh_l[k]` for the reverse direction.
Only present when ``bidirectional=True``.
bias_ih_l[k]_reverse: Analogous to `bias_ih_l[k]` for the reverse direction.
Only present when ``bidirectional=True``.
bias_hh_l[k]_reverse: Analogous to `bias_hh_l[k]` for the reverse direction.
Only present when ``bidirectional=True``.
weight_hr_l[k]_reverse: Analogous to `weight_hr_l[k]` for the reverse direction.
Only present when ``bidirectional=True`` and ``proj_size > 0`` was specified.
.. note::
All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})`
where :math:`k = \frac{1}{\text{hidden\_size}}`
.. note::
For bidirectional LSTMs, forward and backward are directions 0 and 1 respectively.
Example of splitting the output layers when ``batch_first=False``:
``output.view(seq_len, batch, num_directions, hidden_size)``.
.. include:: ../cudnn_rnn_determinism.rst
.. include:: ../cudnn_persistent_rnn.rst
Examples::
>>> rnn = nn.LSTM(10, 20, 2)
>>> input = torch.randn(5, 3, 10)
>>> h0 = torch.randn(2, 3, 20)
>>> c0 = torch.randn(2, 3, 20)
>>> output, (hn, cn) = rnn(input, (h0, c0))
c s t t| jd|| d S )Nr( )r( )r8 r( r9 )r6 r r )rT r r r9 d s z
LSTM.__init__)rl rm r c C sT |d k rt |d }n| jr&|jdn|jd}| jr:dnd}| j| || jf}|S )Nr r r )rr r rp r" r r )r6 rl rm rs rO rt r r r get_expected_cell_sizeg s
zLSTM.get_expected_cell_size)rl r{ rm c C sD | j || | j|d | j||d | j|d | j||d d S )Nr z"Expected hidden[0] size {}, got {}r z"Expected hidden[1] size {}, got {})rq rz ru r )r6 rl r{ rm r r r r| s s
zLSTM.check_forward_args)rw r r c C s( |d kr |S t |d |t |d |fS )Nr r )r )r6 rw r r r r r} s zLSTM.permute_hiddenN)rl rw r c C s d S )Nr )r6 rl rw r r r r s z LSTM.forwardc C s d S )Nr )r6 rl rw r r r r s c
C s |}t |tr,|\}}}}|d }t|}n&d }| jr@|jdn|jd}d }d }|d kr| jrddnd}| jdkrx| jn| j} tj | j
| || |j |j d}
tj | j
| || j|j |j d} |
| f}n | j
||}| j||| |d krtj||| j| j| j
| j| j| j| j } n&tj|||| j| j| j
| j| j| j } | d }
| dd }t |trxt|
|||}|| j
||fS |
| j
||fS d S )Nr r r )r' r& )r; r
rr r rp r" r# r rE r~ r r' r& r} r| r ZlstmrK r r! r )r6 rl rw
orig_inputrm r r r rO rQ Zh_zerosZc_zerosr r r{
output_packedr r r r s@
)N)N)N)r r r r r9 r r r rr r r| r} r rE
_jit_internal_overload_methodr r
r r r )rT r r( s&
""r( c s e Zd ZdZ fddZeejjd
e e
e e e e f dddZ eejjd e
e
e e e
e f dddZ d d dZ ZS )
r* a Applies a multi-layer gated recurrent unit (GRU) RNN to an input sequence.
For each element in the input sequence, each layer computes the following
function:
.. math::
\begin{array}{ll}
r_t = \sigma(W_{ir} x_t + b_{ir} + W_{hr} h_{(t-1)} + b_{hr}) \\
z_t = \sigma(W_{iz} x_t + b_{iz} + W_{hz} h_{(t-1)} + b_{hz}) \\
n_t = \tanh(W_{in} x_t + b_{in} + r_t * (W_{hn} h_{(t-1)}+ b_{hn})) \\
h_t = (1 - z_t) * n_t + z_t * h_{(t-1)}
\end{array}
where :math:`h_t` is the hidden state at time `t`, :math:`x_t` is the input
at time `t`, :math:`h_{(t-1)}` is the hidden state of the layer
at time `t-1` or the initial hidden state at time `0`, and :math:`r_t`,
:math:`z_t`, :math:`n_t` are the reset, update, and new gates, respectively.
:math:`\sigma` is the sigmoid function, and :math:`*` is the Hadamard product.
In a multilayer GRU, the input :math:`x^{(l)}_t` of the :math:`l` -th layer
(:math:`l >= 2`) is the hidden state :math:`h^{(l-1)}_t` of the previous layer multiplied by
dropout :math:`\delta^{(l-1)}_t` where each :math:`\delta^{(l-1)}_t` is a Bernoulli random
variable which is :math:`0` with probability :attr:`dropout`.
Args:
input_size: The number of expected features in the input `x`
hidden_size: The number of features in the hidden state `h`
num_layers: Number of recurrent layers. E.g., setting ``num_layers=2``
would mean stacking two GRUs together to form a `stacked GRU`,
with the second GRU taking in outputs of the first GRU and
computing the final results. Default: 1
bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`.
Default: ``True``
batch_first: If ``True``, then the input and output tensors are provided
as `(batch, seq, feature)` instead of `(seq, batch, feature)`.
Note that this does not apply to hidden or cell states. See the
Inputs/Outputs sections below for details. Default: ``False``
dropout: If non-zero, introduces a `Dropout` layer on the outputs of each
GRU layer except the last layer, with dropout probability equal to
:attr:`dropout`. Default: 0
bidirectional: If ``True``, becomes a bidirectional GRU. Default: ``False``
Inputs: input, h_0
* **input**: tensor of shape :math:`(L, N, H_{in})` when ``batch_first=False`` or
:math:`(N, L, H_{in})` when ``batch_first=True`` containing the features of
the input sequence. The input can also be a packed variable length sequence.
See :func:`torch.nn.utils.rnn.pack_padded_sequence` or
:func:`torch.nn.utils.rnn.pack_sequence` for details.
* **h_0**: tensor of shape :math:`(D * \text{num\_layers}, N, H_{out})` containing the initial hidden
state for each element in the batch. Defaults to zeros if not provided.
where:
.. math::
\begin{aligned}
N ={} & \text{batch size} \\
L ={} & \text{sequence length} \\
D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\
H_{in} ={} & \text{input\_size} \\
H_{out} ={} & \text{hidden\_size}
\end{aligned}
Outputs: output, h_n
* **output**: tensor of shape :math:`(L, N, D * H_{out})` when ``batch_first=False`` or
:math:`(N, L, D * H_{out})` when ``batch_first=True`` containing the output features
`(h_t)` from the last layer of the GRU, for each `t`. If a
:class:`torch.nn.utils.rnn.PackedSequence` has been given as the input, the output
will also be a packed sequence.
* **h_n**: tensor of shape :math:`(D * \text{num\_layers}, N, H_{out})` containing the final hidden state
for each element in the batch.
Attributes:
weight_ih_l[k] : the learnable input-hidden weights of the :math:`\text{k}^{th}` layer
(W_ir|W_iz|W_in), of shape `(3*hidden_size, input_size)` for `k = 0`.
Otherwise, the shape is `(3*hidden_size, num_directions * hidden_size)`
weight_hh_l[k] : the learnable hidden-hidden weights of the :math:`\text{k}^{th}` layer
(W_hr|W_hz|W_hn), of shape `(3*hidden_size, hidden_size)`
bias_ih_l[k] : the learnable input-hidden bias of the :math:`\text{k}^{th}` layer
(b_ir|b_iz|b_in), of shape `(3*hidden_size)`
bias_hh_l[k] : the learnable hidden-hidden bias of the :math:`\text{k}^{th}` layer
(b_hr|b_hz|b_hn), of shape `(3*hidden_size)`
.. note::
All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})`
where :math:`k = \frac{1}{\text{hidden\_size}}`
.. note::
For bidirectional GRUs, forward and backward are directions 0 and 1 respectively.
Example of splitting the output layers when ``batch_first=False``:
``output.view(seq_len, batch, num_directions, hidden_size)``.
.. include:: ../cudnn_persistent_rnn.rst
Examples::
>>> rnn = nn.GRU(10, 20, 2)
>>> input = torch.randn(5, 3, 10)
>>> h0 = torch.randn(2, 3, 20)
>>> output, hn = rnn(input, h0)
c s* d|krt dtt| jd|| d S )Nr# z=proj_size argument is only supported for LSTM, not RNN or GRUr* )r* )r? r8 r* r9 )r6 r r )rT r r r9 ) s z GRU.__init__N)rl rw r c C s d S )Nr )r6 rl rw r r r r . s z GRU.forwardc C s d S )Nr )r6 rl rw r r r r 3 s c
C sJ |}t |tr,|\}}}}|d }t|}n&d }| jr@|jdn|jd}d }d }|d kr| jrddnd}tj| j| || j |j
|j d}n | j ||}| j
||| |d krtj||| j| j| j| j| j| j| j } n&tj|||| j| j| j| j| j| j } | d }
| d } t |tr6t|
|||} | | j | |fS |
| j | |fS d S )Nr r r )r' r& )r; r
rr r rp r" rE r~ r r r' r& r} r| r ZgrurK r r! r )
r6 rl rw r rm r r r rO r r r{ r r r r r 8 s6
)N)N)N)r r r r r9 r rE r r r r r r r
r r r )rT r r* s e $$r* c sh e Zd ZU dddgZeeee
e d
eeeedd fdd
Z e
dd d
Zddd d Z ZS ) RNNCellBaser r r N)r r r
num_chunksr c s ||d}t t| j || _|| _|| _ttj|| |ff|| _ ttj|| |ff|| _
|rttj|| f|| _ ttj|| f|| _ n| j
dd | j
dd | j d S )N)r& r' bias_ihbias_hh)r8 r r9 r r r r rE rF weight_ih weight_hhr r Zregister_parameterrM )r6 r r r r r& r' rN )rT r r r9 l s
zRNNCellBase.__init__)r c C sJ d}d| j kr | jdk r |d7 }d| j kr<| jdkr<|d7 }|jf | j S )Nz{input_size}, {hidden_size}r Tz
, bias={bias}r r z , nonlinearity={nonlinearity})r r r r- )r6 r r r r r ~ s zRNNCellBase.extra_reprc C s6 dt j| j }x | j D ]}tj|| | qW d S )Ng ?)rf rg r rh r ri )r6 rj rk r r r rM s z RNNCellBase.reset_parameters)NN)r r r r rr r r r> r r r r r9 r r rM r r r )rT r r a s
r c sZ e Zd ZU dZddddgZedeeeedd fd
d
Z de
e e
e
d d
dZ Z
S )RNNCella An Elman RNN cell with tanh or ReLU non-linearity.
.. math::
h' = \tanh(W_{ih} x + b_{ih} + W_{hh} h + b_{hh})
If :attr:`nonlinearity` is `'relu'`, then ReLU is used in place of tanh.
Args:
input_size: The number of expected features in the input `x`
hidden_size: The number of features in the hidden state `h`
bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`.
Default: ``True``
nonlinearity: The non-linearity to use. Can be either ``'tanh'`` or ``'relu'``. Default: ``'tanh'``
Inputs: input, hidden
- **input** of shape `(batch, input_size)`: tensor containing input features
- **hidden** of shape `(batch, hidden_size)`: tensor containing the initial hidden
state for each element in the batch.
Defaults to zero if not provided.
Outputs: h'
- **h'** of shape `(batch, hidden_size)`: tensor containing the next hidden state
for each element in the batch
Shape:
- Input1: :math:`(N, H_{in})` tensor containing input features where
:math:`H_{in}` = `input_size`
- Input2: :math:`(N, H_{out})` tensor containing the initial hidden
state for each element in the batch where :math:`H_{out}` = `hidden_size`
Defaults to zero if not provided.
- Output: :math:`(N, H_{out})` tensor containing the next hidden state
for each element in the batch
Attributes:
weight_ih: the learnable input-hidden weights, of shape
`(hidden_size, input_size)`
weight_hh: the learnable hidden-hidden weights, of shape
`(hidden_size, hidden_size)`
bias_ih: the learnable input-hidden bias, of shape `(hidden_size)`
bias_hh: the learnable hidden-hidden bias, of shape `(hidden_size)`
.. note::
All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})`
where :math:`k = \frac{1}{\text{hidden\_size}}`
Examples::
>>> rnn = nn.RNNCell(10, 20)
>>> input = torch.randn(6, 3, 10)
>>> hx = torch.randn(3, 20)
>>> output = []
>>> for i in range(6):
hx = rnn(input[i], hx)
output.append(hx)
r r r r Tr N)r r r r r c s4 ||d}t t| j|||fddi| || _d S )N)r& r' r r )r8 r r9 r )r6 r r r r r& r' rN )rT r r r9 s
zRNNCell.__init__)rl rw r c C s |d kr&t j|jd| j|j|jd}| jdkrNtj||| j | j
| j | j }n<| jdkrvtj
||| j | j
| j | j }n|}tdj| j|S )Nr )r' r& r r zUnknown nonlinearity: {})rE r~ rp r r' r& r r Z
rnn_tanh_cellr r r r Z
rnn_relu_cellro r- )r6 rl rw re r r r r s
zRNNCell.forward)Tr NN)N)r r r r r r r rr r> r9 r r r r r r )rT r r s
8 r c sV e Zd ZdZd
eeedd fdd
Zd eee eef e eef ddd Z
Z S ) LSTMCella A long short-term memory (LSTM) cell.
.. math::
\begin{array}{ll}
i = \sigma(W_{ii} x + b_{ii} + W_{hi} h + b_{hi}) \\
f = \sigma(W_{if} x + b_{if} + W_{hf} h + b_{hf}) \\
g = \tanh(W_{ig} x + b_{ig} + W_{hg} h + b_{hg}) \\
o = \sigma(W_{io} x + b_{io} + W_{ho} h + b_{ho}) \\
c' = f * c + i * g \\
h' = o * \tanh(c') \\
\end{array}
where :math:`\sigma` is the sigmoid function, and :math:`*` is the Hadamard product.
Args:
input_size: The number of expected features in the input `x`
hidden_size: The number of features in the hidden state `h`
bias: If ``False``, then the layer does not use bias weights `b_ih` and
`b_hh`. Default: ``True``
Inputs: input, (h_0, c_0)
- **input** of shape `(batch, input_size)`: tensor containing input features
- **h_0** of shape `(batch, hidden_size)`: tensor containing the initial hidden
state for each element in the batch.
- **c_0** of shape `(batch, hidden_size)`: tensor containing the initial cell state
for each element in the batch.
If `(h_0, c_0)` is not provided, both **h_0** and **c_0** default to zero.
Outputs: (h_1, c_1)
- **h_1** of shape `(batch, hidden_size)`: tensor containing the next hidden state
for each element in the batch
- **c_1** of shape `(batch, hidden_size)`: tensor containing the next cell state
for each element in the batch
Attributes:
weight_ih: the learnable input-hidden weights, of shape
`(4*hidden_size, input_size)`
weight_hh: the learnable hidden-hidden weights, of shape
`(4*hidden_size, hidden_size)`
bias_ih: the learnable input-hidden bias, of shape `(4*hidden_size)`
bias_hh: the learnable hidden-hidden bias, of shape `(4*hidden_size)`
.. note::
All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})`
where :math:`k = \frac{1}{\text{hidden\_size}}`
Examples::
>>> rnn = nn.LSTMCell(10, 20) # (input_size, hidden_size)
>>> input = torch.randn(2, 3, 10) # (time_steps, batch, input_size)
>>> hx = torch.randn(3, 20) # (batch, hidden_size)
>>> cx = torch.randn(3, 20)
>>> output = []
>>> for i in range(input.size()[0]):
hx, cx = rnn(input[i], (hx, cx))
output.append(hx)
>>> output = torch.stack(output, dim=0)
TN)r r r r c s. ||d}t t| j|||fddi| d S )N)r& r' r r) )r8 r r9 )r6 r r r r& r' rN )rT r r r9 " s
zLSTMCell.__init__)rl rw r c C sJ |d kr.t j|jd| j|j|jd}||f}tj||| j| j | j
| j S )Nr )r' r& ) rE r~ rp r r' r& r Z lstm_cellr r r r )r6 rl rw r~ r r r r ' s zLSTMCell.forward)TNN)N) r r r r rr r> r9 r r r r r r r )rT r r s < r c sF e Zd ZdZd
eeedd fdd
Zd eee eddd Z Z
S ) GRUCellaP A gated recurrent unit (GRU) cell
.. math::
\begin{array}{ll}
r = \sigma(W_{ir} x + b_{ir} + W_{hr} h + b_{hr}) \\
z = \sigma(W_{iz} x + b_{iz} + W_{hz} h + b_{hz}) \\
n = \tanh(W_{in} x + b_{in} + r * (W_{hn} h + b_{hn})) \\
h' = (1 - z) * n + z * h
\end{array}
where :math:`\sigma` is the sigmoid function, and :math:`*` is the Hadamard product.
Args:
input_size: The number of expected features in the input `x`
hidden_size: The number of features in the hidden state `h`
bias: If ``False``, then the layer does not use bias weights `b_ih` and
`b_hh`. Default: ``True``
Inputs: input, hidden
- **input** of shape `(batch, input_size)`: tensor containing input features
- **hidden** of shape `(batch, hidden_size)`: tensor containing the initial hidden
state for each element in the batch.
Defaults to zero if not provided.
Outputs: h'
- **h'** of shape `(batch, hidden_size)`: tensor containing the next hidden state
for each element in the batch
Shape:
- Input1: :math:`(N, H_{in})` tensor containing input features where
:math:`H_{in}` = `input_size`
- Input2: :math:`(N, H_{out})` tensor containing the initial hidden
state for each element in the batch where :math:`H_{out}` = `hidden_size`
Defaults to zero if not provided.
- Output: :math:`(N, H_{out})` tensor containing the next hidden state
for each element in the batch
Attributes:
weight_ih: the learnable input-hidden weights, of shape
`(3*hidden_size, input_size)`
weight_hh: the learnable hidden-hidden weights, of shape
`(3*hidden_size, hidden_size)`
bias_ih: the learnable input-hidden bias, of shape `(3*hidden_size)`
bias_hh: the learnable hidden-hidden bias, of shape `(3*hidden_size)`
.. note::
All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})`
where :math:`k = \frac{1}{\text{hidden\_size}}`
Examples::
>>> rnn = nn.GRUCell(10, 20)
>>> input = torch.randn(6, 3, 10)
>>> hx = torch.randn(3, 20)
>>> output = []
>>> for i in range(6):
hx = rnn(input[i], hx)
output.append(hx)
TN)r r r r c s. ||d}t t| j|||fddi| d S )N)r& r' r r )r8 r r9 )r6 r r r r& r' rN )rT r r r9 p s
zGRUCell.__init__)rl rw r c C sB |d kr&t j|jd| j|j|jd}tj||| j| j | j
| j S )Nr )r' r& ) rE r~ rp r r' r& r Zgru_cellr r r r )r6 rl rw r r r r u s zGRUCell.forward)TNN)N) r r r r rr r> r9 r r r r r r )rT r r 2 s < r )r )"rf r@ r< typingr r r r r r rE r moduler
parameterr Z utils.rnnr
r, r r Zrnn_tanhZrnn_relur rr r r r r( r* r r r r r r r r s2
Gy m +XN