/usr/local/lib64/python3.6/site-packages/torch/nn/modules/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/nn/modules/__pycache__/transformer.cpython-36.pyc (20737B)
3
Eg[ @ s d dl Z d dlmZmZmZmZ d dlZd dlmZ ddlm Z
ddl m Z ddl
mZ dd lmZ dd
lmZ dd lmZ dd lmZ dd
lmZ G dd de ZG dd de ZG dd de ZG dd de Z G dd de Z dd Z dd Z dS ) N)OptionalAnyUnionCallable)Tensor )
functional )Module)MultiheadAttention)
ModuleList)xavier_uniform_)Dropout)Linear) LayerNormc s e Zd ZdZddddddejdddd d ddfeeeeeeee e
e ge f f e e
e e
eeedd
fd d
Zde e e e e e e e e e e e e e e d
ddZeee dddZdd Z ZS ) Transformera< A transformer model. User is able to modify the attributes as needed. The architecture
is based on the paper "Attention Is All You Need". Ashish Vaswani, Noam Shazeer,
Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and
Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information
Processing Systems, pages 6000-6010. Users can build the BERT(https://arxiv.org/abs/1810.04805)
model with corresponding parameters.
Args:
d_model: the number of expected features in the encoder/decoder inputs (default=512).
nhead: the number of heads in the multiheadattention models (default=8).
num_encoder_layers: the number of sub-encoder-layers in the encoder (default=6).
num_decoder_layers: the number of sub-decoder-layers in the decoder (default=6).
dim_feedforward: the dimension of the feedforward network model (default=2048).
dropout: the dropout value (default=0.1).
activation: the activation function of encoder/decoder intermediate layer, can be a string
("relu" or "gelu") or a unary callable. Default: relu
custom_encoder: custom encoder (default=None).
custom_decoder: custom decoder (default=None).
layer_norm_eps: the eps value in layer normalization components (default=1e-5).
batch_first: If ``True``, then the input and output tensors are provided
as (batch, seq, feature). Default: ``False`` (seq, batch, feature).
norm_first: if ``True``, encoder and decoder layers will perform LayerNorms before
other attention and feedforward operations, otherwise after. Default: ``False`` (after).
Examples::
>>> transformer_model = nn.Transformer(nhead=16, num_encoder_layers=12)
>>> src = torch.rand((10, 32, 512))
>>> tgt = torch.rand((20, 32, 512))
>>> out = transformer_model(src, tgt)
Note: A full example to apply nn.Transformer module for the word language model is available in
https://github.com/pytorch/examples/tree/master/word_language_model
i i g?Ngh㈵>F)
d_modelnheadnum_encoder_layersnum_decoder_layersdim_feedforwarddropout
activationcustom_encodercustom_decoderlayer_norm_eps batch_first
norm_firstreturnc s |
|d}t t| j |d k r(|| _n
>> output = transformer_model(src, tgt, src_mask=src_mask, tgt_mask=tgt_mask)
r z-the batch number of src and tgt must be equalr r z:the feature number of src and tgt must be equal to d_model)maskr9 )r7 r8 r: r; )r size RuntimeErrorr r& r) ) r- r4 r5 r6 r7 r8 r9 r: r; memoryoutputr2 r2 r3 forwardU s 0
zTransformer.forward)szr c C s t jt j| | ftdddS )zGenerate a square mask for the sequence. The masked positions are filled with float('-inf').
Unmasked positions are filled with float(0.0).
z-infr )Zdiagonal)torchZtriufullfloat)rB r2 r2 r3 generate_square_subsequent_mask s z+Transformer.generate_square_subsequent_maskc C s* x$| j D ]}|j dkr
t| q
W dS )z-Initiate parameters in the transformer model.r N)
parametersZdimr
)r- pr2 r2 r3 r, s z Transformer._reset_parameters)NNNNNN)__name__
__module__ __qualname____doc__FreluintrE r strr r r r boolr% rA staticmethodrF r,
__classcell__r2 r2 )r1 r3 r s !J 8<r c sF e Zd ZdZdgZd fdd Zd
eee ee edddZ Z S ) r( a* TransformerEncoder is a stack of N encoder layers
Args:
encoder_layer: an instance of the TransformerEncoderLayer() class (required).
num_layers: the number of sub-encoder-layers in the encoder (required).
norm: the layer normalization component (optional).
Examples::
>>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)
>>> transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=6)
>>> src = torch.rand(10, 32, 512)
>>> out = transformer_encoder(src)
normNc s* t t| j t||| _|| _|| _d S )N)r$ r( r% _get_cloneslayers
num_layersrT )r- r/ rW rT )r1 r2 r3 r% s zTransformerEncoder.__init__)r4 r< r9 r c C s: |}x | j D ]}||||d}q W | jdk r6| j|}|S )aP Pass the input through the encoder layers in turn.
Args:
src: the sequence to the encoder (required).
mask: the mask for the src sequence (optional).
src_key_padding_mask: the mask for the src keys per batch (optional).
Shape:
see the docs in Transformer class.
)r6 r9 N)rV rT )r- r4 r< r9 r@ modr2 r2 r3 rA s
zTransformerEncoder.forward)N)NN)
rI rJ rK rL
__constants__r% r r rA rS r2 r2 )r1 r3 r( s
r( c sT e Zd ZdZdgZd fdd Zd
eeee ee ee ee edddZ Z S ) r+ a_ TransformerDecoder is a stack of N decoder layers
Args:
decoder_layer: an instance of the TransformerDecoderLayer() class (required).
num_layers: the number of sub-decoder-layers in the decoder (required).
norm: the layer normalization component (optional).
Examples::
>>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8)
>>> transformer_decoder = nn.TransformerDecoder(decoder_layer, num_layers=6)
>>> memory = torch.rand(10, 32, 512)
>>> tgt = torch.rand(20, 32, 512)
>>> out = transformer_decoder(tgt, memory)
rT Nc s* t t| j t||| _|| _|| _d S )N)r$ r+ r% rU rV rW rT )r- r0 rW rT )r1 r2 r3 r% s zTransformerDecoder.__init__)r5 r? r7 r8 r: r; r c C s@ |}x"| j D ]}|||||||d}q W | jdk r<| j|}|S )aM Pass the inputs (and mask) through the decoder layer in turn.
Args:
tgt: the sequence to the decoder (required).
memory: the sequence from the last layer of the encoder (required).
tgt_mask: the mask for the tgt sequence (optional).
memory_mask: the mask for the memory sequence (optional).
tgt_key_padding_mask: the mask for the tgt keys per batch (optional).
memory_key_padding_mask: the mask for the memory keys per batch (optional).
Shape:
see the docs in Transformer class.
)r7 r8 r: r; N)rV rT ) r- r5 r? r7 r8 r: r; r@ rX r2 r2 r3 rA s
zTransformerDecoder.forward)N)NNNN)
rI rJ rK rL rY r% r r rA rS r2 r2 )r1 r3 r+ s r+ c s e Zd ZdZddgZddejdddddfdd fd
d
Z fd d
Zde e
e e
e e dddZ e e
e e
e e dddZ e e dddZ
ZS )r' a TransformerEncoderLayer is made up of self-attn and feedforward network.
This standard encoder layer is based on the paper "Attention Is All You Need".
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in
Neural Information Processing Systems, pages 6000-6010. Users may modify or implement
in a different way during application.
Args:
d_model: the number of expected features in the input (required).
nhead: the number of heads in the multiheadattention models (required).
dim_feedforward: the dimension of the feedforward network model (default=2048).
dropout: the dropout value (default=0.1).
activation: the activation function of the intermediate layer, can be a string
("relu" or "gelu") or a unary callable. Default: relu
layer_norm_eps: the eps value in layer normalization components (default=1e-5).
batch_first: If ``True``, then the input and output tensors are provided
as (batch, seq, feature). Default: ``False``.
norm_first: if ``True``, layer norm is done prior to attention and feedforward
operations, respectivaly. Otherwise it's done after. Default: ``False`` (after).
Examples::
>>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)
>>> src = torch.rand(10, 32, 512)
>>> out = encoder_layer(src)
Alternatively, when ``batch_first`` is ``True``:
>>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8, batch_first=True)
>>> src = torch.rand(32, 10, 512)
>>> out = encoder_layer(src)
r r i g?gh㈵>FN)r c s | |
d} t t| j t||f||d| | _t||f| | _t|| _t||f| | _ || _
t |fd|i| | _ t |fd|i| | _
t|| _t|| _t|trt|| _n|| _d S )N)r! r" )r r r# )r$ r' r% r self_attnr linear1r r linear2r r norm1norm2dropout1dropout2
isinstancerP _get_activation_fnr ) r- r r r r r r r r r! r" r. )r1 r2 r3 r% # s
z TransformerEncoderLayer.__init__c s& d|krt j|d< tt| j| d S )Nr )rM rN r$ r' __setstate__)r- state)r1 r2 r3 rc ; s
z$TransformerEncoderLayer.__setstate__)r4 r6 r9 r c C sh |}| j r8|| j| j||| }|| j| j| }n,| j|| j||| }| j|| j| }|S )aQ Pass the input through the encoder layer.
Args:
src: the sequence to the encoder layer (required).
src_mask: the mask for the src sequence (optional).
src_key_padding_mask: the mask for the src keys per batch (optional).
Shape:
see the docs in Transformer class.
)r _sa_blockr] _ff_blockr^ )r- r4 r6 r9 xr2 r2 r3 rA @ s z TransformerEncoderLayer.forward)rg attn_maskkey_padding_maskr c C s$ | j |||||ddd }| j|S )NF)rh ri need_weightsr )rZ r_ )r- rg rh ri r2 r2 r3 re Y s
z!TransformerEncoderLayer._sa_block)rg r c C s& | j | j| j| j|}| j|S )N)r\ r r r[ r` )r- rg r2 r2 r3 rf b s z!TransformerEncoderLayer._ff_block)NN)rI rJ rK rL rY rM rN r% rc r r rA re rf rS r2 r2 )r1 r3 r' s r' c s e Zd ZdZddgZddejdddddfdd fd
d
Z fd d
Zde e e
e e
e e
e e
e e dddZ e e
e e
e e dddZ e e e
e e
e e dddZ
e e dddZ ZS )r* aj TransformerDecoderLayer is made up of self-attn, multi-head-attn and feedforward network.
This standard decoder layer is based on the paper "Attention Is All You Need".
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in
Neural Information Processing Systems, pages 6000-6010. Users may modify or implement
in a different way during application.
Args:
d_model: the number of expected features in the input (required).
nhead: the number of heads in the multiheadattention models (required).
dim_feedforward: the dimension of the feedforward network model (default=2048).
dropout: the dropout value (default=0.1).
activation: the activation function of the intermediate layer, can be a string
("relu" or "gelu") or a unary callable. Default: relu
layer_norm_eps: the eps value in layer normalization components (default=1e-5).
batch_first: If ``True``, then the input and output tensors are provided
as (batch, seq, feature). Default: ``False``.
norm_first: if ``True``, layer norm is done prior to self attention, multihead
attention and feedforward operations, respectivaly. Otherwise it's done after.
Default: ``False`` (after).
Examples::
>>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8)
>>> memory = torch.rand(10, 32, 512)
>>> tgt = torch.rand(20, 32, 512)
>>> out = decoder_layer(tgt, memory)
Alternatively, when ``batch_first`` is ``True``:
>>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8, batch_first=True)
>>> memory = torch.rand(32, 10, 512)
>>> tgt = torch.rand(32, 20, 512)
>>> out = decoder_layer(tgt, memory)
r r i g?gh㈵>FN)r c s | |
d} t t| j t||f||d| | _t||f||d| | _t||f| | _t|| _ t||f| | _
|| _ t |fd|i| | _
t |fd|i| | _t |fd|i| | _t|| _t|| _t|| _t|trt|| _n|| _d S )N)r! r" )r r r# )r$ r* r% r rZ multihead_attnr r[ r r r\ r r r] r^ norm3r_ r` dropout3ra rP rb r ) r- r r r r r r r r r! r" r. )r1 r2 r3 r% s&
z TransformerDecoderLayer.__init__c s& d|krt j|d< tt| j| d S )Nr )rM rN r$ r* rc )r- rd )r1 r2 r3 rc s
z$TransformerDecoderLayer.__setstate__)r5 r? r7 r8 r: r; r c C s |}| j rR|| j| j||| }|| j| j|||| }|| j| j| }nF| j|| j||| }| j|| j|||| }| j|| j| }|S )aK Pass the inputs (and mask) through the decoder layer.
Args:
tgt: the sequence to the decoder layer (required).
memory: the sequence from the last layer of the encoder (required).
tgt_mask: the mask for the tgt sequence (optional).
memory_mask: the mask for the memory sequence (optional).
tgt_key_padding_mask: the mask for the tgt keys per batch (optional).
memory_key_padding_mask: the mask for the memory keys per batch (optional).
Shape:
see the docs in Transformer class.
)r re r]
_mha_blockr^ rf rl )r- r5 r? r7 r8 r: r; rg r2 r2 r3 rA s z TransformerDecoderLayer.forward)rg rh ri r c C s$ | j |||||ddd }| j|S )NF)rh ri rj r )rZ r_ )r- rg rh ri r2 r2 r3 re s
z!TransformerDecoderLayer._sa_block)rg memrh ri r c C s$ | j |||||ddd }| j|S )NF)rh ri rj r )rk r` )r- rg ro rh ri r2 r2 r3 rn s
z"TransformerDecoderLayer._mha_block)rg r c C s& | j | j| j| j|}| j|S )N)r\ r r r[ rm )r- rg r2 r2 r3 rf s z!TransformerDecoderLayer._ff_block)NNNN)rI rJ rK rL rY rM rN r% rc r r rA re rn rf rS r2 r2 )r1 r3 r* g s ! , r* c s t fddt|D S )Nc s g | ]}t j qS r2 )copydeepcopy).0i)moduler2 r3
s z _get_clones..)r range)rt Nr2 )rt r3 rU s rU c C s. | dkrt jS | dkr t jS tdj| d S )NrN geluz&activation should be relu/gelu, not {})rM rN rx r> format)r r2 r2 r3 rb s
rb ) rp typingr r r r rC r r rM rt r
r r containerr initr
r r Zlinearr Z
normalizationr r r( r+ r' r* rU rb r2 r2 r2 r3 s& ,4ez