/usr/local/lib/python3.6/site-packages/transformers/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/transformers/__pycache__/activations.cpython-36.pyc (7750B)
3
Eg @ sf d dl Z d dlZd dlmZ d dlmZmZ ddlmZ eje Z
G dd dej Z G dd d ej Z
G d
d d ej ZG d d
d
ej ZG dd dej ZG dd dej ZG dd dej ZG dd dej Zej e e e
ej e
dde e e ed$de e ej d
Zdd Zed Zed Zed Zed Z ed Z ed!Z ed"Z ed#Z dS )% N)version)Tensornn )loggingc @ s e Zd ZdZeedddZdS )NewGELUActivationz
Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see
the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415
)inputreturnc C s6 d| dt jtjdtj |dt j|d S )Ng ?g ?g @gHm?g @)torchtanhmathsqrtpipow)selfr r B/usr/local/lib/python3.6/site-packages/transformers/activations.pyforward! s zNewGELUActivation.forwardN)__name__
__module__ __qualname____doc__r r r r r r r s r c sH e Zd ZdZd ed fdd
ZeedddZeedd d
Z Z S ) GELUActivationa
Original Implementation of the GELU activation function in Google BERT repo when initially created. For
information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 +
torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional
Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415
F)use_gelu_pythonc s< t j tjtjtjdk s$|r.| j| _n
tj j
| _d S )Nz1.4) super__init__r parser
__version__ _gelu_pythonactr
functionalgelu)r r ) __class__r r r - s
zGELUActivation.__init__)r r c C s |d dt j|tjd S )Ng ?g ?g @)r
erfr r
)r r r r r r 4 s zGELUActivation._gelu_pythonc C s
| j |S )N)r )r r r r r r 7 s zGELUActivation.forward)F)
r r r r boolr r r r
__classcell__r r )r" r r % s r c @ s e Zd ZdZeedddZdS )FastGELUActivationz}
Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs
)r r c C s* d| dt j|d dd| | S )Ng ?g ?g3E?gHm?)r
r )r r r r r r @ s zFastGELUActivation.forwardN)r r r r r r r r r r r& ; s r& c @ s e Zd ZdZeedddZdS )QuickGELUActivationzr
Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs
)r r c C s |t jd| S )Ng Zd;?)r
sigmoid)r r r r r r I s zQuickGELUActivation.forwardN)r r r r r r r r r r r' D s r' c s8 e Zd ZdZeed fdd ZeedddZ ZS )ClippedGELUActivationa
Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as
it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to
https://arxiv.org/abs/2004.09602.
Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when
initially created.
For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 +
torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://arxiv.org/abs/1606.08415
)minmaxc s8 ||kr t d| d| dt j || _|| _d S )Nz min should be < max (got min: z, max: ))
ValueErrorr r r* r+ )r r* r+ )r" r r r Z s
z ClippedGELUActivation.__init__)xr c C s t jt|| j| jS )N)r
Zclipr! r* r+ )r r. r r r r b s z ClippedGELUActivation.forward) r r r r floatr r r r% r r )r" r r) M s r) c s@ e Zd ZdZ fddZeedddZeedddZ ZS ) SiLUActivationa
See Gaussian Error Linear Units (Hendrycks et al., https://arxiv.org/abs/1606.08415) where the SiLU (Sigmoid Linear
Unit) was originally introduced and coined, and see Sigmoid-Weighted Linear Units for Neural Network Function
Approximation in Reinforcement Learning (Elfwing et al., https://arxiv.org/abs/1702.03118) and Swish: a Self-Gated
Activation Function (Ramachandran et al., https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with
later.
c s8 t j tjtjtjdk r*| j| _n
tj j
| _d S )Nz1.7) r r r r r
r _silu_pythonr r r silu)r )r" r r r o s
zSiLUActivation.__init__)r r c C s |t j| S )N)r
r( )r r r r r r1 v s zSiLUActivation._silu_pythonc C s
| j |S )N)r )r r r r r r y s zSiLUActivation.forward) r r r r r r r1 r r% r r )r" r r0 f s r0 c s@ e Zd ZdZ fddZeedddZeedddZ ZS ) MishActivationz
See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://arxiv.org/abs/1908.08681). Also
visit the official repository for the paper: https://github.com/digantamisra98/Mish
c s8 t j tjtjtjdk r*| j| _n
tj j
| _d S )Nz1.9) r r r r r
r _mish_pythonr r r mish)r )r" r r r s
zMishActivation.__init__)r r c C s |t jtjj| S )N)r
r r r Zsoftplus)r r r r r r4 s zMishActivation._mish_pythonc C s
| j |S )N)r )r r r r r r s zMishActivation.forward) r r r r r r r4 r r% r r )r" r r3 } s r3 c @ s e Zd ZdZeedddZdS )LinearActivationz[
Applies the linear activation function, i.e. forwarding input directly to output.
)r r c C s |S )Nr )r r r r r r s zLinearActivation.forwardN)r r r r r r r r r r r6 s r6 T)r
)
Zrelur2 Zswishr! r gelu_pythongelu_new gelu_fast
quick_geluZgelu_10r5 linearr( c C s0 | t krt | S td| dtt j d S )Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)Zactivation_stringr r r get_activation s rA r8 r9 r! r: r; r2 r5 r< i)!r r
packagingr r r utilsr
get_loggerr loggerModuler r r& r' r) r0 r3 r6 ZReLUZTanhZSigmoidr= rA r8 r9 r! r: r; r2 r5 Z
linear_actr r r r
sF