/usr/local/lib/python3.6/site-packages/transformers/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/transformers/__pycache__/optimization.cpython-36.pyc (22653B)
3
Egll @ s d Z ddlZddlZddlmZmZmZmZmZ ddl Z ddl m
Z
ddl m Z ddl
mZ ddlmZ dd lmZ dd
lmZ ejeZd*e ed d d
Zd,e eedddZd.ddZd0e eee edddZ d2e eeeedddZ d4dd Z ej eej!e ej"e ej#e ej$eej%eiZ&d5ee'ef e ee ee d d d Z(G d d! d!e Z)G d"d# d#e Z*G d$d% d%eZ+d6d'd(Z,dS )7z$PyTorch optimization for BERT model. N)CallableIterableOptionalTupleUnion)nn) Optimizer)LambdaLR )
SchedulerType)logging)require_version) optimizer
last_epochc C s t | dd |dS )a
Create a schedule with a constant learning rate, using the learning rate set in optimizer.
Args:
optimizer ([`~torch.optim.Optimizer`]):
The optimizer for which to schedule the learning rate.
last_epoch (`int`, *optional*, defaults to -1):
The index of the last epoch when resuming training.
Return:
`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.
c S s dS )Nr
)_r r C/usr/local/lib/python3.6/site-packages/transformers/optimization.py
/ s z'get_constant_schedule..)r )r )r r r r r get_constant_schedule" s
r )r num_warmup_stepsr c s t d fdd }t| ||dS )ad
Create a schedule with a constant learning rate preceded by a warmup period during which the learning rate
increases linearly between 0 and the initial lr set in the optimizer.
Args:
optimizer ([`~torch.optim.Optimizer`]):
The optimizer for which to schedule the learning rate.
num_warmup_steps (`int`):
The number of steps for the warmup phase.
last_epoch (`int`, *optional*, defaults to -1):
The index of the last epoch when resuming training.
Return:
`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.
) current_stepc s" | k r t | t td S dS )Ng ?)floatmax)r )r r r lr_lambdaC s z4get_constant_schedule_with_warmup..lr_lambda)r )intr )r r r r r )r r !get_constant_schedule_with_warmup2 s r c s t d fdd }t| ||S )a
Create a schedule with a learning rate that decreases linearly from the initial lr set in the optimizer to 0, after
a warmup period during which it increases linearly from 0 to the initial lr set in the optimizer.
Args:
optimizer ([`~torch.optim.Optimizer`]):
The optimizer for which to schedule the learning rate.
num_warmup_steps (`int`):
The number of steps for the warmup phase.
num_training_steps (`int`):
The total number of training steps.
last_epoch (`int`, *optional*, defaults to -1):
The index of the last epoch when resuming training.
Return:
`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.
)r c sB | k r t | t td S tdt | t td S )Nr
g )r r )r )num_training_stepsr r r r ^ s z2get_linear_schedule_with_warmup..lr_lambda)r r )r r r r r r )r r r get_linear_schedule_with_warmupK s r ?)r r r
num_cyclesr c s fdd}t | ||S )a
Create a schedule with a learning rate that decreases following the values of the cosine function between the
initial lr set in the optimizer to 0, after a warmup period during which it increases linearly between 0 and the
initial lr set in the optimizer.
Args:
optimizer ([`~torch.optim.Optimizer`]):
The optimizer for which to schedule the learning rate.
num_warmup_steps (`int`):
The number of steps for the warmup phase.
num_training_steps (`int`):
The total number of training steps.
num_cycles (`float`, *optional*, defaults to 0.5):
The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0
following a half-cosine).
last_epoch (`int`, *optional*, defaults to -1):
The index of the last epoch when resuming training.
Return:
`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.
c sf | k r t | t td S t | t td }tdddtjtjt d | S )Nr
g g ?g ?g @)r r mathcospi)r progress)r r r r r r s z2get_cosine_schedule_with_warmup..lr_lambda)r )r r r r r r r )r r r r get_cosine_schedule_with_warmuph s r$ c s fdd}t | ||S )a
Create a schedule with a learning rate that decreases following the values of the cosine function between the
initial lr set in the optimizer to 0, with several hard restarts, after a warmup period during which it increases
linearly between 0 and the initial lr set in the optimizer.
Args:
optimizer ([`~torch.optim.Optimizer`]):
The optimizer for which to schedule the learning rate.
num_warmup_steps (`int`):
The number of steps for the warmup phase.
num_training_steps (`int`):
The total number of training steps.
num_cycles (`int`, *optional*, defaults to 1):
The number of hard restarts to use.
last_epoch (`int`, *optional*, defaults to -1):
The index of the last epoch when resuming training.
Return:
`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.
c sr | k r t | t td S t | t td }|dkrHdS tdddtjtjt | d S )Nr
g ?g g ?)r r r r! r" )r r# )r r r r r r s zEget_cosine_with_hard_restarts_schedule_with_warmup..lr_lambda)r )r r r r r r r )r r r r 2get_cosine_with_hard_restarts_schedule_with_warmup s r% Hz> ?c sN | j d ks(td d dtd fdd }t| ||S )a
Create a schedule with a learning rate that decreases as a polynomial decay from the initial lr set in the
optimizer to end lr defined by *lr_end*, after a warmup period during which it increases linearly from 0 to the
initial lr set in the optimizer.
Args:
optimizer ([`~torch.optim.Optimizer`]):
The optimizer for which to schedule the learning rate.
num_warmup_steps (`int`):
The number of steps for the warmup phase.
num_training_steps (`int`):
The total number of training steps.
lr_end (`float`, *optional*, defaults to 1e-7):
The end LR.
power (`float`, *optional*, defaults to 1.0):
Power factor.
last_epoch (`int`, *optional*, defaults to -1):
The index of the last epoch when resuming training.
Note: *power* defaults to 1.0 as in the fairseq implementation, which in turn is based on the original BERT
implementation at
https://github.com/google-research/bert/blob/f39e881b169b9d53bea03d2d341b31707a6c052b/optimization.py#L37
Return:
`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.
lrzlr_end (z&) must be be smaller than initial lr ())r c sj | k r t | t td S | kr. S } }d| | }|| }| S d S )Nr
)r r )r Zlr_rangeZ decay_stepsZ
pct_remainingZdecay)lr_endlr_initr r powerr r r s z.lr_lambda)defaults
ValueErrorr r )r r r r* r, r r r )r* r+ r r r, r )get_polynomial_decay_schedule_with_warmup s
r/ )namer r r c C sr t | } t| }| t jkr"||S |dkr8t| d| t jkrN|||dS |dkrdt| d||||dS )a
Unified API to get any scheduler from its name.
Args:
name (`str` or `SchedulerType`):
The name of the scheduler to use.
optimizer (`torch.optim.Optimizer`):
The optimizer that will be used during training.
num_warmup_steps (`int`, *optional*):
The number of warmup steps to do. This is not required by all schedulers (hence the argument being
optional), the function will raise an error if it's unset and the scheduler type requires it.
num_training_steps (`int``, *optional*):
The number of training steps to do. This is not required by all schedulers (hence the argument being
optional), the function will raise an error if it's unset and the scheduler type requires it.
Nz; requires `num_warmup_steps`, please provide that argument.)r z= requires `num_training_steps`, please provide that argument.)r r )r TYPE_TO_SCHEDULER_FUNCTIONCONSTANTr. CONSTANT_WITH_WARMUP)r0 r r r Z
schedule_funcr r r
get_scheduler s
r4 c sT e Zd ZdZdeejj ee eef eee
e
d fd
d
Z de d
ddZ
ZS )AdamWa5
Implements Adam algorithm with weight decay fix as introduced in [Decoupled Weight Decay
Regularization](https://arxiv.org/abs/1711.05101).
Parameters:
params (`Iterable[nn.parameter.Parameter]`):
Iterable of parameters to optimize or dictionaries defining parameter groups.
lr (`float`, *optional*, defaults to 1e-3):
The learning rate to use.
betas (`Tuple[float,float]`, *optional*, defaults to (0.9, 0.999)):
Adam's betas parameters (b1, b2).
eps (`float`, *optional*, defaults to 1e-6):
Adam's epsilon for numerical stability.
weight_decay (`float`, *optional*, defaults to 0):
Decoupled weight decay to apply.
correct_bias (`bool`, *optional*, defaults to `True`):
Whether or not to correct bias in Adam (for instance, in Bert TF repository they use `False`).
no_deprecation_warning (`bool`, *optional*, defaults to `False`):
A flag used to disable the deprecation warning (set to `True` to disable the warning).
MbP??+?ư> TF)paramsr( betaseps weight_decay correct_biasno_deprecation_warningc s |st jdt td |dk r0td| dd|d koFdk n s`td|d d d|d
kovdk n std|d
d d|kstd | dt|||||d }t j|| d S )
NzThis implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warningz torch>=1.5.0g zInvalid learning rate: z - should be >= 0.0r g ?zInvalid beta parameter: z - should be in [0.0, 1.0)r
zInvalid epsilon value: )r( r< r= r> r? )warningswarn
FutureWarningr
r. dictsuper__init__) selfr; r( r< r= r> r? r@ r- ) __class__r r rF ' s
zAdamW.__init__N)closurec C s d}|dk r| }x|| j D ]p}xh|d D ]Z}|jdkr@q.|jj}|jrVtd| j| }t|dkrd|d< tj|j|d< tj|j|d< |d |d }}|d \} }
|d d 7 < |j | j
|d
| d |j |
j ||d
|
d |j j
|d
} |d } |d rJd
| |d }
d
|
|d }| t
j | |
} |jj|| | d |d dkr.|jj
|j|d |d d q.W q W |S )z
Performs a single optimization step.
Arguments:
closure (`Callable`, *optional*): A closure that reevaluates the model and returns the loss.
Nr; zJAdam does not support sparse gradients, please consider SparseAdam insteadr stepexp_avg
exp_avg_sqr< r
g ?)alpha)valuer= r( r? r> g ) param_groupsgraddata is_sparse RuntimeErrorstatelentorch
zeros_likemul_add_Zaddcmul_sqrtr Zaddcdiv_)rG rI lossgroupprP rT rK rL beta1Zbeta2ZdenomZ step_sizeZbias_correction1Zbias_correction2r r r rJ C s<
(z
AdamW.stepr7 r8 )r6 r_ r9 r: TF)N)__name__
__module__ __qualname____doc__r r parameter Parameterr r boolrF r rJ
__classcell__r r )rH r r5 s 0r5 c s\ e Zd ZdZd fd
d Zed d
Zedd Zedd Zedd Z dddZ
Z S ) Adafactora!
AdaFactor pytorch implementation can be used as a drop in replacement for Adam original fairseq code:
https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py
Paper: *Adafactor: Adaptive Learning Rates with Sublinear Memory Cost* https://arxiv.org/abs/1804.04235 Note that
this optimizer internally adjusts the learning rate depending on the `scale_parameter`, `relative_step` and
`warmup_init` options. To use a manual (external) learning rate schedule you should set `scale_parameter=False` and
`relative_step=False`.
Arguments:
params (`Iterable[nn.parameter.Parameter]`):
Iterable of parameters to optimize or dictionaries defining parameter groups.
lr (`float`, *optional*):
The external learning rate.
eps (`Tuple[float, float]`, *optional*, defaults to (1e-30, 1e-3)):
Regularization constants for square gradient and parameter scale respectively
clip_threshold (`float`, *optional*, defaults 1.0):
Threshold of root mean square of final gradient update
decay_rate (`float`, *optional*, defaults to -0.8):
Coefficient used to compute running averages of square
beta1 (`float`, *optional*):
Coefficient used for computing running averages of gradient
weight_decay (`float`, *optional*, defaults to 0):
Weight decay (L2 penalty)
scale_parameter (`bool`, *optional*, defaults to `True`):
If True, learning rate is scaled by root mean square
relative_step (`bool`, *optional*, defaults to `True`):
If True, time-dependent learning rate is computed instead of external learning rate
warmup_init (`bool`, *optional*, defaults to `False`):
Time-dependent learning rate computation depends on whether warm-up initialization is being used
This implementation handles low-precision (FP16, bfloat) values, but we have not thoroughly tested.
Recommended T5 finetuning settings (https://discuss.huggingface.co/t/t5-finetuning-tips/684/3):
- Training without LR warmup or clip_threshold is not recommended.
- use scheduled LR warm-up to fixed LR
- use clip_threshold=1.0 (https://arxiv.org/abs/1804.04235)
- Disable relative updates
- Use scale_parameter=False
- Additional optimizer operations like gradient clipping should not be used alongside Adafactor
Example:
```python
Adafactor(model.parameters(), scale_parameter=False, relative_step=False, warmup_init=False, lr=1e-3)
```
Others reported the following combination to work well:
```python
Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None)
```
When using `lr=None` with [`Trainer`] you will most likely need to use [`~optimization.AdafactorSchedule`]
scheduler as following:
```python
from transformers.optimization import Adafactor, AdafactorSchedule
optimizer = Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None)
lr_scheduler = AdafactorSchedule(optimizer)
trainer = Trainer(..., optimizers=(optimizer, lr_scheduler))
```
Usage:
```python
# replace AdamW with Adafactor
optimizer = Adafactor(
model.parameters(),
lr=1e-3,
eps=(1e-30, 1e-3),
clip_threshold=1.0,
decay_rate=-0.8,
beta1=None,
weight_decay=0.0,
relative_step=False,
scale_parameter=False,
warmup_init=False,
)
```NKH9MbP? ?g? TFc sZ t d |d k r | r td|
r.| r.tdt|||||||| |
d } t j|| d S )Nz torch>=1.5.0z;Cannot combine manual `lr` and `relative_step=True` optionsz0`warmup_init=True` requires `relative_step=True`) r( r= clip_threshold
decay_rater^ r> scale_parameter
relative_step warmup_init)r
r. rD rE rF ) rG r; r( r= rm rn r^ r> ro rp rq r- )rH r r rF s
zAdafactor.__init__c C sj | d }| d r@| d r$d|d nd}t |dtj|d }d}| d rbt| d d
|d }|| S ) Nr( rp rq gư>rJ g{Gz?g ?ro r= r
RMS)minr rZ r ) param_groupZ param_stateZ rel_step_szZmin_stepZ param_scaler r r _get_lr s zAdafactor._get_lrc C s t |dk}| d d k }||fS )N r^ )rU )rt Z param_shapefactoreduse_first_momentr r r _get_options s zAdafactor._get_optionsc C s | j d| j d S )Nrv g ?)ZnormZnumel)Ztensorr r r _rms s zAdafactor._rmsc C s6 | | j ddd j jd}|jdj }tj||S )Nr
T)dimZkeepdimrv r| )meanZrsqrt_Z unsqueezersqrtrV mul)exp_avg_sq_rowexp_avg_sq_colZr_factorZc_factorr r r _approx_sq_grad s zAdafactor._approx_sq_gradc C sB d}|dk r| }x(| j D ] }x|d D ]}|jdkr@q.|jj}|jtjtjhkrb|j }|jrpt d| j
| }|j }| j ||\}} t
|dkr d|d< | rtj||d< |rtj|dd j||d< tj|dd |dd j||d
< ntj||d < d|d < nV| r4|d j||d< |r`|d j||d< |d
j||d
< n|d j||d < |j}
|jjtjtjhkr|
j }
|d d7 < | j|
|d < | j||} d
tj|d |d } |d |d d }
|rZ|d }|d
}|j| j|
jddd
| d |j| j|
jddd
| d | j||}
|
j| n.|d }|j| j|
d
| d |j j|}
|
j| j|
|d jd
d |
j| | r|d }|j|d j|
d|d d |}
|d dkr
|
j|
|d | d |
j|
|jjtjtjhkr.|jj |
q.W q W |S )z
Performs a single optimization step
Arguments:
closure (callable, optional): A closure that reevaluates the model
and returns the loss.
Nr; z,Adafactor does not support sparse gradients.r rJ rK r
r rv r rL rr g ?rn r= )r{ )rM rm )rs r^ r> r| r} r| r| r} ) rO rP rQ ZdtyperV Zfloat16Zbfloat16r rR rS rT shapery rU rW ZzerosZtorz ru r powrX rY r~ r r Zdiv_Zclamp_Zcopy_)rG rI r[ r\ r] rP rT Z
grad_shaperw rx Z p_data_fp32r( Zbeta2tupdater r rL rK r r r rJ sx
*
zAdafactor.stepri rj 皙) Nr rk r Nrl TTF)N) r` ra rb rc rF staticmethodru ry rz r rJ rg r r )rH r rh s S rh c s* e Zd ZdZd fdd Zdd Z ZS )AdafactorSchedulea8
Since [`~optimization.Adafactor`] performs its own scheduling, if the training loop relies on a scheduler (e.g.,
for logging), this class creates a proxy object that retrieves the current lr values from the optimizer.
It returns `initial_lr` during startup and the actual `lr` during stepping.
c sL fdd}x|j D ] } |d< qW t j|| x|j D ]
}|d= q:W d S )Nc s S )Nr )r )
initial_lrr r r z s z-AdafactorSchedule.__init__..lr_lambdar )rO rE rF )rG r r r r\ )rH )r r rF y s zAdafactorSchedule.__init__c s0 | j fdd jD }t|dkr,| j}|S )Nc s8 g | ]0}|d d j dk r j| j|d d qS )r; r N)rP ru rT ).0r\ )optr r
s z,AdafactorSchedule.get_lr..r )r rO rU Zbase_lrs)rG Zlrsr )r r get_lr s
zAdafactorSchedule.get_lr)r )r` ra rb rc rF r rg r r )rH r r q s
r c C s
t | |S )aX
Get a proxy schedule for [`~optimization.Adafactor`]
Args:
optimizer ([`~torch.optim.Optimizer`]):
The optimizer for which to schedule the learning rate.
initial_lr (`float`, *optional*, defaults to 0.0):
Initial lr
Return:
[`~optimization.Adafactor`] proxy schedule object.
)r )r r r r r get_adafactor_schedule s r r| )r| r| )r| r| )r| r| )r r| r| )r
r| r| )r& r' r| )NN)r )-rc r rA typingr r r r r rV r Z torch.optimr Ztorch.optim.lr_schedulerr Z
trainer_utilsr utilsr Zutils.versionsr
get_loggerr` loggerr r r r r r$ r% r/ ZLINEARZCOSINEZCOSINE_WITH_RESTARTSZ
POLYNOMIALr2 r3 r1 strr4 r5 rh r r r r r r s<
"#
2
&$p q