/usr/local/lib/python3.6/site-packages/transformers/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/transformers/__pycache__/trainer.cpython-36.pyc (81840B)
3
Eg. @ sB d Z ddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddl m Z ddl
mZ ddlmZmZmZmZmZmZmZmZ ddlmZ ddlmZm Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z& ddl'Z(ddl)Z)ddl*j+Z,dd l-m.Z. dd
l)m/Z/ dd l0m1Z1m2Z2m3Z3m4Z4 dd l5m6Z6 dd
l7m8Z8 ddl9m:Z: ddl;m
Z>m?Z?m@Z@ ddlAmBZBmCZC ddlDmEZEmFZFmGZG ddlHmIZI ddlJmKZK ddlLmMZMmNZN ddlOmPZPmQZQ ddlRmSZS ddlTmUZUmVZVmWZWmXZXmYZYmZZZm[Z[ ddl\m]Z]m^Z^m_Z_m`Z`maZambZbmcZcmdZdmeZemfZfmgZgmhZhmiZimjZjmkZkmlZlmmZmmnZn ddlompZpmqZqmrZrmsZsmtZtmuZumvZvmwZwmxZxmyZymzZzm{Z{m|Z|m}Z}m~Z~mZmZmZmZ ddlmZmZmZ dd lmZmZmZmZmZmZmZmZmZmZmZ d Zd ZeVgZeXZe rdd lmZ eZe r dd lmZ e.je)j:e.jd krHd!Zd!Zdd"lmZ e rXddlZe rddljjZddljjZddlj+jZe reId# ddlZdd$lmZ dd%lmZ dd&lmZ dd'lmZ dd(lmZ e rddljj)Zdd)l\mZmZmZmZ erddlZejeZd*Zd+Zd,Zd-Zd.ZG d/d0 d0ZdS )1uc
The Trainer class, to easily train a 🤗 Transformers from scratch or finetune it on a new task.
N)Mapping)Path)
TYPE_CHECKINGAnyCallableDictListOptionalTupleUnion)tqdm ) default_hp_search_backend#get_reporting_integration_callbacks hp_paramsis_fairscale_availableis_optuna_availableis_ray_tune_availableis_sigopt_availableis_wandb_availablerun_hp_search_optunarun_hp_search_rayrun_hp_search_sigoptrun_hp_search_wandb)version)nn)
DataLoaderDataset
RandomSamplerSequentialSampler)DistributedSampler)
Repository) __version__)PretrainedConfig) DataCollatorDataCollatorWithPaddingdefault_data_collator) DebugOptionDebugUnderflowOverflow)deepspeed_initdeepspeed_reinitis_deepspeed_zero3_enabled)dep_version_check)TrainingSummary)PreTrainedModel unwrap_model) Adafactor
get_scheduler)PreTrainedTokenizerBase)CallbackHandlerDefaultFlowCallbackPrinterCallbackProgressCallbackTrainerCallbackTrainerControl TrainerState) DistributedLengthGroupedSamplerDistributedSamplerWithLoopDistributedTensorGathererIterableDatasetShard
LabelSmootherLengthGroupedSampler SequentialDistributedSampler ShardSampler distributed_broadcast_scalarsdistributed_concatfind_batch_sizeget_parameter_names
nested_concat
nested_detachnested_numpifynested_truncatenested_xla_mesh_reducereissue_pt_warnings)PREFIX_CHECKPOINT_DIRBestRunEvalLoopOutputEvalPredictionHPSearchBackend HubStrategyIntervalStrategyPredictionOutputShardedDDPOptionTrainerMemoryTracker TrainOutputdefault_compute_objectivedefault_hp_spacedenumpify_detensorizeget_last_checkpoint
has_lengthnumber_of_argumentsset_seed
speed_metrics)OptimizerNames ParallelModeTrainingArguments) CONFIG_NAME WEIGHTS_NAME find_labelsget_full_repo_nameis_apex_availableis_datasets_availableis_in_notebookis_sagemaker_dp_enabledis_sagemaker_mp_enabledis_torch_tpu_availableloggingF)NotebookProgressCallback)ampz1.6T)autocast fairscale)FullyShardedDataParallel)ShardedDataParallel) auto_wrap)OSS)ShardedGradScaler)smp_forward_backwardsmp_forward_only
smp_gathersmp_nested_concatztraining_args.binztrainer_state.jsonz optimizer.ptz scheduler.ptz scaler.ptc @ s e Zd ZdZddlmZmZmZmZm Z de
e e j
f eee ee ee ee eg e f eeegef eee eejjejj j f eej ej gej f d ddZ dd Z d
d Z!d d
Z"dd Z#ddee$ dddZ%eej&j'j( dddZ)e*dddZ+eeej&j'j( dddZ,dee e*dd d Z-ee*d d d Z.e/d!d"d#Z0d$d% Z1e2eee3e3f d&d'd(Z4de/ejjd)d*d+Z5e*e/d,d-d.Z6e
d/ee$e3f f d0d1d2Z7e
d/ee$e3f f e/ee$e8f d3d4d5Z9d6d7 Z:dd8d9Z;dd;d<Zd?Z>d@dA Z?dBdC Z@dDdE ZAddFdGZBdHdI ZCdeed/gee$e8f f eeee$e8f ge8f e/e$ee
dLeDf eed/ge$f eEdMdNdOZFee$e8f ddPdQdRZGe
ej e3f e
ej e3f dSdTdUZHee$e
ej e3f f ee$e
ej e3f f dVdWdXZIdYdZ ZJe j
ee$e
ej e3f f ej d[d\d]ZKdd_d`ZLe=ddadbZMe=ddcddZNdee$ e=dedfdgZOdee$ dhdidjZPdee$ dhdkdlZQdmdn ZRdeSd^fee$ ddodpZTddddqdrZUdee eee$ e$ee$e8f dtdudvZVdeeee$ e$eWdxdydzZXde*e$ee= eee$ e$eYd{d|d}ZZdd~dZ[dddZ\de j
ee$e
ej e3f f e=eee$ eeej eej eej f dddZ]ee$e
ej e3f f dddZ^de=dddZ_dee$ ee$ ee$ ee$ ee$ ee$ ee
e$ee$ f ee
e$ee$ f ee
e$ee$ f d ddZ`dd Zadee$ e=e$dddZbde*e$ee= eee$ e$eWd{ddZcdd ZddS )Traineru
Trainer is a simple but feature-complete training and eval loop for PyTorch, optimized for 🤗 Transformers.
Args:
model ([`PreTrainedModel`] or `torch.nn.Module`, *optional*):
The model to train, evaluate or use for predictions. If not provided, a `model_init` must be passed.
[`Trainer`] is optimized to work with the [`PreTrainedModel`] provided by the library. You can still use
your own models defined as `torch.nn.Module` as long as they work the same way as the 🤗 Transformers
models.
args ([`TrainingArguments`], *optional*):
The arguments to tweak for training. Will default to a basic instance of [`TrainingArguments`] with the
`output_dir` set to a directory named *tmp_trainer* in the current directory if not provided.
data_collator (`DataCollator`, *optional*):
The function to use to form a batch from a list of elements of `train_dataset` or `eval_dataset`. Will
default to [`default_data_collator`] if no `tokenizer` is provided, an instance of
[`DataCollatorWithPadding`] otherwise.
train_dataset (`torch.utils.data.Dataset` or `torch.utils.data.IterableDataset`, *optional*):
The dataset to use for training. If it is an `datasets.Dataset`, columns not accepted by the
`model.forward()` method are automatically removed.
Note that if it's a `torch.utils.data.IterableDataset` with some randomization and you are training in a
distributed fashion, your iterable dataset should either use a internal attribute `generator` that is a
`torch.Generator` for the randomization that must be identical on all processes (and the Trainer will
manually set the seed of this `generator` at each epoch) or have a `set_epoch()` method that internally
sets the seed of the RNGs used.
eval_dataset (`torch.utils.data.Dataset`, *optional*):
The dataset to use for evaluation. If it is an `datasets.Dataset`, columns not accepted by the
`model.forward()` method are automatically removed.
tokenizer ([`PreTrainedTokenizerBase`], *optional*):
The tokenizer used to preprocess the data. If provided, will be used to automatically pad the inputs the
maximum length when batching inputs, and it will be saved along the model to make it easier to rerun an
interrupted training or reuse the fine-tuned model.
model_init (`Callable[[], PreTrainedModel]`, *optional*):
A function that instantiates the model to be used. If provided, each call to [`~Trainer.train`] will start
from a new instance of the model as given by this function.
The function may have zero argument, or a single one containing the optuna/Ray Tune/SigOpt trial object, to
be able to choose different architectures according to hyper parameters (such as layer count, sizes of
inner layers, dropout probabilities etc).
compute_metrics (`Callable[[EvalPrediction], Dict]`, *optional*):
The function that will be used to compute metrics at evaluation. Must take a [`EvalPrediction`] and return
a dictionary string to metric values.
callbacks (List of [`TrainerCallback`], *optional*):
A list of callbacks to customize the training loop. Will add those to the list of default callbacks
detailed in [here](callback).
If you want to remove one of the default callbacks used, use the [`Trainer.remove_callback`] method.
optimizers (`Tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`, *optional*): A tuple
containing the optimizer and the scheduler to use. Will default to an instance of [`AdamW`] on your model
and a scheduler given by [`get_linear_schedule_with_warmup`] controlled by `args`.
preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`, *optional*):
A function that preprocess the logits right before caching them at each evaluation step. Must take two
tensors, the logits and the labels, and return the logits once processed as desired. The modifications made
by this function will be reflected in the predictions received by `compute_metrics`.
Note that the labels (second parameter) will be `None` if the dataset does not have them.
Important attributes:
- **model** -- Always points to the core model. If using a transformers model, it will be a [`PreTrainedModel`]
subclass.
- **model_wrapped** -- Always points to the most external model in case one or more other modules wrap the
original model. This is the model that should be used for the forward pass. For example, under `DeepSpeed`,
the inner model is wrapped in `DeepSpeed` and then again in `torch.nn.DistributedDataParallel`. If the inner
model hasn't been wrapped, then `self.model_wrapped` is the same as `self.model`.
- **is_model_parallel** -- Whether or not a model has been switched to a model parallel mode (different from
data parallelism, this means some of the model layers are split on different GPUs).
- **place_model_on_device** -- Whether or not to automatically place the model on the device - it will be set
to `False` if model parallel or deepspeed is used, or if the default
`TrainingArguments.place_model_on_device` is overridden to return `False` .
- **is_in_train** -- Whether or not a model is currently running `train` (e.g. when `evaluate` is called while
in `train`)
r
)_get_learning_rate log_metricsmetrics_format save_metrics
save_stateN) modelargs
data_collator
train_dataset eval_dataset tokenizer
model_initcompute_metrics callbacks
optimizers preprocess_logits_for_metricsc C s |d kr(d} t jd| d t| d}|| _t| jj d | _d | _d| _t | jj
| _ | j j |j
}
tj|
|j |d kr|d k r|| _| j }qtdn|d k rtjdt || _t|dr|jr|jrd | _nd| _d | _t |jd
kr|jr
t d |j d%kr t d
nt s2t dnrt!j"|jkr^t#d kr^t dt$j% dnFt!j"|jkrvt!j"| _n.t!j&|jkrt!j&| _nt!j'|jkrt!j'| _|j(| _(| js|js|j)s|j*r|j+ s| jt!j&t!j'gkrd| _(|d krt,nt-|}|d k r|n|| _.|| _/|| _0|| _1| j(rB| j2||j3 | jrRd | j_4|| _5|| _6|| _7| | _8|
\| _9| _:|d k r| j9d k s| j:d k rtdt;t<| jj= }| d kr|n|| } t>| | j6| j1| j9| j:| _?| j@| jjArtBntC d| _D| jjEr<| jFd d tG r(tHjId n|j d&kr= 0.3, found zD. Upgrade your fairscale library: `pip install --upgrade fairscale`.zPassing a `model_init` is incompatible with providing the `optimizers` argument. You should subclass `Trainer` and override the `create_optimizer_and_scheduler` method.)at_initz
init git repo)exist_okZ
collate_batchzRThe `data_collator` should be a simple callable (function, class with `__call__`).zHmax_steps is given, it will override any value given in num_train_epochszGtrain_dataset does not implement __len__, max_steps has to be specifiedzTthe `--group_by_length` option is only available for `Dataset`, not `IterableDatasetautorn z3Tried to use `bf16` but native amp is not availableapexzUsing z half precision backend)
GradScalerzcUsing FP16 with APEX but APEX is not installed, please refer to https://www.github.com/nvidia/apex.zSageMaker Model Parallelism in mixed precision mode does not support gradient clipping yet. Pass along 'max_grad_norm': 0 in your hyperparameters.)epsilon)is_local_process_zerois_world_process_zeror )loggerinfora r r] seedhp_name deepspeed is_in_trainrU Zskip_memory_metrics_memory_trackerstartZget_process_log_levelrl
set_verbosityZ_setup_devicesr call_model_init RuntimeErrorwarningswarn
FutureWarninghasattrr Zmodel_parallelZis_model_parallel sharded_ddplen
ValueError
local_rankr ImportErrorrT SIMPLEFullyShardedDDPrp r" ZERO_DP_2 ZERO_DP_3place_model_on_devicefp16_full_evalbf16_full_evaldo_trainr& r% r r r r _move_model_to_devicedevice_n_gpu
model_wrappedr r r optimizer lr_schedulerDEFAULT_CALLBACKSr Z report_tor3 callback_handler add_callback disable_tqdmr5 DEFAULT_PROGRESS_CALLBACKZ_loggers_initialized push_to_hub
init_git_repork xm
rendezvousdistbarrier should_saveosmakedirsr callablegetattr max_stepsr[
isinstancetorchutilsdataIterableDatasetgroup_by_length_signature_columnsuse_apexuse_ampfp16bf16Zhalf_precision_backend_is_native_amp_availabledo_grad_scalingfloat16bfloat16 amp_dtyperj smprn r scalerru Z
torch_xla.ampcudarf
max_grad_normZlabel_smoothing_factorr> label_smootherr9 r r stater8 control current_floshp_search_backenduse_tune_checkpointsrd __class__ label_namesZ on_init_endstop_and_update_metrics)selfr r r r r r r r r r r r log_levelZdefault_collatorZdefault_callbacksr Zdefault_label_names r >/usr/local/lib/python3.6/site-packages/transformers/trainer.py__init__ s
"
"
"
( zTrainer.__init__c C s | j j| dS )ac
Add a callback to the current list of [`~transformer.TrainerCallback`].
Args:
callback (`type` or [`~transformer.TrainerCallback`]):
A [`~transformer.TrainerCallback`] class or an instance of a [`~transformer.TrainerCallback`]. In the
first case, will instantiate a member of that class.
N)r r )r callbackr r r r s zTrainer.add_callbackc C s | j j|S )aF
Remove a callback from the current list of [`~transformer.TrainerCallback`] and returns it.
If the callback is not found, returns `None` (and no error is raised).
Args:
callback (`type` or [`~transformer.TrainerCallback`]):
A [`~transformer.TrainerCallback`] class or an instance of a [`~transformer.TrainerCallback`]. In the
first case, will pop the first member of that class found in the list of callbacks.
Returns:
[`~transformer.TrainerCallback`]: The callback removed, if found.
)r pop_callback)r r r r r r s zTrainer.pop_callbackc C s | j j| dS )a
Remove a callback from the current list of [`~transformer.TrainerCallback`].
Args:
callback (`type` or [`~transformer.TrainerCallback`]):
A [`~transformer.TrainerCallback`] class or an instance of a [`~transformer.TrainerCallback`]. In the
first case, will remove the first member of that class found in the list of callbacks.
N)r remove_callback)r r r r r r s zTrainer.remove_callbackc C s. |j |}| jjtjkr*t|dr*|j d S )N tie_weights)tor
parallel_moder` TPUr r )r r r r r r r # s
z Trainer._move_model_to_devicezdatasets.Dataset)dataset descriptionc s | j js S | jd krFtj| jj}t|jj | _| jddg7 _tt
j t
| j }t |dkr|d krvdn
d| d}t
jd| d| jjj d d
j| d d
j| d | jjj d
fdd| jD }tjtjtjdk r j jd | jd d S j|S d S )Nlabel label_idsr zin the z set zThe following columns z) don't have a corresponding argument in `z!.forward` and have been ignored: z, z. If z are not expected by `z/.forward`, you can safely ignore this message.c s g | ]}| j kr|qS r ) column_names).0k)r r r
= s z2Trainer._remove_unused_columns..z1.4.0type
format_kwargs)r columnsr )r Zremove_unused_columnsr inspect signaturer Zforwardlist
parameterskeyssetr r r r r __name__joinr parsedatasetsr" Z
set_formatformatZremove_columns)r r r r Zignored_columnsZdset_descriptionr r )r r _remove_unused_columns) s"
>z Trainer._remove_unused_columns)returnc C s | j d kst| j rd S d }| jjdkrrtrrtj }| jjd kr`ttj f tj
dj j }n| jj}|j
| | jjd k r| jjn| jj}| jjrJt rt| j tjr| jj| j jkr| j | jj nd }nd }| jd k r| jjd nd }| jjdkr t| jj| jj | j |||dS t| jj| jj | j | jj| jj |||dS n| jjdkrvtrlt | j |dS t | j S | jj t j t j!gkr| jj" rt#| j | jj$| jj| jj |dS t%| j | jj| jj |dS d S ) Nr
)dtyper )r lengthsmodel_input_name generator)r num_replicasrankr r
r )r )
batch_sizer r r )r r r )&r r[ r
world_size _is_torch_generator_availabler GeneratorZ data_seedintemptyint64Zrandom_itemZ manual_seedr r rg r r r Zlength_column_namer r Zmodel_input_namesr? train_batch_sizegradient_accumulation_stepsr:
process_indexr r r` r ZSAGEMAKER_MODEL_PARALLELdataloader_drop_lastr; per_device_train_batch_sizer )r r r r r
r r r _get_train_samplerG s`
zTrainer._get_train_samplerc C s | j dkrtd| j }t r8t|tjr8| j|dd}t|tjj j
r| j j dkrvt
|| j j| j j| j j | j jd}t|| j j| j| j j| j jdS | j }t|| j j|| j| j j| j j| j jdS ) a@
Returns the training [`~torch.utils.data.DataLoader`].
Will use no sampler if `train_dataset` does not implement `__len__`, a random sampler (adapted to distributed
training if necessary) otherwise.
Subclass and override this method if you want to inject some custom behavior.
Nz+Trainer: training requires a train_dataset.training)r r
)r drop_last
num_processesr )r
collate_fn num_workers
pin_memory)r samplerr" r r# r$ )r r rg r r r r r r r r r r r= r r r r r r dataloader_num_workersdataloader_pin_memoryr )r r Z
train_samplerr r r get_train_dataloader s8
z Trainer.get_train_dataloader)r r
c C s | j jrbt r$t|tj tj dS t rFt|tj tj
| j j dS | j j dkrZt|S t
|S | j jdkrvt
|S t|| j j | j j| j jdS d S )N)r r )r r r r
)r r! r r )r use_legacy_prediction_looprk r@ r xrt_world_sizeZ get_ordinalrj r Zdp_sizeZdp_rankper_device_eval_batch_sizer r r rA r )r r r r r _get_eval_sampler s( zTrainer._get_eval_samplerc C s |dkr| j dkrtd|dk r&|n| j }t rLt|tjrL| j|dd}t|tjj j
r| j j dkrt
|| j j| j j| j j | j jd}t|| j j| j| j j| j jdS | j|}t||| j j| j| j j| j j| j jdS ) a
Returns the evaluation [`~torch.utils.data.DataLoader`].
Subclass and override this method if you want to inject some custom behavior.
Args:
eval_dataset (`torch.utils.data.Dataset`, *optional*):
If provided, will override `self.eval_dataset`. If it is an `datasets.Dataset`, columns not accepted by
the `model.forward()` method are automatically removed. It must implement `__len__`.
Nz-Trainer: evaluation requires an eval_dataset.Z
evaluation)r r
)r r r! r )r r" r# r$ )r% r r" r r# r$ )r r rg r r r r r r r r r r r= r+ r r r eval_batch_sizer r& r' r, )r r Z eval_samplerr r r get_eval_dataloader s8
zTrainer.get_eval_dataloader) test_datasetr
c C s t r t|tjr | j|dd}t|tjjjr~| j j
dkr^t || j j | j j
| j j
| j jd}t|| j j | j| j j| j jdS | j|}t||| j j | j| j j
| j jdS )a
Returns the test [`~torch.utils.data.DataLoader`].
Subclass and override this method if you want to inject some custom behavior.
Args:
test_dataset (`torch.utils.data.Dataset`, *optional*):
The test dataset to use. If it is an `datasets.Dataset`, columns not accepted by the `model.forward()`
method are automatically removed. It must implement `__len__`.
test)r r
)r r r! r )r r" r# r$ )r% r r" r r$ )rg r r r r r r r r r r r= r- r r r r r& r' r, )r r/ Z test_samplerr r r get_test_dataloader
s0
zTrainer.get_test_dataloader)num_training_stepsc C s | j | j|| jd dS )aZ
Setup the optimizer and the learning rate scheduler.
We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the
Trainer's init through `optimizers`, or subclass and override this method (or `create_optimizer` and/or
`create_scheduler`) in a subclass.
)r2 r N)create_optimizercreate_schedulerr )r r2 r r r create_optimizer_and_scheduler5 s z&Trainer.create_optimizer_and_schedulerc s | j dkrt| jtjg dd D fdd| jj D | jjd fdd| jj D ddg}tj | j\}}| j
t j krt
f ||d|| _ n||f|| _ t rtj| j | _ | j S ) a
Setup the optimizer.
We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the
Trainer's init through `optimizers`, or subclass and override this method in a subclass.
Nc S s g | ]}d |kr|qS )Zbiasr )r namer r r r I s z,Trainer.create_optimizer..c s g | ]\}}| kr|qS r r )r np)decay_parametersr r r L s )params weight_decayc s g | ]\}}| kr|qS r r )r r7 r8 )r9 r r r P s g )r: optim)r rE r r Z LayerNormZnamed_parametersr r; rz get_optimizer_cls_and_kwargsr rT r rt rj r ZDistributedOptimizer)r Z optimizer_grouped_parameters
optimizer_clsoptimizer_kwargsr )r9 r r3 @ s"
zTrainer.create_optimizer)r r
c C s@ d| j i}| j| jf| jd}| jtjkr@t}|jddd n| jtj krhddl
m } |}|j| n| jtj krddl
m } |}|j| n| jtjkry ddlm } |}|j| W n tk
r tdY nX n^| jtjkr(y dd lm} |}|j| W n tk
r$ td
Y nX ntd | j ||fS ) z
Returns the optimizer class and optimizer parameters based on the training arguments.
Args:
args (`transformers.training_args.TrainingArguments`):
The training arguments for the training session.
lr)ZbetasZepsF)Zscale_parameterZ
relative_stepr
)AdamWr z7Trainer failed to import syncfree AdamW from torch_xla.) FusedAdamzFTrainer tried to instantiate apex FusedAdam but apex is not installed!z2Trainer cannot instantiate unsupported optimizer: )
learning_rateZ
adam_beta1Z
adam_beta2Z adam_epsilonr< r_ Z ADAFACTORr0 updateZADAMW_HF optimizationrA Z ADAMW_TORCHZ torch.optimZADAMW_TORCH_XLAZtorch_xla.amp.syncfreer r ZADAMW_APEX_FUSEDZapex.optimizersrB )r r? Z adam_kwargsr> rA rB r r r r= e s<
z$Trainer.get_optimizer_cls_and_kwargs)r2 r c C s< | j dkr6t| jj|dkr | jn|| jj||d| _ | j S )z
Setup the scheduler. The optimizer of the trainer must have been set up either before this method is called or
passed as an argument.
Args:
num_training_steps (int): The number of training steps to do.
N)r Znum_warmup_stepsr2 )r r1 r Zlr_scheduler_typer Zget_warmup_steps)r r2 r r r r r4 s
zTrainer.create_scheduler)
dataloaderr
c
C s6 y
t |jS tttfk
r0 t || jj S X dS )z
Helper to get number of samples in a [`~torch.utils.data.DataLoader`] by accessing its dataset. When
dataloader.dataset does not exist or has no length, estimates as best it can
N)r r NameErrorAttributeError TypeErrorr r )r rF r r r num_examples s
zTrainer.num_examplesz optuna.Trial)trialc C sn || _ | jdks|dkr dS | jtjkr4| j|}nP| jtjkrR|}|jdd n2| jtjkrtdd |jj D }n| jtj
kr|}xd|j D ]X\}}t | j |st
jd| d qt| j |d}|dk rt||}t| j || qW | jtjkrt
jd|j | jtjkr&t
jd|j | jtj
krDt
jd | | j jrjd
d lm} || j j| j _dS ) zHP search setup codeNwandbc S s( i | ] \}}t |tr t|n||qS r )r strr )r r vr r r
s z,Trainer._hp_search_setup..zTrying to set zY in the hyperparameter search but there is no corresponding field in `TrainingArguments`.zTrial:zSigOpt Assignments: zW&B Sweep parameters: r )HfDeepSpeedConfig)Z_trialr rP OPTUNAhp_spaceRAYpopSIGOPT assignmentsitemsWANDBr r r warningr r setattrr r: r Ztransformers.deepspeedrP hf_deepspeed_config)r rK r: keyvalueold_attrrP r r r _hp_search_setup s<
zTrainer._hp_search_setup)rK epochmetricsc C s | j d ks|d krd S | j|j | _| j tjkrpdd l}|j| j| |j r| j j
| j | j | j
|j n>| j tjkrddlm} | j
jr| j |jf d| ji| d S )Nr )tune objective)r compute_objectivecopyrc rP rQ optunareportZ should_pruner on_train_endr r r Z TrialPrunedrS rayrb r _tune_save_checkpoint)r rK r` ra rf rb r r r _report_to_hp_search s
z Trainer._report_to_hp_searchc C s ddl m} | jsd S |j| jjd}tjj|t d| jj }| j
|dd | j j r| jj
tjj|t tj| jj tjj|t tj| jj tjj|t W d Q R X d S )Nr )rb )step-T)_internal_call)ri rb r checkpoint_dirr global_stepr pathr rL
save_modelr r save_to_jsonTRAINER_STATE_NAMEr saver
state_dictOPTIMIZER_NAMEr SCHEDULER_NAME)r rb ro r r r r rj s z Trainer._tune_save_checkpointc C sL t | j}|dkr | j }n |dkr0| j|}ntd|d krHtd|S )Nr r
z'model_init should have 0 or 1 argument.z"model_init should not return None.)r\ r r )r rK Zmodel_init_argcountr r r r r s
zTrainer.call_model_initTc C s t r.t| jtjjr | jS tj|| jjdS | jr:| jS t ||k rJ|S | j
rp|rpt j || j
| jjd\}| _
| jjdkrtj|}|s|S | jd k r| jtjkrt|| j
}n`| jjp| jj}tj| jjk}| jtjk}tj| jjkrt|}t ||||dj | jj | _}nt r}|}
| |j8 }|j8| }nt d|j8 t?j@| jjAkr | jjBdkrt dn
tC| j}| jDdk r:| jDtEjFkp>tG }|j+r~tH| ||d\}}}|jI| _|| _0|| _+|| _|| _n|s| jJ|d tK | _L|dk | jL_M|jNr| jjO | jP| j0}|| jk r|| _0|r| jJ|d | jQ| t#j$d t#j$d| t#j$d| t#j$d |jR t#j$d | t#j$d |j3 t#j$d | d| jL_StTjT }d}d}d}|dk rXt j j!t j j|tUrXtKjVt j j|tU| _L| jLjW|
}|jXs| jLjW|
}||j39 }nd}t#j$d t#j$d!| t#j$d"| jLjW |jXsXt#j$d#| d$| d% | jY rX|jZ rXt[|d&}|j\d' | j| j]_| j| j]_| j| j]_|
| j]_^| j_dk r| j_|nd| jL_`|dk r| jatbjckr|jdn|} te| | jL_fnd| jL_f|| jL_8|| jL_<| jY | jL_Y| jg | jL_gt,jhd(ji|j } d(| _j| jLjW| _k|jl | j]jm|| jL| jn| _n|jXsxhto|D ]\} tp|
d)oht|
jqtr} tsjtt,j)tsjtd*k s| rx |
D ]} P qW n
t|
jq} qNW xto||D ]} t|
turt|
jqtvr|
jqjw| n&tp|
d+rt|
jxtyr|
jxjw| tz r6t{j||
|j gj}|j }!|!}"n|
}"|j~dkrLd| _| dk r^t|"n
|j8|j3 }#| j]j|| jL| jn| _ndB}$x~t|"D ]p\}$}%|dkr|d8 }|dk r|jd |dkr| j| qn|dk r|j d}|$|j3 dkr| j]j|| jL| jn| _n|$d |j3 dkr`|jdCkr`|jr`|j | j||%}&W dQ R X n | j||%}&|jrtz rt,j|&st,j|&r| | d| jLjW | jk 7 } n| |&7 } | jt| j|%7 _| j+r| j+j |$d |j3 dk s|#|j3k
r|$d |#k
r|jdk r|jdk r| j+ r| j rvtz rhtj| j}'tjd,|'d-tj d. | jj| j tp| jd/ r| jj|j nBtp|d0 r|j|j n(tjj| j rtj| jn|j |j d}(| j+ rntz
r | j
r| jj| j | jj n tj| j nH| j
rZ| jj })| jj| j | jj | jj }*|)|*k}(n
| jj |(
r~| j+
r~| jj |jl | jL jWd7 _W| |$d |# | jL_S| j]j|| jL| jn| _n| j| ||| | n| j]j|| jL| jn| _n| jnj
s| jnjrP qW |$dk r4t#j*d1| jLjW d2| d3 d| jn_| j]j|| jL| jn| _n| j| ||| | t?j| jjAk rtz rtjtj n
t#j*d4 | jnjrP qW |j~ rtp| d5 rt| d5 t#j$d6 |j r| jLjdk rtz rtjd7 n|jdDk rtj t#j$d8| jLj d9| jLj d t j j| jLjt"}+t j j|+ r| j+ rt| \}}}|jI| _|| _0|| _+|| _|| _| j+j| jLjddd: nt,j-|+dd} | j.| nt#j*d;|+ d< | jj| j 7 _j| jj| jLjW },td=||| jLj8d>}-| j | jLj|-d?< |,|-d@< d | _| j j|- | j|- | j]j|| jL| jn| _nt| jLjW|,|-S )Ea
Main training entry point.
Args:
resume_from_checkpoint (`str` or `bool`, *optional*):
If a `str`, local path to a saved checkpoint as saved by a previous instance of [`Trainer`]. If a
`bool` and equals `True`, load the last checkpoint in *args.output_dir* as saved by a previous instance
of [`Trainer`]. If present, training will resume from the model/optimizer/scheduler states loaded here.
trial (`optuna.Trial` or `Dict[str, Any]`, *optional*):
The trial run or the hyperparameter dictionary for hyperparameter search.
ignore_keys_for_eval (`List[str]`, *optional*)
A list of keys in the output of your model (if it is a dictionary) that should be ignored when
gathering predictions for evaluation during the training.
kwargs:
Additional keyword arguments used to hide deprecated arguments
NTZ
model_pathzi`model_path` is deprecated and will be removed in a future version. Use `resume_from_checkpoint` instead.r z3train() received got unexpected keyword arguments: z, .Fz/No valid checkpoint found in output directory ()z!Can't find a valid checkpoint at zLoading model from z).z9You are resuming training from a checkpoint trained with z- of Transformers but your current version is zJ. This is not recommended and could yield to errors or unwanted behaviors.cpu) map_locationr
zYargs.max_steps must be set to a positive value if dataloader does not have a length, was zjCurrently --debug underflow_overflow is not supported under DP. Please use DDP (torch.distributed.launch).)r2 r )r2 z ***** Running training *****z Num examples = z Num Epochs = z( Instantaneous batch size per device = zE Total train batch size (w. parallel, distributed & accumulation) = z Gradient Accumulation steps = z Total optimization steps = zE Continuing training from checkpoint, will skip to saved global_stepz! Continuing training from epoch z' Continuing training from global step z Will skip the first z epochs then the first z batches in the first epoch. If this takes a lot of time, you can add the `--ignore_data_skip` flag to your launch command, but you will resume the training on data already seen by your model.)totalzSkipping the first batchesg r% z1.11r sumg ?)scaleclip_grad_normclip_grad_norm_zXThere seems to be not a single sample in your epoch_iterator, stopping training at step zI! This is expected if you're using an IterableDataset and set num_steps (z.) higher than the number of available samples.zYou enabled PyTorch/XLA debug metrics but you don't have a TPU configured. Check your training configuration if this is unexpected._pastzU
Training completed. Do not forget to share your model on huggingface.co/models =)
load_best_model_at_endzLoading best model from z (score: )Zload_optimizer_statesZload_lr_scheduler_statesz#Could not locate the best model at zi, if you are running a distributed training on multiple nodes, you should activate `--save_on_each_node`.train) num_samples num_steps
total_flos
train_loss)NNr r r )r r r r r r r r r r rT r r r r rI r r r r_ r r] r r r r r boolrZ r r r rq isfilerc r r rb r# Zfrom_json_fileZtransformers_versionr" rY r r load_load_state_dict_in_modelr r r( r r r r[ maxrJ r r mathceilnum_train_epochssysmaxsizer' ZUNDERFLOW_OVERFLOWdebugr} r( r rT r rj r) moduler5 r9 r Zis_hyper_param_searchZgradient_checkpointingZ gradient_checkpointing_enabler _load_optimizer_and_schedulerr r` timert Zload_from_jsonrp Zignore_data_skipr r r set_descriptionr train_dataloaderr Z
trial_namer rP rU rV r Z trial_paramsr tensorr _total_loss_scalar_globalstep_last_loggedZ zero_gradZon_train_beginr ranger r% r r r r r Z set_epochr r= rk plParallelLoaderper_device_loader
past_indexr Zon_epoch_begin enumeraterD _load_rng_statecloseZ
on_step_beginr Z!_no_sync_in_gradient_accumulationZno_sync
training_stepZlogging_nan_inf_filterisnanisinfr floatfloating_point_opsrl r r r Z_fetch_gradientsZ
all_reducer* r Zunscale_r r r r r rn Z
master_paramsr Zoptimizer_stepZ get_scaleZ on_step_end_maybe_log_save_evaluateZon_substep_endZshould_epoch_stopZshould_training_stopZ on_epoch_endTPU_METRICS_DEBUG master_printmetmetrics_reportdelattrr best_model_checkpointr r r best_metricexistsr* Zload_checkpointr r^
store_flosr r logrh rV ).r r rK r r r Zmodel_reloadedconfigZcheckpoint_versionrv r Ztotal_train_batch_sizeZlen_dataloaderZnum_update_steps_per_epochrJ r r Znum_train_samplesZdebug_overflowZdelay_optimizer_creationdeepspeed_enginer r r
start_timeZepochs_trainedZ steps_trained_in_current_epochZsteps_trained_progress_barrV tr_lossr` Zis_random_sampler_parallel_loaderZepoch_iteratorZsteps_in_epochrl inputsZ tr_loss_stepZ gradientsZoptimizer_was_runZ scale_beforeZ scale_afterZbest_model_pathr ra r r r r O s0
"
z
Trainer.trainc C s | j j|dd}t|jdkr`| j jd k rLt|jt| j jkrL| j j ntjd|j d t|j dkrtjd|j d d S )NF)strictr z8There were missing keys in the checkpoint model loaded: r z;There were unexpected keys in the checkpoint model loaded: )
r load_state_dictr Z missing_keysZ_keys_to_ignore_on_saver r r rY Zunexpected_keys)r rv Z load_resultr r r r 1 s z!Trainer._load_state_dict_in_modelc C s | j jrt rtj i }| j|j j }||8 }t|| j j
| j d|d< | j |d< | j
|7 _
| j j
| _ | j | j| d }| j jr| j|d}| j||| | j jr| j|||d | jj| j| j | j | _ d S )N lossrC ) ignore_keys)ra )r Z
should_logrk r mark_step_nested_gathermeanr roundr rp r r{ r r r Zshould_evaluateevaluaterk r _save_checkpointr Zon_saver ) r r r rK r` r logsZtr_loss_scalarra r r r r @ s&
z Trainer._maybe_log_save_evaluatec C sd |d kr d S t rtj n| jj}|dkrntjj|d| d}tjjtjj||st j
d| d d S n(tjj|d}tjj|st j
d d S t j |}t
j|d tj
j|d t j
j|d
t jj rJ| jjdk rt jj
j|d nNyt jj
j|d W n6 tk
rH } zt j
d | d
W Y d d }~X nX t r`tj|d d S )Nr
rng_state_z.pthz$Didn't find an RNG file for process zr, if you are resuming a training that wasn't launched in a distributed fashion, reproducibility is not guaranteed.z
rng_state.pthzDidn't find an RNG file, if you are resuming a training that was launched in a distributed fashion, reproducibility is not guaranteed.pythonnumpyr r zUDidn't manage to set back the RNG states of the GPU because of the following error:
zO
This won't yield the same results as if the training had not been interrupted.xlar r )rk r get_local_ordinalr r r rq r r r r r r randomsetstatenpZ set_stateZ
set_rng_stater is_availableZset_rng_state_all Exception)r
checkpointr Zrng_fileZcheckpoint_rng_stateer r r r _ s8
zTrainer._load_rng_statec C sd t d| jj }| jd k r|d k r| jtjkr8|j}nR| jtjkrZddlm } |j
}n0| jtj krn|j }n | jtj
krdd l}|jj }| jd k r| j|nd| }tjj| jj|} n| jj} | j tjj| |}
| j|
dd | jr| jj|
| jtj kr| j j t rt j!d t j"| j j# tjj|
t$ t%j&dd*} t j"| j'j# tjj|
t( t)| W d Q R X n2t* r"t+j, dkr| j j# } | jj-rt.j"| tjj|
t$ t%j&dd"} t.j"| j'j# tjj|
t( W d Q R X t)| | j/rt.j"| j0j# tjj|
t1 n| jj-r| j rt.j"| j j# tjj|
t$ t%j&dd"} t.j"| j'j# tjj|
t( W d Q R X t)| | j/rt.j"| j0j# tjj|
t1 |d k rB| jj2d k rB| jj2}
|
j3d sd |
}
||
}| jj4r t5j6nt5j7}| jj8d ks2| jj9d ks2||| jj8rB|| j_8|
| j_9| jj-rb| jj:tjj|
t; t t.j
To use this method, you need to have provided a `model_init` when initializing your [`Trainer`]: we need to
reinitialize the model at each new run. This is incompatible with the `optimizers` argument, so you need to
subclass [`Trainer`] and override the method [`~Trainer.create_optimizer_and_scheduler`] for custom
optimizer/scheduler.
Args:
hp_space (`Callable[["optuna.Trial"], Dict[str, float]]`, *optional*):
A function that defines the hyperparameter search space. Will default to
[`~trainer_utils.default_hp_space_optuna`] or [`~trainer_utils.default_hp_space_ray`] or
[`~trainer_utils.default_hp_space_sigopt`] depending on your backend.
compute_objective (`Callable[[Dict[str, float]], float]`, *optional*):
A function computing the objective to minimize or maximize from the metrics returned by the `evaluate`
method. Will default to [`~trainer_utils.default_compute_objective`].
n_trials (`int`, *optional*, defaults to 100):
The number of trial runs to test.
direction(`str`, *optional*, defaults to `"minimize"`):
Whether to optimize greater or lower objects. Can be `"minimize"` or `"maximize"`, you should pick
`"minimize"` when optimizing the validation loss, `"maximize"` when optimizing one or several metrics.
backend(`str` or [`~training_utils.HPSearchBackend`], *optional*):
The backend to use for hyperparameter search. Will default to optuna or Ray Tune or SigOpt, depending
on which one is installed. If all are installed, will default to optuna.
kwargs:
Additional keyword arguments passed along to `optuna.create_study` or `ray.tune.run`. For more
information see:
- the documentation of
[optuna.create_study](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.create_study.html)
- the documentation of [tune.run](https://docs.ray.io/en/latest/tune/api_docs/execution.html#tune-run)
- the documentation of [sigopt](https://app.sigopt.com/docs/endpoints/experiments/create)
Returns:
[`trainer_utils.BestRun`]: All the information about the best run.
NzAt least one of optuna or ray should be installed. To install optuna run `pip install optuna`. To install ray run `pip install ray[tune]`. To install sigopt run `pip install sigopt`.zQYou picked the optuna backend, but it is not installed. Use `pip install optuna`.zXYou picked the Ray Tune backend, but it is not installed. Use `pip install 'ray[tune]'`.zQYou picked the sigopt backend, but it is not installed. Use `pip install sigopt`.zOYou picked the wandb backend, but it is not installed. Use `pip install wandb`.zXTo use hyperparameter search, you need to pass your model through a model_init function.)r r rP rQ r rS r rU r rX r r r rX rR r rW rd r r r r )
r rR rd r r r r r Z backend_dictZbest_runr r r hyperparameter_search! s: 4
z Trainer.hyperparameter_search)r r
c C sZ | j jdk r t| j jd|d< |d| j ji}| j jj| | jj| j| j | j || _ dS )z
Log `logs` on the various objects watching training.
Subclass and override this method to inject custom behavior.
Args:
logs (`Dict[str, float]`):
The values to log.
N r` rl )
r r` r rp Z log_historyappendr Zon_logr r )r r outputr r r r ~ s
z Trainer.log)r r
c s t |tr(t| fdd|j D S t |ttfrPt| fdd|D S t |tjrt j j
d} j r|j tj
kr|jt j jj d |jf |S |S )z|
Prepares one `data` before feeding it to the model, be it a tensor or a nested list/dictionary of tensors.
c s i | ]\}} j ||qS r )_prepare_input)r r rN )r r r rO s z*Trainer._prepare_input..c 3 s | ]} j |V qd S )N)r )r rN )r r r s z)Trainer._prepare_input..)r )r )r r r rW tupler r Tensordictr r r r r rD r[ r )r r r r )r r r s
zTrainer._prepare_input)r r
c C sR | j |}t|dkr.tddj| j d| jjdkrN| jdk rN| j|d< |S )z
Prepare `inputs` before feeding them to the model, converting them to tensors if they are not already and
handling potential state.
r zThe batch received was empty, your model won't be able to train on it. Double-check that your training dataset contains keys expected by the model: ,r NZmems)r r r r r r r r )r r r r r _prepare_inputs s
zTrainer._prepare_inputsc C sP | j r2tjtjtjdkr*t| jd}qLt }ntjdkrDt j
nt j }|S )z
A helper wrapper that creates an appropriate context manager for `autocast` while feeding it the desired
arguments, depending on the situation.
z1.10)r )r r ) r r r r r" ro r r version_info
contextlibZ nullcontextsuppress)r Z ctx_managerr r r autocast_smart_context_manager s z&Trainer.autocast_smart_context_manager)r r r
c C s |j | j|}t rR| jr$| jnd}t||| jj|d}|j j j
| jj S | j | j
||}W dQ R X | jjdkr|j }| jjdkr| j r|| jj }| jr| jj|j nF| jrtj|| j}|j W dQ R X n | jr| jj|}n|j |j S )aq
Perform a training step on a batch of inputs.
Subclass and override to inject custom behavior.
Args:
model (`nn.Module`):
The model to train.
inputs (`Dict[str, Union[torch.Tensor, Any]]`):
The inputs and targets of the model.
The dictionary will be unpacked before being fed to the model. Most models expect the targets under the
argument `labels`. Check your model's documentation for all accepted arguments.
Return:
`torch.Tensor`: The tensor with training loss on this batch.
N)r r
)r r rj r r rv r r reduce_meandetachr r r compute_lossr} r r r Zbackwardr rn Z
scale_lossr )r r r r loss_mbr Z scaled_lossr r r r s*
zTrainer.training_stepFc C s | j dk r d|kr |jd}nd}|f |}| jjdkrF|| jj | _|dk r\| j ||}nt|trn|d n|d }|r||fS |S )z
How the loss is computed by Trainer. By default, all models return the loss in the first element.
Subclass and override for custom behavior.
Nlabelsr r )r rT r r r r r )r r r return_outputsr outputsr r r r r s
zTrainer.compute_lossc C s | j jdkS )z
Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on several
machines) main process.
r )r Zlocal_process_index)r r r r r s z Trainer.is_local_process_zeroc C s" t rtj dkS | jjdkS dS )z
Whether or not this process is the global main process (when training in a distributed fashion on several
machines, this is only going to be `True` for one process).
r N)rj r r r r )r r r r r s z Trainer.is_world_process_zero)r rn c C s. |dkr| j j}t r"| j| nt rJ| jj }| j jrH| j||d nt j
| j j ksft j | j j kr| j
j }| j jr| j||d n| jr| j jr| j| t r| j jrtjj|t}tjj|rtj| | jj|tstjd | jj| n| j jr | j| | j jr*| r*| jdd dS )z
Will save the model, so you can reload it using `from_pretrained()`.
Will only save from the main process.
N)rv zdeepspeed.save_16bit_model didn't save the model, since stage3_gather_16bit_weights_on_model_save=false. Saving the full checkpoint instead, use zero_to_fp32.py to recover weightsz
Model save)commit_message)r r rk _save_tpurj r rv r _saverT r r r r r r+ r rq r rc r removeZsave_16bit_modelr rY r r )r r rn rv filer r r rr s8
zTrainer.save_model)r c C s |d k r |n| j j}tjd| tj rRtj|dd tj | j tj
j |t tj
d t| jtstt| jtrt| jj|| j j| jj tj d qtjd | jj }tj |tj
j |t n| jj|| j jtj d | jd k o| j jr| jj| d S )NzSaving model checkpoint to T)r Zsaving_checkpoint) save_configrv
save_functionzETrainer.model is not a `PreTrainedModel`, only saving its state dict.)r r )r r r r r Zis_master_ordinalr r r ru rq r TRAINING_ARGS_NAMEr r r r. r/ save_pretrainedr rv rc r )r r rv r r r r V s&
zTrainer._save_tpuc C s |d k r |n| j j}tj|dd tjd| t| jtstt | jtrv|d kr`| jj
}t | jj ||d qtjd |d kr| jj
}t j
|tjj|t n| jj ||d | jd k r| jj | t j
| j tjj|t d S )NT)r zSaving model checkpoint to )rv zETrainer.model is not a `PreTrainedModel`, only saving its state dict.)r r r r r r r r r. r/ rv r r ru rq r rc r r )r r rv r r r r
r s
z
Trainer._savec C sX | j jdkr<| j jt| jg| j jdj j 7 _d| _n| j j| j7 _d| _d S )Nr
)r r r ) r r r r rB r r r r )r r r r r s zTrainer.store_flosc C s
g }dd t |j| dD }xj|D ]b}|rH|jtjj||f q(tjd| d|}|d k r(|j d k r(|jt |j d |f q(W t
|}dd |D }| j j d k r|j
tt | j j } x.z-*z.*z -([0-9]+)r c S s g | ] }|d qS )r
r )r r r r r r s r r
)r globr r rq getmtimerematchgroupsr sortedr r indexrM r r ) r r Zcheckpoint_prefixr Z ordering_and_checkpoint_pathZglob_checkpointsrq Z regex_matchcheckpoints_sortedZbest_model_indexir r r _sorted_checkpoints s
&zTrainer._sorted_checkpointsc C s | j jd ks| j jdkr d S | j||d}t|| j jkr>d S | j j}| jjd k rr| j jdkrr|d | jjkrrd}tdt|| }|d | }x(|D ] }tjd| d t j
| qW d S )Nr )r r r
r zDeleting older checkpoint [z ] due to args.save_total_limitr ) r save_total_limitr r r r r r r shutilrmtree)r r r r r Z number_of_checkpoints_to_deleteZcheckpoints_to_be_deletedr r r r r s
zTrainer._rotate_checkpointseval)r r metric_key_prefixr
c C s | j j | j|}tj }| jjr*| jn| j}||d| jdkrDdnd||d}| jj | jj
}|j j t
|||jtj|j| d | j|j tj| jjkrtjtj | jj| j| j| j |j | _ | j j |j |j S )a$
Run evaluation and returns metrics.
The calling script will be responsible for providing a method to compute metrics, as they are task-dependent
(pass it to the init `compute_metrics` argument).
You can also subclass and override this method to inject custom behavior.
Args:
eval_dataset (`Dataset`, *optional*):
Pass a dataset if you wish to override `self.eval_dataset`. If it is an `datasets.Dataset`, columns not
accepted by the `model.forward()` method are automatically removed. It must implement the `__len__`
method.
ignore_keys (`Lst[str]`, *optional*):
A list of keys in the output of your model (if it is a dictionary) that should be ignored when
gathering predictions.
metric_key_prefix (`str`, *optional*, defaults to `"eval"`):
An optional prefix to be used as the metrics key prefix. For example the metrics "bleu" will be named
"eval_bleu" if the prefix is "eval" (default)
Returns:
A dictionary containing the evaluation loss and the potential metrics computed from the predictions. The
dictionary also contains the epoch number which comes from the training state.
Z
EvaluationNT)r prediction_loss_onlyr r# )r r ) r r r. r r r) prediction_loopevaluation_loopr r- r ra rD r^ r r r r r' r r r r r r r Z on_evaluater r r ) r r r r# eval_dataloaderr eval_loopr total_batch_sizer r r r s.
zTrainer.evaluater0 )r/ r r# r
c C s | j j | j|}tj }| jjr*| jn| j}||d||d}| jj| jj }|j
j t |||j
tj|j
| d | j j|j
t|j|j|j
dS )a
Run prediction and returns predictions and potential metrics.
Depending on the dataset and your use case, your test dataset may contain labels. In that case, this method
will also return metrics, like in `evaluate()`.
Args:
test_dataset (`Dataset`):
Dataset to run the predictions on. If it is an `datasets.Dataset`, columns not accepted by the
`model.forward()` method are automatically removed. Has to implement the method `__len__`
ignore_keys (`Lst[str]`, *optional*):
A list of keys in the output of your model (if it is a dictionary) that should be ignored when
gathering predictions.
metric_key_prefix (`str`, *optional*, defaults to `"test"`):
An optional prefix to be used as the metrics key prefix. For example the metrics "bleu" will be named
"test_bleu" if the prefix is "test" (default)
If your predictions or labels have different sequence length (for instance because you're doing dynamic padding
in a token classification task) the predictions will be padded (on the right) to allow for concatenation into
one array. The padding index is -100.
Returns: *NamedTuple* A namedtuple with the following keys:
- predictions (`np.ndarray`): The predictions on `test_dataset`.
- label_ids (`np.ndarray`, *optional*): The labels (if the dataset contained some).
- metrics (`Dict[str, float]`, *optional*): The potential dictionary of metrics (if the dataset contained
labels).
Z
Prediction)r r r# )r r ) predictionsr ra )r r r1 r r r) r% r& r- r ra rD r^ r r r r rS r* r ) r r/ r r# Ztest_dataloaderr r( r r) r r r predict s $
zTrainer.predict)rF r r$ r r# r
c C s@ | j }|dk r|n|j}|jrP| j rPt| dddd\}}}|j| _|| _|| _| j| jdd} | js|j r| j
t j |j
d} n|jr| j
t j|j
d} | j j}
tjd| d t|rtjd
| j| n
tjd tjd |
| j || j_t|d
d} t r(tj||j
gj |j
}|j dkr:d| _ d} d}
d}d}d}d}d}xt |D ]\}}t |}|dk r||7 }|
dkr|}
| j!| |||d\}}}t rt"j# |dk r| j$|j%|
}| dkr|nt j&| |fdd} |dk r0| j'|}| j$|}|dkr"|n t(||dd}|dk r| j'|}| j$|}| j)dk rf| j)||}|
dkrt|n t(|
|dd}
| jj*|| j+| j,| _,|j-dk rb|d |j- dkrb| dk rt.| }|dkr|nt/j0||fdd}|
dk rt.|
}|dkr
|n t(||d d}|dk rFt.|}|dkr8|n t(||d d}d \} }
}qbW |j rtt1| drtt2| d | dk rt.| }|dkr|nt/j0||fdd}|
dk rt.|
}|dkr|n t(||d d}|dk rt.|}|dkr|n t(||d d}t| rt3| }n:t4| t5r6t1| dr6| j}nt|rL| j|}n|}|dk rf|d| }|dk rzt6||}|dk rt6||}| j7dk r|dk r|dk r| j7t8||d}ni }t9|}|dk r|j: j; || d< x@t<|j= D ]0} | j>| ds|j?| || d| < qW t@||||dS )!z
Prediction/evaluation loop, shared by `Trainer.evaluate()` and `Trainer.predict()`.
Works both with or without labels.
Nr T)r2 r Z inferenceF)r )r r z***** Running z *****z Num examples = z Num examples: Unknownz Batch size = r )r )dimd )
padding_indexr
)Zaxisr rJ )r* r _lossr )r* r ra r iiii)NNNii)Ar r$ r r) r r r r r r r r r r r r r+ r r r[ rJ r" r r' r rk r r r r r r rD prediction_stepr r r repeatcat_pad_across_processesrF r on_prediction_stepr r eval_accumulation_stepsrH r Z concatenater r r r r= rI r rO rY r r r r r rT rN ) r rF r r$ r r# r r r r r r losses_host
preds_host labels_hostZ
all_lossesZ all_predsZ
all_labelsZobserved_num_examplesrl r Zobserved_batch_sizer logitsr lossesr ra r\ r r r r& G s
zTrainer.evaluation_loopc C sR |dkr dS t r*|dkr d}t||}n$t r:t|}n| jjdkrNt|}|S )z
Gather value of `tensors` (tensor or list/tuple of nested tensors) and convert them to numpy before
concatenating them to `gathered`
NZ
nested_gatherr
r )rk rJ rj rx r r rC )r tensorsr6 r r r r s
zTrainer._nested_gatherr- c s t |ttfr*t| fdd|D S t |trTt| fdd|j D S t |tjsttdt| dt |j
dk r|S tj |j
|j dd }j
|j }td
d |D }|j
d |kr|S |j
}t|}||d < |jt| }||d d d |d f< |S ) z
Recursively pad the tensors in a nested list/tuple/dictionary of tensors from all devices to the same size so
they can safely be gathered.
c 3 s | ]}j | d V qdS )) pad_indexN)r3 )r t)r< r r r r
s z0Trainer._pad_across_processes..c s i | ]\}}j | d |qS ))r< )r3 )r r rN )r< r r r rO
s z1Trainer._pad_across_processes..z Can't pad the values of type z-, only of nested list/tuple/dicts of tensors.r )r Nc s s | ]}|d V qdS )r
Nr )r sr r r r
s r
)r r r r r rW r r rI r shaper r r r r Z new_zeros) r r r< sizesizesmax_sizeZold_sizeZnew_sizeZ
new_tensorr )r< r r r3
s(
z Trainer._pad_across_processes)r r r$ r r
c
s` t fdd| jD }| j dkrLt| jdrHt| jjdg ng |rttfdd| jD }t |dkr|d }nd}t
j t r@t
|}|rt|tr|d }t fd
d|j D } n|d }|dd } |j j j }
t| } n8d}
t|tr2t fd d|j D } n|} t| } n|r| j | j|d d
\}
} W dQ R X |
j j }
t| trt fdd| j D } n | dd } nnd}
| j |f } W dQ R X t| trt fdd| j D } n| } | jjdkr | | jjd | _W dQ R X |r8|
ddfS t| } t | dkrV| d } |
| |fS )a
Perform an evaluation step on `model` using `inputs`.
Subclass and override to inject custom behavior.
Args:
model (`nn.Module`):
The model to evaluate.
inputs (`Dict[str, Union[torch.Tensor, Any]]`):
The inputs and targets of the model.
The dictionary will be unpacked before being fed to the model. Most models expect the targets under the
argument `labels`. Check your model's documentation for all accepted arguments.
prediction_loss_only (`bool`):
Whether or not to return the loss only.
ignore_keys (`Lst[str]`, *optional*):
A list of keys in the output of your model (if it is a dictionary) that should be ignored when
gathering predictions.
Return:
Tuple[Optional[torch.Tensor], Optional[torch.Tensor], Optional[torch.Tensor]]: A tuple with the loss,
logits and labels (each being optional).
c 3 s | ]} j |d k V qd S )N)get)r r )r r r r H
s z*Trainer.prediction_step..Nr Zkeys_to_ignore_at_inferencec 3 s | ]} j |V qd S )N)rC )r r6 )r r r r R
s r
r r c 3 s$ | ] \}}| d g kr|V qdS )r Nr )r r rN )r r r r ^
s c 3 s | ]\}}| kr|V qd S )Nr )r r rN )r r r r h
s T)r c 3 s$ | ] \}}| d g kr|V qdS )r Nr )r r rN )r r r r s
s c 3 s | ]\}}| kr|V qd S )Nr )r r rN )r r r r {
s )allr r r r r r rG r r r Zno_gradrj rw r r rW r r r ry r r r r r r )
r r r r$ r Z
has_labelsr Z raw_outputsr Z logits_mbr r9 r
r )r r r r0 *
s`
zTrainer.prediction_step)r c C s t | jdr| jj|S dS dS )a
For models that inherit from [`PreTrainedModel`], uses that method to compute the number of floating point
operations for every backward + forward pass. If using another model, either implement such a method in the
model or subclass and override this method.
Args:
inputs (`Dict[str, Union[torch.Tensor, Any]]`):
The inputs and targets of the model.
Returns:
`int`: The number of floating-point operations.
r r N)r r r )r r r r r r
s
zTrainer.floating_point_ops)r c C s8 | j s dS | jjdkr dn| jj}| jjdkrDt| jjj j}n| jj}d|krdt|| jjd}yt | jj||d| _
W nF t k
r | jj r|rt
j| jj t | jj||d| _
n Y nX | j
j tjjtjj| jjd o| jjtjkr.ttjj| jjdddd }|jd
g W dQ R X d| _dS ) a
Initializes a git repo in `self.args.hub_model_id`.
Args:
at_init (`bool`, *optional*, defaults to `False`):
Whether this function is called before any training or not. If `self.args.overwrite_output_dir` is
`True` and `at_init` is `True`, the path to the repo (which is `self.args.output_dir`) might be wiped
out.
NT/)token)Z
clone_fromuse_auth_tokenz
.gitignorewzutf-8)encodingz
checkpoint-*/)r r Z hub_token hub_model_idr r absoluter6 re r! repoEnvironmentErrorZoverwrite_output_dirr r! Zgit_pullr rq r r hub_strategyrQ ZALL_CHECKPOINTSopen
writelinespush_in_progress)r r rG repo_namewriterr r r r
s6
zTrainer.init_git_repo) languagelicensetags
model_namefinetuned_fromtasks dataset_tagsr dataset_argsc
C sd | j s d S tj| ||||||||| d
}
|
j } ttjj| jj dd} | j
| W d Q R X d S )N) rT rU rV rW rX rY rZ r r[ z README.mdrH ) r r- Z from_trainerZ
to_model_cardrO r rq r r r write)
r rT rU rV rW rX rY rZ r r[ Ztraining_summaryZ
model_cardfr r r create_model_card
s zTrainer.create_model_cardc C sf | j s| jjtjkr d S | jd k r4| jj r4d S | jj}tt g}xB|D ]:}t
j j t
j j
||rJtjt
j j
||t
j j
|| qJW | jd k r| jj| tj| jt
j j
|t z| jjtjkrt
j j
|d}t
j j|rtj| tj|| | jjtjkrd| j j }ndt | j j }| j j!|ddd\}| _W d | jjtjkr`tj|| X d S )Nzlast-checkpointzTraining in progress, step z Training in progress, epoch FT)r blockingauto_lfs_prune)"r r rN rQ ZENDrQ Zis_doner rb rc r rq r r r re r r r ru r
CHECKPOINTisdirr! moveZ
save_strategyrR ZSTEPSr rp r r` rL r )r r r Zmodeling_filesZ
modeling_fileZtmp_checkpointr r r r r r
s2
$
z Trainer._push_from_checkpointEnd of training)r r_ r
c K s t | ds| j | jjrH| jjdkr6t| jjj}n| jjjdd }| j dd | j
s`dS | j j ||dd}| jjr| j
f d|i| y| j j d |dd W n2 tk
r } ztjd
| W Y dd}~X nX |S ) u
Upload *self.model* and *self.tokenizer* to the 🤗 model hub on the repo *self.args.hub_model_id*.
Parameters:
commit_message (`str`, *optional*, defaults to `"End of training"`):
Message to commit while pushing.
blocking (`bool`, *optional*, defaults to `True`):
Whether the function should return only when the `git push` has finished.
kwargs:
Additional keyword arguments passed along to [`~Trainer.create_model_card`].
Returns:
The url of the commit of your model in the given repository if `blocking=False`, a tuple with the url of
the commit and an object to track the progress of the commit if `blocking=True`
rL NrE r
T)rn )r r_ r` rW zupdate model card README.mdzEError pushing update to the model card. Please read logs and retry.
$r )r r r r rJ r r r6 splitrr r rL r r^ rM r error)r r r_ r rW Zgit_head_commit_urlexcr r r r s&
"zTrainer.push_to_hubc C s | j }t|std|dk r"|n|j}|jrl| j rlt| ddd\}}}|j| _|| _|| _d|j _ d|_
| j | jdd} | j s|j
r| jtj|jd} n|jr| jtj|jd} |j}
| j|} tjd| d tjd
| tjd |
d} d}
d}td |j}t|| |
d
}|sdd}t|drHt |j t rH|j j}t|| |d
}t|| |d
}| j t rt!j"||jgj#|j}|j$dkrd| _%|| j&_'x,t(|D ] \}}| j)| |||d\}}}|dk r|j*|
}| dkr|ntj+| |fdd} |dk r.|
dkr |n t,|
|dd}
|dk rT|dkrF|n t,||d d}| j&j-|| j.| j/| _/|j0dk r|d |j0 dkr|j1| j2| d |s|j1| j2|
d |j1| j2|d d \} }
}qW |j$rt| drt3| d |j1| j2| d |s0|j1| j2|
d |j1| j2|d |j4 }|sF|j4 nd}|sX|j4 nd} | j5dk r|dk r| dk r| j5t6|| d} ni } t7| } |dk r|j8 j9 | | d< x@t:| j; D ]0} | j<| ds| j=| | | d| < qW t>|| | dS ) z
Prediction/evaluation loop, shared by `Trainer.evaluate()` and `Trainer.predict()`.
Works both with or without labels.
z+dataloader must implement a working __len__Nr )r2 r F)r )r r z***** Running z *****z Num examples = z Batch size = r
)make_multiple_ofr% )r )r, r- )r. Z eval_lossesZ
eval_predsZeval_label_idsr )r* r r/ r )r* r ra ii)NNN)?r r[ r r$ r r) r r r r r r r r r r r r r r r rJ r r r r r< r r r% r@ r" rk r r r r r r r' r r0 r1 r2 rF r4 r r r5 Z
add_arrays_gather_and_numpifyr finalizer rO rY r r r r r rT rS ) r rF r r$ r r# r r r r r rJ r6 r7 r8 r Zeval_losses_gathererrh Zpreds_gathererZlabels_gathererrl r r r9 r r: Z eval_losspredsr ra r\ r r r r% N s
zTrainer.prediction_loopc C sJ |dkr dS t r t||}n$t r.t|}n| jjdkrBt|}t|S )z
Gather value of `tensors` (tensor or list/tuple of nested tensors) and convert them to numpy before
concatenating them to `gathered`
Nr
r )rk rJ rj rx r r rC rH )r r; r6 r r r ri s
zTrainer._gather_and_numpifyNN) NNNNNNNNNrl N)N)N)N)N)T)NNN)N)NNr r NN)F)NF)N)NN)FN)NNr" )Nr0 )NNr" )N)rm )N)F) NNNNNNNNN)rd T)NNr" )er
__module__ __qualname____doc__trainer_pt_utilsr{ r| r} r~ r r r. r Modulera r r$ r r2 r rO r r r7 r
r r< Z Optimizerr ZLambdaLRr r r r r r rM r r r ZSamplerr r r( r, r. r1 r r5 r3 staticmethodr r= r4 rJ r_ r rk rj r r r r r r r r r rP rM r r r r r r r r r rr r r
r rL r r r rS r+ rN r& r r3 r0 r r r^ r r r% ri r r r r rz s P [ I. /+ %/
(*
K 8 a )
u& TT$4(1
: A= +
%&*[ 4 h*: zrz )rp r r r r r r r r r r collections.abcr pathlibr typingr r r r r r r
r Z tqdm.autor Z integrationsr r r r r r r r r r r r r r r Ztorch.distributedZ distributedr packagingr r Ztorch.utils.datar r r r Z torch.utils.data.distributedr Zhuggingface_hubr! r r" Zconfiguration_utilsr# Zdata.data_collatorr$ r% r& Z debug_utilsr' r( r r) r* r+ Zdependency_versions_checkr, Z modelcardr- Zmodeling_utilsr. r/ rE r0 r1 Ztokenization_utils_baser2 Ztrainer_callbackr3 r4 r5 r6 r7 r8 r9 rq r: r; r< r= r> r? r@ rA rB rC rD rE rF rG rH rI rJ rK Z
trainer_utilsrL rM rN rO rP rQ rR rS rT rU rV rW rX rY rZ r[ r\ r] r^ Z
training_argsr_ r` ra r rb rc rd re rf rg rh ri rj rk rl r r r r Zutils.notebookrm r rn r Ztorch.cuda.ampro r Ztorch_xla.core.xla_modelcoreZ xla_modelr Ztorch_xla.debug.metricsr ra r Z%torch_xla.distributed.parallel_loaderr r rp Zfairscale.nn.data_parallelrq r rr r~ Zfairscale.nn.wraprs Zfairscale.optimrt Zfairscale.optim.grad_scalerru Z!smdistributed.modelparallel.torchZ
modelparallelr rv rw rx ry rf
get_loggerr r r rt rw rx r rz r r r r s ( 8
$ PT4