/usr/local/lib/python3.6/site-packages/transformers/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/transformers/__pycache__/deepspeed.cpython-36.pyc (12887B)
3
EgI @ s d Z ddlZddlZddlZddlZddlmZ ddlm Z ddl
m Z ddl m
Z
mZ e
rfddlZejeZdd ZG d
d d ZG d d
d
eZdadd Zdd Zdd Zdd Zdd ZdddZ dS ) z
Integration with Deepspeed
N)deepcopy)
partialmethod )dep_version_check)is_torch_availableloggingc C s t jjdd k S )N deepspeed) importlibutil find_spec r r @/usr/local/lib/python3.6/site-packages/transformers/deepspeed.pyis_deepspeed_available# s r c @ s\ e Zd ZdZdd Zdd ZdddZdd
d Zd d
Zdd Z dd Z
dd Z dd Z dS )HfDeepSpeedConfigaJ
This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage.
A `weakref` of this object is stored in the module's globals to be able to access the config from areas where
things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore
it's important that this object remains alive while the program is still running.
[`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration
with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic
the DeepSpeed configuration is not modified in any way.
Args:
config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict.
c C s t | td t|tr$t|}n:t|trVtj|ddd}tj |}W d Q R X nt
d|| _ | j dd| _
d| _| j s| j rtd d
g}t| j d | j d g}t||@ d
krd| _d S )Nr rzutf-8)encodingzJexpecting either a path to a DeepSpeed config file or a pre-populated dictzzero_optimization.stager FcpuZnvmez*zero_optimization.offload_optimizer.devicez&zero_optimization.offload_param.devicer T)set_hf_deepspeed_configr
isinstancedictr strioopenjsonload
ValueErrorconfig get_value_stage_offloadis_zero2is_zero3setlen)selfconfig_file_or_dictr fZoffload_devices_validZoffload_devicesr r r
__init__8 s$
zHfDeepSpeedConfig.__init__c C sH | j }|jd}|j }x&|D ] }|j|}|d kr d |fS q W ||fS )N.)r splitpopget)r% ds_key_longr nodesds_keynoder r r
find_config_node[ s
z"HfDeepSpeedConfig.find_config_nodeNc C s& | j |\}}|dkr|S |j||S )zG
Returns the set value or `default` if no value is set
N)r1 r, )r% r- defaultr r/ r r r
r h s zHfDeepSpeedConfig.get_valueFc C sj | j }|jd}xB|D ]:}|}|j|}|dkr|rLtd| d| j qdS qW |dk rf|j| dS )z
Deletes a sub-section of the config file if it's found.
Unless `must_exist` is `True` the section doesn't have to exist.
r) Nz Can't find z entry in the config: )r r* r, r r+ )r% r- Z
must_existr r. r0 Z
parent_configr r r
del_config_sub_treeq s
z%HfDeepSpeedConfig.del_config_sub_treec C s | j |}|dkrdS t|S )z
Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very
specific question of whether the value is set to `True` (and it's not set to `False`` or isn't set).
NF)r bool)r% r- valuer r r
is_true s
zHfDeepSpeedConfig.is_truec C s | j |}|dkrdS t| S )z
Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very
specific question of whether the value is set to `False` (and it's not set to `True`` or isn't set).
NF)r r4 )r% r- r5 r r r
is_false s
zHfDeepSpeedConfig.is_falsec C s
| j dkS )N )r )r% r r r
r! s zHfDeepSpeedConfig.is_zero2c C s
| j dkS )N )r )r% r r r
r" s zHfDeepSpeedConfig.is_zero3c C s | j S )N)r )r% r r r
is_offload s z HfDeepSpeedConfig.is_offload)N)F)
__name__
__module__ __qualname____doc__r( r1 r r3 r6 r7 r! r" r: r r r r
r ' s #
r c sN e Zd ZdZ fddZdd Zddd Zeed
d Zd d
Z dd Z
Z S )HfTrainerDeepSpeedConfigz
The `HfTrainerDeepSpeedConfig` object is meant to be created during `TrainingArguments` object creation and has the
same lifespan as the latter.
c s t j| d | _g | _d S )N)superr( _dtype
mismatches)r% r& ) __class__r r
r( s z!HfTrainerDeepSpeedConfig.__init__c C s | j d krtd| j S )Nz8trainer_config_process() wasn't called yet to tell dtype)rA r )r% r r r
dtype s
z HfTrainerDeepSpeedConfig.dtypeNTc C s~ | j |\}}|dkrdS |j|dkr4|||< dS |s
r( rD rI r rS rW r^
__classcell__r r )rC r
r? s
5r? c C s t j| ad S )N)weakrefref _hf_deepspeed_config_weak_ref)Zhf_deepspeed_config_objr r r
r " s r c C s$ t d k r t d k r t j S dS d S )NF)rb r" r r r r
is_deepspeed_zero3_enabled* s
rc c C s" t d k rt d k rt jS d S d S )N)rb r r r r r
deepspeed_config1 s rd c s |j }d}d|kr"|jrDtdn"|j r4tjd j }d|d< fdd}d}d |krz|dkrl|}nj |d
}||fS ) zY
A convenience wrapper that deals with optimizer and lr scheduler configuration.
N optimizerz|--adafactor was passed, but also found `optimizer` configured in the DeepSpeed config. Only one optimizer can be configured.zDetected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)TZ zero_allow_untested_optimizerc s j | dS )N)r] re )create_scheduler)re )r] trainerr r
_lr_scheduler_callable` s z5deepspeed_optim_sched.._lr_scheduler_callableZ scheduler)r] re )r Z adafactorr r: loggerinfoZcreate_optimizerrf )rg hf_deepspeed_configrV r] r re rh lr_schedulerr )r] rg r
deepspeed_optim_sched8 s$ rm c C s( ddl }|jf | j\}}}}|||fS )a
this is a temp hack based on: https://github.com/microsoft/DeepSpeed/issues/1394#issuecomment-937405374 until
Deepspeed fixes a bug where it can't resume from a checkpoint after it did some stepping
https://github.com/microsoft/DeepSpeed/issues/1612
r N)r
initializedeepspeed_initialize_kwargs)rg r deepspeed_enginere _rl r r r
deepspeed_reinitn s rr Fc C s^ ddl }ddlm} | j}| j}|j}|j||| |j} |j|j |r~|j
s\t d|j d |j d d\}
} d} n(t
| |||\}
} ttdd |j } t|| | |
| d }
|jf |
\}}
}} |
| _|dk rTddl}t|j| d
}t|dkrDtjd | |j|d d d
\}}|dkrTt d| ntj| d ||
| fS )a
Init DeepSpeed, after updating the DeepSpeed configuration with any relevant Trainer's args.
If `resume_from_checkpoint` was passed then an attempt to resume from a previously saved checkpoint will be made.
Args:
trainer: Trainer object
num_training_steps: per single gpu
resume_from_checkpoint: path to a checkpoint if to resume from after normal DeepSpeedEngine load
inference: launch in inference mode (no optimizer and no lr scheduler)
Returns: model, optimizer, lr_scheduler
r N)ri zMZeRO inference only makes sense with ZeRO Stage 3 - please adjust your configre rl c S s | j S )N)Z
requires_grad)pr r r
s z deepspeed_init..)r\ model_parametersZ
config_paramsre rl z
/global_step*zAttempting to resume from T)Zload_optimizer_statesZload_lr_scheduler_statesz-[deepspeed] failed to resume from checkpoint z2 doesn't have deepspeed checkpoints, doing nothing)NN)r Zdeepspeed.utilsri r\ rV rk r^ r setLevelZget_process_log_levelr" r r3 rm listfilter
parametersr rn ro globsortedr$ rj Zload_checkpoint)rg r] Zresume_from_checkpointZ inferencer Z ds_loggerr\ rV rk r re rl ru kwargsrp rq rz Zdeepspeed_checkpoint_dirsZ load_pathr r r
deepspeed_initz sH
r} )NF) r> importlib.utilr r r r` copyr functoolsr Zdependency_versions_checkr utilsr r rU
get_loggerr; ri r r r? rb r rc rd rm rr r} r r r r
s*
||6