/usr/local/lib/python3.6/site-packages/transformers/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/transformers/__pycache__/integrations.cpython-36.pyc (31404B)
3
Eg @ sv d Z ddlZddlZddlZddlZddlZddlZddlm Z ddl
m Z m Z e j
eZejjddk o|ejddj d kZeryJddlZeed
rejjd rd Zn"ejddj d krejd
dZW n eefk
r dZY nX ddl m Z m Z ddl m Z m!Z!m"Z" ddl
m#Z#m$Z$ dd Z%dd Z&dd Z'dd Z(dd Z)d d Z*d d Z+d d! Z,d"d# Z-d$d% Z.d&d' Z/d(d) Z0d*d+ Z1d,d- Z2e3e4e!d.d/d0Z5e3e4e!d.d1d2Z6e3e4e!d.d3d4Z7e3e4e!d.d5d6Z8d7d8 Z9d9d: Z:G d;d< d
d>e ZG dCdD dDe Z?G dEdF dFe Z@G dGdH dHe ZAe>e=e?e@e;e._objectivetimeoutn_jobsr r< )r; rN rO )N)
r" popZ create_studyoptimize
best_trialr
strnumbervaluer4 )
rL r; r< kwargsr" rM rN rO ZstudyrR r )rL r run_hp_search_optuna s rW c sR dd l d$ fdd }| jjs@ddlm} tjd |dd| _| jt}d | _ d |krd
di|d < | j
j dkr|d|d d < d | j
j dkrd
nd }tj d| d |d j
d d}|| j
_d|krddlm} | dgd|d< d|ko|d dkr"d| _|d dkr"tjd|d d d|krddlm}
m} m} m}
t|d |
rf| jsftjd t|d |
| | |
fr| j
j s| j
jtjkrt dj t |d j d j j!|| d t"j#fd d }t$d rj%|_% j j&|f| j'd |d!|}|j(d|d d" d#}t)|j*|j+d |j,}|d k rN| j-| |S )%Nr c s y$ddl m} |j|r"|jt W n tk
r8 Y nX d }|rnx*tj|D ] }|jt rNtj
j ||}qNW d |_ |j
|| d t|dd d kr|j }|j||_ |j jjf d|j i|ddi d S )Nr )NotebookProgressCallback)r> r8 r? doneT)Ztransformers.utils.notebookrX pop_callback add_callbackr
ModuleNotFoundErrorr r@ rA r rB rC r? rD rE rF rG Z_tune_save_checkpointtunereport)r8
local_trainerrH rX rI rJ rK )r$ r r rM s$
z%run_hp_search_ray.._objectiver )TrainerMemoryTrackerzMemory tracking for your Trainer is currently enabled. Automatically disabling the memory tracker since the memory tracker is not serializable.T)skip_memory_metricsZresources_per_trialcpuZgpuz1 CPUz
and 1 GPUr zgNo `resources_per_trial` arg was passed into `hyperparameter_search`. Setting it to a default value of z for each trial.Zprogress_reporter) CLIReporterr? )Zmetric_columnsZkeep_checkpoints_numzCurrently keeping ze checkpoints for each trial. Checkpoints are usually huge, consider setting `keep_checkpoints_num=1`.Z scheduler)
ASHASchedulerHyperBandForBOHBMedianStoppingRulePopulationBasedTraininga/ You are using PopulationBasedTraining but you haven't enabled checkpointing. This means your trials will train from scratch everytime they are exploiting new configurations. Consider enabling checkpointing by passing `keep_checkpoints_num=1` as an additional argument to `Trainer.hyperparameter_search`.a You are using {cls} as a scheduler but you haven't enabled evaluation during training. This means your trials will not report intermediate results to Ray Tune, and can thus not be stopped early or used to exploit other trials parameters. If this is what you want, do not use {cls}. If you would like to use {cls}, make sure you pass `do_eval=True` and `evaluation_strategy='steps'` in the Trainer `args`.)cls)r_ c s^ t rTddl}tjj|jj d}tjj d|}tjj
|}|t j |j
< |jj| | |S )a"
Wrapper around `tune.with_parameters` to ensure datasets_modules are loaded on each Actor.
Without this, an ImportError will be thrown. See https://github.com/huggingface/transformers/issues/11565.
Assumes that `_objective`, defined above, is a function.
r Nz __init__.pydatasets_modules)r Z
datasets.loadr rB rC loadZinit_dynamic_modulesr r spec_from_file_locationmodule_from_specsysmodulesnameloader exec_module)argsrV ZdatasetsZdynamic_modules_pathspecri ) trainabler r dynamic_modules_import_trainable
s z;run_hp_search_ray..dynamic_modules_import_trainable
__mixins__)r Z num_samples )metricmode)N).r$ Z_memory_trackerra
trainer_utilsr` r r rZ TensorBoardCallbackmodelrr Zn_gpuinfogetZ_n_gpuZray.tunerc Zuse_tune_checkpointsZray.tune.schedulersrd re rf rg r3 Zdo_evalZevaluation_strategyr NOr6 formattype__name__r] Zwith_parameters functoolswrapshasattrrv runhp_spaceZget_best_trialr
Ztrial_idZ last_resultr r[ )rL r; r< rV rM r` Z
_tb_writerZ resource_msgZgpus_per_trialrc rd re rf rg ru ZanalysisrR best_runr )r$ rt r run_hp_search_ray sh
r c K s^ ddl m} | }|jdd }|d k r0|j| |j jd| jd td|ddgd|d d
}tj d |j
x|j j |j
k r&|j|j
j j }d | _| jd |d t| dd d kr| j } | j| | _td| jd
g}
|j|j
j j|j
|
d} tj d|j
d| j
d |j|j
j }qpW t|j|j
j j j d } t| j
| j| j}
|
S )Nr )
Connectionproxieszhuggingface-tuner? rQ )ro r? Zstrategyr huggingface)ro
parametersrK Zparallel_bandwidthobservation_budgetprojectz6created experiment: https://app.sigopt.com/experiment/)r> r8 )ro rU )
suggestionvaluesz"[suggestion_id, observation_id]: [z, ]) r' r rP Z set_proxiesZ experimentscreater r5 r r} idprogressZobservation_countr Z suggestionsr? rD rE rF rG Z observationsfetchlistZbest_assignmentsZ
iterate_pagesr
rU assignments)rL r; r< rV r connr
experimentr rK r Zobsbestr r r r run_hp_search_sigopt2 s6
r c
s< ddl m} | stddd ld}x jjD ]}t|tr0d}P q0W |sXjt dj _
d d d d |j d d }|j d
d }|j d d } |j d d }
|j d
dj d } | d
d< | d
d < | r| | d < fdd} |s j
| ||
dn|}tjd| j|| |d t d d d S )Nr )r z8This function needs wandb installed: `pip install wandb`r FTr )run_idr? hyperparameterssweep_idr ro entityrx z eval/lossZgoalc s j r j nj } | jj_| jji d j}d _jd t |d d t
dd d krj }j |_t
|}|krtjd d|j d} d d k rވd krȈj d k }nd
krވj d k}|s d d kr| j d< j d< t| d < jS ) N)r rx Z_items)r> r8 r? zProvided metric zU not found. This might result in unexpected sweeps charts. The available metrics are Fr ZminimizeZmaximizer )r initro state
trial_namer updater? rD varsrE rF rG rewrite_logsr r keysr r5 )r r rK Zformat_metricsZ
best_score)rR r< rx rL r r r rM v s0
z'run_hp_search_wandb.._objective)r r zwandb sweep id - )functioncountr r? r ) integrationsr ImportErrorr Zcallback_handler callbacksr3
WandbCallbackr[ rr report_torP r Zsweepr r} Zagentr
)
rL r; r< rV r Zreporting_to_wandbcallbackr r ro r Z sweep_configrM r )rR r< rx rL r r run_hp_search_wandbY s8
!r c C sh g } t r| jd t r$| jd t r4| jd t rD| jd t rT| jd t rd| jd | S )Nazure_mlr r, r r r1 )r) appendr r- r! r r2 )r r r r $get_available_reporting_integrations s
r c C s i }d}t |}d}t |}x`| j D ]T\}}|j|rN||d||d < q&|j|rn||d||d < q&||d| < q&W |S )NZeval_Ztest_zeval/ztest/ztrain/)lenitemsrA )dZnew_dZ eval_prefixZeval_prefix_lenZ test_prefixZtest_prefix_lenkvr r r r s
r c @ s> e Zd ZdZd
ddZdddZdd Zdd d
Zd d ZdS )r{ z
A [`TrainerCallback`] that sends the logs to [TensorBoard](https://www.tensorflow.org/tensorboard).
Args:
tb_writer (`SummaryWriter`, *optional*):
The writer to use. Will instantiate one if not set.
Nc C s t }|std|rvyddlm} || _W q| tk
rr yddlm} || _W n tk
rl d | _Y nX Y q|X nd | _|| _d S )NzuTensorBoardCallback requires tensorboard to be installed. Either update your PyTorch version or install tensorboardX.r )
SummaryWriter)r! r6 Ztorch.utils.tensorboardr _SummaryWriterr r tb_writer)selfr Zhas_tensorboardr r r r __init__ s
z TensorBoardCallback.__init__c C s& |p|j }| jd k r"| j|d| _d S )N)log_dir) logging_dirr r )r rr r r r r _init_summary_writer s
z(TensorBoardCallback._init_summary_writerc K s |j s
d S d }|jr2|j}|d k r2tjj|j|}| jd krH| j|| | jd k r| jj d|j
d|kr|d }t |dr|j d k r|j j
}| jj d| t | jdr| jj
|j i d d S )Nrr r| r model_config add_hparams)Z metric_dict)is_world_process_zerois_hyper_param_searchr r rB rC r r r add_textZto_json_stringr r r to_sanitized_dict) r rr r controlrV r r r| Zmodel_config_jsonr r r on_train_begin s$
z"TensorBoardCallback.on_train_beginc K s |j s
d S | jd kr | j| | jd k rt|}xX|j D ]L\}}t|ttfrd| jj|||j q:t
j d| dt | d| d q:W | jj
d S )Nz)Trainer is attempting to log a value of "z
" of type z
for key "zn" as a scalar. This invocation of Tensorboard's writer.add_scalar() is incorrect so we dropped this attribute.)r r r r r r3 intfloatZ
add_scalar global_stepr r r flush)r rr r r logsrV r r r r r on_log s
"zTensorBoardCallback.on_logc K s | j r| j j d | _ d S )N)r close)r rr r r rV r r r on_train_end s
z TensorBoardCallback.on_train_end)N)N)N) r
__module__ __qualname____doc__r r r r r r r r r r{ s
r{ c @ s> e Zd ZdZdd Zdd Zd
ddZdd d
Zdd d ZdS )r zc
A [`TrainerCallback`] that sends the logs to [Weight and Biases](https://www.wandb.com/).
c C sL t }|std|r$dd l}|| _d| _tjddj tj dhk| _
d S )NzFWandbCallback requires wandb to be installed. Run `pip install wandb`.r FZWANDB_LOG_MODELr* r+ ) r r6 r _wandb _initializedr r r r union
_log_model)r Z has_wandbr r r r r s zWandbCallback.__init__c
K s& | j dkrdS d| _|jr"tjd |j }t|drV|jdk rV|jj }||}|j }i }|dk rx|} |j
|d< n|j
} | j j dkr| j j f t
jdd| d| | j jj|dd t| j d
dr| j jd | j jd d dd
t ot
jddkr"| j j|t
jddtd|jd dS )az
Setup the optional Weights & Biases (*wandb*) integration.
One can subclass and override this method to customize the setup if needed. Find more information
[here](https://docs.wandb.ai/integrations/huggingface). You can also override the following environment
variables:
Environment:
WANDB_LOG_MODEL (`bool`, *optional*, defaults to `False`):
Whether or not to log model as artifact at the end of training. Use along with
*TrainingArguments.load_best_model_at_end* to upload best model.
WANDB_WATCH (`str`, *optional* defaults to `"gradients"`):
Can be `"gradients"`, `"all"` or `"false"`. Set to `"false"` to disable gradient logging or `"all"` to
log gradients and parameters.
WANDB_PROJECT (`str`, *optional*, defaults to `"huggingface"`):
Set this to a custom string to store results in a different project.
WANDB_DISABLED (`bool`, *optional*, defaults to `False`):
Whether or not to disable wandb entirely. Set *WANDB_DISABLED=true* to disable.
NTz`Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"r groupZ
WANDB_PROJECTr )r ro )Zallow_val_change
define_metricztrain/global_step*)Z step_metricZ step_syncZ WANDB_WATCHfalseZ gradientsd )logZlog_freq)r r r r r} r r r to_dictr run_namer r r r r rE r r ZwatchmaxZ
logging_steps)
r rr r r| rV
combined_dictr r Z init_argsr r r r setup( s8
zWandbCallback.setupNc K sJ | j d krd S |j}|r.| j j d| _d |_| jsF| j|||f| d S )NF)r r finishr r r )r rr r r r| rV Z hp_searchr r r r d s
z WandbCallback.on_train_beginc K s | j d krd S | jo | jo |jrddlm} ||||d}tj } |j| |j spdd t
| j j j D nd|j
|jd|ji}
| j jd| j jj d |
d
} xHt| jd D ]6} | j r| j| jd d
}
|
j| j W d Q R X qW | j jj| W d Q R X d S )Nr )Trainer)rr r| tokenizerc S s. i | ]&\}}t |tjr|jd r||qS )_)r3 numbersNumberrA ).0r r r r r
y s z.WandbCallback.on_train_end..zeval/ztrain/total_flosszmodel-r| )ro r metadatar wb)ry )r r r r rL r tempfileTemporaryDirectoryZ
save_modelZload_best_model_at_endr5 summaryr Zmetric_for_best_modelZ best_metricZ
total_flosZArtifactr r r globis_fileZnew_filero write
read_bytesZ log_artifact)r rr r r r| r rV r Z fake_trainertemp_dirr artifactffar r r r o s
zWandbCallback.on_train_endc K sJ | j d krd S | js"| j||| |jrFt|}| j j|d|ji d S )Nztrain/global_step)r r r r r r r )r rr r r r| r rV r r r r s
zWandbCallback.on_log)N)NN)NN) r r r r r r r r r r r r r r s <
r c @ s< e Zd ZdZdd Zdd Zd
ddZdd d
Zd d ZdS )
CometCallbackz^
A [`TrainerCallback`] that sends the logs to [Comet ML](https://www.comet.ml/site/).
c C s t s tdd| _d| _d S )NzLCometCallback requires comet-ml to be installed. Run `pip install comet-ml`.F)r r6 r _log_assets)r r r r r s zCometCallback.__init__c C s d| _ tjddj }|dkr$d| _|jrtjddj }d}d tjd
d i}|dkrztjf |}|jd d
t j
d n:|dkrtjdd|d< tj f |}|jd d
t j
d |dk r|j |d
d |j
|dd
d t|dr|j
|jdd
d dS )a
Setup the optional Comet.ml integration.
Environment:
COMET_MODE (`str`, *optional*):
Whether to create an online, offline experiment or disable Comet logging. Can be "OFFLINE", "ONLINE",
or "DISABLED". Defaults to "ONLINE".
COMET_PROJECT_NAME (`str`, *optional*):
Comet project name for experiments
COMET_OFFLINE_DIRECTORY (`str`, *optional*):
Folder to use for saving offline experiments when `COMET_MODE` is "OFFLINE"
COMET_LOG_ASSETS (`str`, *optional*):
Whether or not to log training assets (tf event logs, checkpoints, etc), to Comet. Can be "TRUE", or
"FALSE". Defaults to "TRUE".
For a number of configurable items in the environment, see
[here](https://www.comet.ml/docs/python-sdk/advanced/#comet-configuration-variables).
TZCOMET_LOG_ASSETSr* r+ 1r ZONLINEN project_nameZCOMET_PROJECT_NAMEr z Created from transformersz)Automatic Comet.ml online logging enabledZOFFLINEZCOMET_OFFLINE_DIRECTORYz./Zoffline_directoryzLAutomatic Comet.ml offline logging enabled; use `comet upload` when finished) frameworkzargs/)prefixr r zconfig/> r+ r )r r r r r r r Z
ExperimentZ log_otherr r} ZOfflineExperimentZ_set_model_graphZ_log_parametersr r )r rr r r| Z
log_assetsZ
comet_moder Zexperiment_kwargsr r r r s,
zCometCallback.setupNc K s | j s| j||| d S )N)r r )r rr r r r| rV r r r r s z CometCallback.on_train_beginc K sF | j s| j||| |jrBtjj }|d k rB|j||j|jdd d S )Nr )stepepochr ) r r r r r get_global_experimentZ _log_metricsr r )r rr r r r| r rV r r r r r s
zCometCallback.on_logc K sT | j rP|jrPtjj }|d k rH| jdkrHtjd |j|j dd|j
d |j d S )NTz(Logging checkpoints. This may take time.) recursiveZ
log_file_namer ) r r r r r r r r} Zlog_asset_folder
output_dirr end)r rr r r rV r r r r r s
zCometCallback.on_train_end)N)NN) r r r r r r r r r r r r r r s *
r c @ s, e Zd ZdZd ddZdd Zd
ddZdS ) AzureMLCallbackzh
A [`TrainerCallback`] that sends the logs to [AzureML](https://pypi.org/project/azureml-sdk/).
Nc C s t std|| _d S )NzPAzureMLCallback requires azureml to be installed. Run `pip install azureml-sdk`.)r) r6 azureml_run)r r r r r r s zAzureMLCallback.__init__c K s* ddl m} | jd kr&|jr&|j | _d S )Nr )Run)Zazureml.core.runr r r get_context)r rr r r rV r r r r on_init_end s zAzureMLCallback.on_init_endc K sF | j rB|jrBx4|j D ](\}}t|ttfr| j j|||d qW d S )N) description)r r r r3 r r r )r rr r r r rV r r r r r r s zAzureMLCallback.on_log)N)N)r r r r r r r r r r r r s
r c @ sD e Zd ZdZdd Zdd ZdddZdd d
Zd d Zd
d Z dS )MLflowCallbackz
A [`TrainerCallback`] that sends the logs to [MLflow](https://www.mlflow.org/). Can be disabled by setting
environment variable `DISABLE_MLFLOW_INTEGRATION = TRUE`.
c C sD t stddd l}|jjj| _|jjj| _d| _ d| _
|| _ d S )NzIMLflowCallback requires mlflow to be installed. Run `pip install mlflow`.r F) r- r6 r, utilsZ
validationZMAX_PARAM_VAL_LENGTH_MAX_PARAM_VAL_LENGTHZMAX_PARAMS_TAGS_PER_BATCH_MAX_PARAMS_TAGS_PER_BATCHr _log_artifacts_ml_flow)r r, r r r r s zMLflowCallback.__init__c C s
t jddj }|d
kr d| _|jr | jjdkrB| jj|jd |j }t
|drp|j dk rp|j j }||}xHt |j
D ]8\}}tt|| jkr~tjd | d
| d ||= q~W t |j
} x8td t| | jD ]"}
| jjt| |
|
| j qW d| _dS )a$
Setup the optional MLflow integration.
Environment:
HF_MLFLOW_LOG_ARTIFACTS (`str`, *optional*):
Whether to use MLflow .log_artifact() facility to log artifacts.
This only makes sense if logging to a remote server, e.g. s3 or GCS. If set to `True` or *1*, will copy
whatever is in [`TrainingArguments`]'s `output_dir` to the local or remote artifact storage. Using it
without a remote storage will just copy the files to your artifact location.
ZHF_MLFLOW_LOG_ARTIFACTSr* r+ r TN)r r z)Trainer is attempting to log a value of "z " for key "zv" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.r > r+ r )r r r r r r
active_runZ start_runr r r r r r r rS r
r r ranger Z
log_paramsr5 r ) r rr r r|
log_artifactsr r ro rU Zcombined_dict_itemsir r r r
s&
"zMLflowCallback.setupNc K s | j s| j||| d S )N)r r )r rr r r r| rV r r r r 1 s z MLflowCallback.on_train_beginc
K s | j s| j||| |jri }xN|j D ]B\}} t| ttfrH| ||< q(tjd| dt | d| d q(W | j
j ||j d d S )Nz)Trainer is attempting to log a value of "z
" of type z
for key "zc" as a metric. MLflow's log_metric() only accepts float and int types so we dropped this attribute.)rK r )
r r r r r3 r r r r r r Z log_metricsr )
r rr r r r r| rV rK r r r r r r 5 s
"zMLflowCallback.on_logc K s. | j r*|jr*| jr*tjd | jj|j d S )Nz&Logging artifacts. This may take time.)r r r r r} r r r )r rr r r rV r r r r F s
zMLflowCallback.on_train_endc C s | j jd k r| j j d S )N)r r Zend_run)r r r r __del__L s zMLflowCallback.__del__)N)N)
r r r r r r r r r r r r r r r s '
r c @ s< e Zd ZdZdd Zdd Zd
ddZdd d
Zd d ZdS )NeptuneCallbackzU
A [`TrainerCallback`] that sends the logs to [Neptune](https://neptune.ai).
c C s. t stddd lj}|| _d| _d| _d S )NzZNeptuneCallback requires neptune-client to be installed. Run `pip install neptune-client`.r F)r0
ValueErrorZ neptune.newnew_neptuner r )r r/ r r r r X s
zNeptuneCallback.__init__c C s |j rz| jjtjdtjdtjddtjddtjddd| _|j }t|d rp|jdk rp|jj }||}|| jd
< d | _ dS ) a(
Setup the Neptune integration.
Environment:
NEPTUNE_PROJECT (`str`, *required*):
The project ID for neptune.ai account. Should be in format *workspace_name/project_name*
NEPTUNE_API_TOKEN (`str`, *required*):
API-token for neptune.ai account
NEPTUNE_CONNECTION_MODE (`str`, *optional*):
Neptune connection mode. *async* by default
NEPTUNE_RUN_NAME (`str`, *optional*):
The name of run process on Neptune dashboard
ZNEPTUNE_PROJECTZNEPTUNE_API_TOKENZNEPTUNE_CONNECTION_MODEasyncZNEPTUNE_RUN_NAMENZNEPTUNE_RUN_ID)r Z api_tokenry ro r r r T)
r r r r r _neptune_runr r r r )r rr r r| r r r r r r c s
zNeptuneCallback.setupNc K s | j s| j||| d S )N)r r )r rr r r r| rV r r r r s z NeptuneCallback.on_train_beginc K sJ | j s| j||| |jrFx*|j D ] \}}| j| j||jd q$W d S )N)r )r r r r r r r ) r rr r r r r| rV r r r r r r s
zNeptuneCallback.on_logc C sF y,t jd}|rt|nd}| jj|d W n tk
r@ Y nX dS )a
Environment:
NEPTUNE_STOP_TIMEOUT (`int`, *optional*):
Number of seconsds to wait for all Neptune.ai tracking calls to finish, before stopping the tracked
run. If not set it will wait for all tracking calls to finish.
ZNEPTUNE_STOP_TIMEOUTN)seconds)r r r r stopAttributeError)r Z stop_timeoutr r r r s
zNeptuneCallback.__del__)N)N) r r r r r r r r r r r r r r S s
r c @ s2 e Zd ZdZdd Zdd Zd ddZd d
ZdS ) CodeCarbonCallbackzI
A [`TrainerCallback`] that tracks the CO2 emission of training.
c C s& t stddd l}|| _d | _d S )NzWCodeCarbonCallback requires `codecarbon` to be installed. Run `pip install codecarbon`.r )r2 r6 r1 _codecarbontracker)r r1 r r r r s zCodeCarbonCallback.__init__c K s& | j d kr"|jr"| jj|jd| _ d S )N)r )r! is_local_process_zeror ZEmissionsTrackerr )r rr r r rV r r r r s z CodeCarbonCallback.on_init_endNc K s | j r|jr| j j d S )N)r! r" start)r rr r r r| rV r r r r s z!CodeCarbonCallback.on_train_beginc K s | j r|jr| j j d S )N)r! r" r )r rr r r rV r r r r s z CodeCarbonCallback.on_train_end)N)r r r r r r r r r r r r r s
r )r r r, r/ r r r1 c C sB x2| D ]*}|t krt| ddjt j dqW dd | D S )Nz is not supported, only z, z are supported.c S s g | ] }t | qS r )INTEGRATION_TO_CALLBACK)r integrationr r r
s z7get_reporting_integration_callbacks..)r$ r rC r )r r% r r r #get_reporting_integration_callbacks s
r' )Dr r importlib.utilr r r rm r pathlibr r r r
get_loggerr r r r r r r r r r Z
get_configr r r Ztrainer_callbackr
r rz r r
r r r r r r! r# r% r& r( r) r- r. r0 r2 r9 r: r rS rW r r r r r r{ r r r r r r r$ r' r r r r sv
$
'EX~L[G