/usr/local/lib/python3.6/site-packages/transformers/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/transformers/__pycache__/modelcard.cpython-36.pyc (26165B)
3
EgΉ @ s d Z ddlZddlZddlZddlZddlmZ ddlmZ ddl m
Z
m Z m Z m
Z
mZ ddlZddlZddlmZ ddlmZ dd lmZmZmZmZmZmZm Z m Z m Z m Z m Z dd
l!m"Z" dd l#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0 eeeeee e e e e ed Z1e0j2e3Z4G d
d dZ5dZ6dZ7dddddddddddd d d d Z8d d!d"d#d$d%d&d'd(d)g
Z9d*d+ Z:d,d- Z;d.d/ Z
d4d5 Z?eG d6d7 d7Z@d8d9 ZAd:d; ZBd t jj|r t jj|t}n|}| j| tjd| dS )zKSave a model card object to the directory or file `save_directory_or_file`.zModel card saved in N)ospathisdirjoinr to_json_filer6 info)r8 Zsave_directory_or_fileZoutput_model_card_filer= r= r> save_pretraineds s
zModelCard.save_pretrainedc
K s ddl m} |jdd}|jdd}|jdd}|jdd }|jd
d}d d i} |dk r`|| d
< ||krr|| }
nDtjj|rtjj|t}
n(tjj|st |r|}
nt
|tdd}
|s||kr|
j t t}
|
j t
t}
|
j tt}
yPt|
||| d} | |
krtjd|
ntjd|
d| | j| } W n" ttjfk
rX | } Y nX g }
x8|j D ],\}}t| |rht| || |
j| qhW x|
D ]}|j|d qW tjd| |r| |fS | S dS )a
Instantiate a [`ModelCard`] from a pre-trained model model card.
Parameters:
pretrained_model_name_or_path: either:
- a string, the *model id* of a pretrained model card hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a
user or organization name, like `dbmdz/bert-base-german-cased`.
- a path to a *directory* containing a model card file saved using the [`~ModelCard.save_pretrained`]
method, e.g.: `./my_model_directory/`.
- a path or url to a saved model card JSON *file*, e.g.: `./my_model_directory/modelcard.json`.
cache_dir: (*optional*) string:
Path to a directory in which a downloaded pre-trained model card should be cached if the standard cache
should not be used.
kwargs: (*optional*) dict: key/value pairs with which to update the ModelCard object after loading.
- The values in kwargs of any keys which are model card attributes will be used to override the loaded
values.
- Behavior concerning key/value pairs whose keys are *not* model card attributes is controlled by the
*return_unused_kwargs* keyword parameter.
proxies: (*optional*) dict, default None:
A dictionary of proxy servers to use by protocol or endpoint, e.g.: {'http': 'foo.bar:3128',
'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
find_from_standard_name: (*optional*) boolean, default True:
If the pretrained_model_name_or_path ends with our standard model or config filenames, replace them
with our standard modelcard filename. Can be used to directly feed a model/config url and access the
colocated modelcard.
return_unused_kwargs: (*optional*) bool:
- If False, then this function returns just the final model card object.
- If True, then this functions returns a tuple *(model card, unused_kwargs)* where *unused_kwargs* is a
dictionary consisting of the key/value pairs whose keys are not model card attributes: ie the part of
kwargs which has not been used to update *ModelCard* and is otherwise ignored.
Examples:
```python
modelcard = ModelCard.from_pretrained(
"bert-base-uncased"
) # Download model card from huggingface.co and cache.
modelcard = ModelCard.from_pretrained(
"./test/saved_model/"
) # E.g. model card was saved using *save_pretrained('./test/saved_model/')*
modelcard = ModelCard.from_pretrained("./test/saved_model/modelcard.json")
modelcard = ModelCard.from_pretrained("bert-base-uncased", output_attentions=True, foo=False)
```r )!ALL_PRETRAINED_CONFIG_ARCHIVE_MAP cache_dirNproxiesfind_from_standard_nameTreturn_unused_kwargsFZ_from_pipeline file_type
model_cardZusing_pipeline)filenameZmirror)rH rI
user_agentzloading model card file z from cache at z Model card: )Z+transformers.models.auto.configuration_autorG r2 r@ rA rB rC r isfiler r replacer r r r r6 rE from_json_fileEnvironmentErrorjsonJSONDecodeErrorr3 hasattrr4 append)clsZ pretrained_model_name_or_pathr9 rG rH rI rJ rK Z
from_pipelinerO Zmodel_card_fileZresolved_model_card_fileZ modelcardZ to_remover: r; r= r= r> from_pretrained~ sP 7
zModelCard.from_pretrainedc C s
| f |S )z@Constructs a `ModelCard` from a Python dictionary of parameters.r= )rX Z json_objectr= r= r> from_dict s zModelCard.from_dictc
C s6 t |ddd}|j }W dQ R X tj|}| f |S )z8Constructs a `ModelCard` from a json file of parameters.rzutf-8)encodingN)openreadrT loads)rX Z json_filereadertextdict_objr= r= r> rR s
zModelCard.from_json_filec C s | j |j kS )N)__dict__)r8 otherr= r= r> __eq__ s zModelCard.__eq__c C s t | j S )N)strto_json_string)r8 r= r= r> __repr__ s zModelCard.__repr__c C s t j| j}|S )z0Serializes this instance to a Python dictionary.)copydeepcopyrc )r8 outputr= r= r> to_dict s zModelCard.to_dictc C s t j| j dddd S )z*Serializes this instance to a JSON string. T)indent sort_keys
)rT dumpsrl )r8 r= r= r> rg s zModelCard.to_json_stringc C s, t |ddd}|j| j W dQ R X dS )z"Save this instance to a json file.wzutf-8)r\ N)r] writerg )r8 Zjson_file_pathwriterr= r= r> rD s zModelCard.to_json_fileN)__name__
__module__ __qualname____doc__r? rF classmethodrY rZ rR re rh rl rg rD r= r= r= r> r% O s r r% z
z
zMasked Language ModelingzImage ClassificationzImage SegmentationzMultiple ChoicezObject DetectionzQuestion AnsweringZ
SummarizationzTable Question AnsweringzText ClassificationzCausal Language Modelingz&Sequence-to-sequence Language ModelingzToken ClassificationZ TranslationzZero Shot Classification)z fill-maskzimage-classificationzimage-segmentationzmultiple-choicezobject-detectionzquestion-answeringZ
summarizationztable-question-answeringztext-classificationztext-generationztext2text-generationztoken-classification translationzzero-shot-classificationZaccuracyZbleuf1Zmatthews_correlationZpearsonrZ precisionZrecallrougeZ sacrebleuZ spearmanrc C s$ | d kr g S t | tr | gS | S d S )N)
isinstancerf )objr= r= r> _listify= s
r c C sF |d kr | S t |tr |g}dd |D }t|dkr:| S || |< | S )Nc S s g | ]}|d k r|qS )Nr= ).0vr= r= r>
K s z*_insert_values_as_list..r )r} rf len)metadatanamevaluesr= r= r> _insert_values_as_listF s
r c C sd | d kr i S i }xN| j D ]B}|j jddtkrH|||j jdd< q|j dkr||d< qW |S )N _Zrouge1r| )keyslowerrQ METRIC_TAGS) eval_resultsresultr: r= r= r> #infer_metric_tags_from_eval_resultsR s r c C s |d kr | S || |< | S )Nr= )r r r; r= r= r>
_insert_value^ s r c C s t s
dS ddlm} t| |S )NFr )Dataset)r datasetsr r} )datasetr r= r= r>
is_hf_datasete s r c C s@ g }x6| j D ]*}t|ttfr.|t|7 }q|j| qW |S )N)r r} tuplelistrW )mappingr r r= r= r> _get_mapping_valuesn s r c
@ s: e Zd ZU edZeeeee f dZ ee dZ
eeeee f
dZ ee dZ eeeee f dZ
eeeee f
dZeeeee f dZeeeee f dZeeeef dZeee dZeeeef dZee dd Zdd Zdd Zd d
Zedd d Z edd
dZ dS )TrainingSummaryNtrainerc C s~ | j d krzt rz| jd k rzt| jdkrzy6t| j}x&|jD ] }|jdr>|dd | _ q>W W n tjj k
rx Y nX d S )Nr zlicense: )
licenser finetuned_fromr r tags
startswithrequests
exceptions HTTPError)r8 rE tagr= r= r>
__post_init__ s
z TrainingSummary.__post_init__c s d| j i}t| j}t| j}t| j}t|t|k rR|d gt|t| }dd t||D dd t||D }dd t| jD }g |d< t|dkrt dkr|gS t|dkrd d i}t dkrd d i fdd |D }x|D ]\} }
i } | d k r || | d
| d < |
d k rV |
|
d
| d < ||
d k rV||
| d d
< t|dkrg | d< x2|j D ]&\} }
| d j |
| | j
|
d qvW d | krd | krd| kr|d j | qt j d| qW |gS )Nr c S s i | ]\}}||qS r= r= )r r r r= r= r>
s z6TrainingSummary.create_model_index..c S s i | ]\}}||qS r= r= )r r argr= r= r> r s c S s i | ]}|t krt | |qS r= )TASK_TAG_TO_NAME_MAPPING)r taskr= r= r> r s resultsr c s g | ]} D ] }||fq qS r= r= )r task_tagds_tag)dataset_mappingr= r> r s z6TrainingSummary.create_model_index..)r typer r argsr) )r r r; zLDropping the following result as it does not have all the necessary fields:
)
model_namer r dataset_tags dataset_argsr ziptasksr3 rW r r6 rE )r8 metric_mappingZ model_indexZ
dataset_namesr r Zdataset_arg_mappingZ task_mappingZall_possibilitiesr r r Z
metric_tagZ metric_namer= )r r> create_model_index sJ
z"TrainingSummary.create_model_indexc C sl t | j}i }t|d| j}t|d| j}t|d| j}t|d| j}t|dt|j }| j
||d< |S )Nlanguager r r r) z model-index) r r r r r r r r r r r )r8 r r r= r= r> create_metadata s
z TrainingSummary.create_metadatac C s d}t j| j dd}t|dkr.d| d}| jdkrB|t7 }n|t7 }|d| j d7 }| jd krp|d 7 }n|d
| j d | j d 7 }| j d kr|d
7 }nt
| j t r|d| j d7 }njt
| j t t
frt| j dkr|d| j d d7 }n4|djdd | j d d) D d| j d* d 7 }| jd k rZ|d7 }|djdd | jj D 7 }|d7 }|d7 }|d7 }|d7 }|d 7 }|d 7 }| jd k r|d 7 }|djd d | jj D 7 }|d7 }n|d 7 }| jd k r|d!7 }|t| j7 }|d7 }|d"7 }|d#t d7 }| jdkr@t r@dd l}|d$|j d7 }n.| jd%krnt rndd l}|d&|j d7 }t rdd l}|d'|j d7 }t rdd l }|d(|j d7 }|S )+N F)ro r z---
r z
# z
z'This model was trained from scratch on z'This model is a fine-tuned version of [z](https://huggingface.co/z) on zan unknown dataset.zthe z dataset.r
z, c S s g | ]}d | qS )zthe r= )r Zdsr= r= r> r s z1TrainingSummary.to_model_card..z and the z
datasets.z:
It achieves the following results on the evaluation set:
rp c S s$ g | ] \}}d | dt | qS )z- z: ) _maybe_round)r r r; r= r= r> r s z/
## Model description
More information needed
z9
## Intended uses & limitations
More information needed
z:
## Training and evaluation data
More information needed
z
## Training procedure
z
### Training hyperparameters
z:
The following hyperparameters were used during training:
c S s g | ]\}}d | d| qS )z- z: r= )r r r; r= r= r> r
s z
More information needed
z
### Training results
z
### Framework versions
z- Transformers z
- Pytorch kerasz
- TensorFlow z - Datasets z
- Tokenizers r ) yamldumpr r source AUTOGENERATED_TRAINER_COMMENTAUTOGENERATED_KERAS_COMMENTr r r r} rf r r rC r r3 hyperparameters
eval_linesmake_markdown_tabler r# torchr!
tensorflowr r r"
tokenizers)r8 rM r r tfr r r= r= r>
to_model_card sh
2
z TrainingSummary.to_model_cardc C sp |j d k r|j n|j} t| rZ|d ks.|
d krZ| j} | d krZ|d krJ| g}|
d krZ| jg}
| d krn|d k rn|} |d krt|jjdrtj j
|jjj r|jjj }|d kr|jj j
}
x$tj D ]\}}|
t|kr|}qW |d krt|jjj}|d krdg}n4t|tr"|dkr"|dg}nd|kr6|jd t|jj\}}}t|}| |||||||| |
|||d S )
NcsvrT pandasparquetra
_name_or_pathZgenerated_from_trainer) r r r r r r r r r r r r )r rT r r ra ) Z
train_datasetZ eval_datasetr builder_name config_namerV modelconfigr@ rA rB r __class__ru TASK_MAPPINGr3 r r r
output_dirr r} rf rW parse_log_historystate log_history$extract_hyperparameters_from_trainer)rX r r r r r r r r r r Z one_dataset default_tagmodel_class_namer r r r r r r= r= r> from_trainer* sT
z TrainingSummary.from_trainerc C sR |
d k rLt |
rL| d ks | d krL|
j} | d
krL| d kr<| g} | d krL|
jg} |
d kr`| d k r`| }
|d krt|jdrtjj|jj r|jj}|d kr|j j
}
x$t j D ]\}}|
t
|kr|}qW |d krdg}n0t|tr|dkr|dg}nd|kr|jd |d k r t|\}}}n
g }t }t|}| ||||||| |
| |||dd
S ) Nr rT r r ra r Z generated_from_keras_callbackr )
r r r r r r r r r r r r r )r rT r r ra )r r r rV r r@ rA rB r r ru r r3 r r} rf rW parse_keras_historydict"extract_hyperparameters_from_keras)rX r r Z
keras_historyr r r r r r r r r r r r r r r r r= r= r>
from_kerasr sX
zTrainingSummary.from_keras) NNNNNNNNN) NNNNNNNNN) ru rv rw rf r r r r r r r r r r r r r r floatr r r r r r r r ry r r r= r= r= r> r x sJ
8
M < r c s t dr4t ds dg t fS jjd< jnfddd D g }xttd D ] fddj D }t }xf|j D ]Z\}}|jdrd |d
d }n|dkrd | }|jd }d
j dd |D }|||< qW |j
| q`W |d }||fS )z
Parse the `logs` of either a `tf.keras.History` object returned by `model.fit()` or an accumulated logs `dict`
passed to the `PushToHubCallback`. Returns lines and logs compatible with those returned by `parse_log_history`.
historyepochNc s i | ] fd dD qS )c s g | ] }| qS r= r= )r Z single_dict)log_keyr= r> r s z2parse_keras_history...r= )r )logs)r r> r s z'parse_keras_history..r c s i | ]\}}| |qS r= r= )r r Zlog_value_list)ir= r> r s Zval_Z validation_ Ztrain_r r c S s g | ] }|j qS r= )
capitalize)r partr= r= r> r s z'parse_keras_history..r
r ) rV r r r ranger r3 r splitrC rW ) r linesZ
epoch_dictr kr splitsr r r= )r r r> r s*
r c C s> d}x$|t | k r(d| | kr(|d7 }qW |t | krz|d8 }x |dkr^d| | kr^|d8 }q@W |dkrvdd| | fS dS | | }g }d}xt|D ]}d| | kr| | d }d| | kr| | j }|jdd}|jd d}|jd
d} |jd d}|jd d}|jd
d}||| d}
xX|j D ]L\} } | dkrB| |
d< n.| jd}
djdd |
dd D }| |
|< q&W |j|
qW t | d }x&|dkrd| | kr|d8 }qW |dkr0i }x^| | j D ]N\}}|jdr|dd }|dkrdjdd |jdD }|||< qW |||fS ||dfS dS ) zd
Parse the `log_history` of a Trainer to get the intermediate and final evaluation results.
r Z
train_runtimer
Z eval_lossNzNo logZlossZ
total_flosr stepZ eval_runtimeZeval_samples_per_secondZeval_steps_per_second)z
Training LossZEpochZStepzValidation Lossr r c S s g | ] }|j qS r= )r )r r r= r= r> r s z%parse_log_history..Zeval_ runtimesamples_per_secondsteps_per_secondc S s g | ] }|j qS r= )r )r r r= r= r> r s )NNN)r r r r r ) r r ri r2 r3 r rC rW r )r idxZ train_logr Z
training_lossr r) r r r r r r r r r r: r; Zcamel_cased_keyr= r= r> r sZ
r c C sP dd l }t }t| dr2| jd k r2| jj |d< nd |d< |jjj j|d< |S )Nr optimizerZtraining_precision) r r rV r Z
get_configr Zmixed_precisionZ
global_policyr )r r r r= r= r> r s r r c C sR t | trJtt| jddkrJtt| jdd |krJ| d| dS t| S )N.r
f)r} r r rf r )r Zdecimalsr= r= r> r + s :r c C s" dd t | |D }dj|d S )Nc S s. g | ]&\}}d | d|t | d qS )z| r r
)r )r r rr r= r= r> r 2 s z'_regular_table_line..r z|
)r rC )r
col_widthsZvalues_with_spacer= r= r> _regular_table_line1 s r c C s dd | D }dj |d S )Nc S s g | ]}d d| d qS )z|:-:r= )r rr r= r= r> r 7 s z&_second_table_line..r z|
)rC )r r r= r= r> _second_table_line6 s r c C s | dkst | dkrdS dd | d j D }xF| D ]>}x8|j D ],\}}|| t t|k rBt t|||< qBW q4W tt| d j t|j }|tt|j 7 }x0| D ](}|tdd |j D t|j 7 }qW |S )zC
Create a nice Markdown table from the results in `lines`.
Nr r c S s i | ]}t t||qS r= )r rf )r r: r= r= r> r A s z'make_markdown_table..c S s g | ] }t |qS r= )r )r r r= r= r> r J s z'make_markdown_table..)r r r3 r r r r r )r r liner: r; tabler= r= r> r ; s
(r Z
learning_ratetrain_batch_sizeeval_batch_sizeseedc s fddt D } jjtjtjgkrF jjtjkr8dn jjj|d< jjdkr^ jj|d< jj dkrv jj |d< jj
jj jj }||d kr||d < jj jj }||d
kr||d < jj rd |d
< n&d jj
d jj d jj |d
< jjj|d< jjdkr& jj|d< jjdkr@ jj|d< jjdkr\ jj|d< n jj|d< jjr jrd|d< n jrd jj |d< jjdkr jj|d< |S ) Nc s i | ]}t j||qS r= )getattrr )r r )r r= r> r W s z8extract_hyperparameters_from_trainer..z multi-GPUZdistributed_typer
Z num_devicesgradient_accumulation_stepsr total_train_batch_sizer total_eval_batch_sizeZ Adafactorr zAdam with betas=(,z) and epsilon=lr_scheduler_typeg Zlr_scheduler_warmup_ratioZlr_scheduler_warmup_stepsZtraining_stepsZ
num_epochsz
Native AMPZmixed_precision_trainingzApex, opt level label_smoothing_factorr )_TRAINING_ARGS_KEYSr Z
parallel_moder Z NOT_PARALLELZNOT_DISTRIBUTEDZ DISTRIBUTEDr; Z
world_sizer r r Z adafactorZ
adam_beta1Z
adam_beta2Z adam_epsilonr Z warmup_ratioZ warmup_stepsZ max_stepsZnum_train_epochsZfp16Zuse_ampZuse_apexZfp16_opt_levelr )r r r r r= )r r> r V s@
&
r )r )Jrx ri rT r@ r/ Z dataclassesr pathlibr typingr r r r r r r Zhuggingface_hubr r r Zmodels.auto.modeling_autor r
r r r r r r r r r Z
training_argsr utilsr r r r r r r r r r! r" r# r$ r
get_loggerru r6 r% r r r r r r r r r r r r r r r r r r r r r= r= r= r> s 4
<
G
H#=