/usr/local/lib/python3.6/site-packages/transformers/data/datasets/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/transformers/data/datasets/__pycache__/squad.cpython-36.pyc (6058B)
3
EgJ# @ s d dl Z d dlZd dlmZmZ d dlmZ d dlmZm Z m
Z
m Z d dl Z d dl
mZ d dlmZ ddlmZ dd lmZ dd
lmZ d d lmZmZmZmZ ej e Z e ej Z!e"d
d e!D Z#eG dd dZ$G dd deZ%G dd deZ&dS ) N) dataclassfield)Enum)DictListOptionalUnion)Dataset)FileLock )$MODEL_FOR_QUESTION_ANSWERING_MAPPING)PreTrainedTokenizer)logging )
SquadFeaturesSquadV1ProcessorSquadV2Processor"squad_convert_examples_to_featuresc c s | ] }|j V qd S )N)
model_type).0conf r J/usr/local/lib/python3.6/site-packages/transformers/data/datasets/squad.py
# s r c @ s e Zd ZU dZeddddje idZeedddidZ e eddd idZ
e
eddd
idZ e ed dd idZ
e
ed
ddidZe edddidZeedddidZeedddidZeedddidZe edddidZe edddidZe dS )SquadDataTrainingArgumentszb
Arguments pertaining to what data we are going to input our model for training and eval.
Nhelpz!Model type selected in the list: z, )defaultmetadatazFThe input data dir. Should contain the .json files for the SQuAD task. zThe maximum total input sequence length after tokenization. Sequences longer than this will be truncated, sequences shorter will be padded.zVWhen splitting up a long document into chunks, how much stride to take between chunks.@ zkThe maximum number of tokens for the question. Questions longer than this will be truncated to this length. zThe maximum length of an answer that can be generated. This is needed because the start and end predictions are not conditioned on one another.Fz1Overwrite the cached training and evaluation setszDIf true, the SQuAD examples contain some that do not have an answer.g zIIf null_score - best_non_null is greater than the threshold predict null. r zjlanguage id of input for language-specific xlm models (see tokenization_xlm.PRETRAINED_INIT_CONFIGURATION) z3multiple threads for converting example to features)__name__
__module__ __qualname____doc__r join MODEL_TYPESr strdata_dirmax_seq_lengthint
doc_stridemax_query_lengthZmax_answer_lengthoverwrite_cacheboolversion_2_with_negativeZnull_score_diff_thresholdfloatZ n_best_sizelang_idthreadsr r r r r & s:
r c @ s e Zd ZdZdZdS )SplittraindevN)r# r$ r% r6 r7 r r r r r5 ` s r5 c @ s e Zd ZU dZeee e
e de j
dddfeeee eee f ee ee ee dddZdd Zeeejf d
d d ZdS )
SquadDatasetzH
This will be superseded by a framework-agnostic approach soon.
NFpt)args tokenizer limit_lengthmodeis_language_sensitive cache_dirdataset_formatc C s || _ || _|jrt nt | _t|trTy t| }W n t k
rR t dY nX || _
|jrddnd}t j j
|d k rz|n|jd|j d|jj d|j d| } | d }
t|
F t j j| o|j rVtj } tj| | _| jd | _| jjdd | _ | jjd d | _ t j d
| d tj | | j d ksB| j d krt j d | d
n|tj!krt| jj"|j| _ n| jj#|j| _ t$| j ||j|j%|j&|tj'k|j(|d\| _| _ tj } tj)| j| j | j d| t j d| dtj | dd W d Q R X d S )Nz mode is not a valid split namev2v1Zcached__z.lockfeaturesdatasetexamplesz"Loading features from cached file z [took %.3f s]zDeleting cached file z; will allow dataset and examples to be cached in future run)rF r; r+ r- r. Z is_trainingr4 Zreturn_dataset)rD rE rF z!Saving features into cached file z [took z.3fz s])*r: r> r1 r r processor
isinstancer) r5 KeyErrorr= ospathr' r* value __class__r# r+ r
existsr/ timetorchloadZ old_featuresrD getrE rF loggerinfowarningr7 Zget_dev_examplesZget_train_examplesr r- r. r6 r4 save) selfr: r; r< r= r> r? r@ Z version_tagZcached_features_fileZ lock_pathstartr r r __init__o sX
& zSquadDataset.__init__c C s
t | jS )N)lenrD )rW r r r __len__ s zSquadDataset.__len__)returnc C s6 | j | }tj|jtjd}tj|jtjd}tj|jtjd}tj|jtjd}tj|jtj d}tj|j
tj d}|||d} | j j d
kr| d= | j j dkr| j
||d | j jr| j
d
|i | jr| j
d tj|jtjd| j j i | jtjkr2tj|jtjd}
tj|jtjd} | j
|
| d | S )N)Zdtype) input_idsattention_masktoken_type_idsxlmroberta
distilbert camembertr_ xlnet) cls_indexp_mask
is_impossibleZlangs)start_positions
end_positions)r` ra rb rc )rd r` )rD rP Ztensorr] longr^ r_ re rf r2 rg r: r updater1 r> ZonesshapeZint64r3 r= r5 r6 Zstart_positionZ end_position) rW ifeaturer] r^ r_ re rf rg inputsrh ri r r r __getitem__ s.
$zSquadDataset.__getitem__)r# r$ r% r& r r: r r rD r5 r= r0 r> r6 r
r r, r r) rY r[ r rP ZTensorrp r r r r r8 e s
6Br8 )'rJ rO Z dataclassesr r enumr typingr r r r rP Ztorch.utils.datar Zfilelockr
Zmodels.auto.modeling_autor Ztokenization_utilsr
utilsr Zprocessors.squadr r r r
get_loggerr# rS listkeysZMODEL_CONFIG_CLASSEStupler( r r5 r8 r r r r s"
: