/usr/local/lib/python3.6/site-packages/transformers/data/datasets/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/transformers/data/datasets/__pycache__/glue.cpython-36.pyc (4701B)
3
Eg @ s d dl Z d dlZd dlZd dlmZmZ d dlmZ d dlm Z m
Z
m Z d dl Z d dl
mZ d dlmZ ddlmZ dd lmZ d
d lmZmZmZ d
d lmZ eje Z eG d
d dZ G dd deZ G dd deZ dS ) N) dataclassfield)Enum)ListOptionalUnion)Dataset)FileLock )PreTrainedTokenizerBase)logging )!glue_convert_examples_to_featuresglue_output_modesglue_processors)
InputFeaturesc @ st e Zd ZU dZedddjej idZe eddidZ
e
edddid Z e ed
dd id Z
e
d d
ZdS )GlueDataTrainingArgumentsz
Arguments pertaining to what data we are going to input our model for training and eval.
Using `HfArgumentParser` we can turn this class into argparse arguments to be able to specify them on the command
line.
helpz"The name of the task to train on: z, )metadatazUThe input data dir. Should contain the .tsv files (or other data files) for the task. zThe maximum total input sequence length after tokenization. Sequences longer than this will be truncated, sequences shorter will be padded.)defaultr Fz1Overwrite the cached training and evaluation setsc C s | j j | _ d S )N) task_namelower)self r I/usr/local/lib/python3.6/site-packages/transformers/data/datasets/glue.py
__post_init__< s z'GlueDataTrainingArguments.__post_init__N)__name__
__module__ __qualname____doc__r joinr keysr strdata_dirmax_seq_lengthintoverwrite_cacheboolr r r r r r $ s
r c @ s e Zd ZdZdZdZdS )SplittraindevtestN)r r r r* r+ r, r r r r r) @ s r) c @ sp e Zd ZU dZeeee
de j dfee
ee eee f ee dddZdd Ze dd d
Zd d ZdS )
GlueDatasetzH
This will be superseded by a framework-agnostic approach soon.
N)args tokenizer limit_lengthmode cache_dirc C s t jdt || _t|j | _t|j | _t |t
r`y t | }W n t k
r^ t dY nX t
jj|d k rr|n|jd|j d|jj d|j d|j }| jj }|jdkr|jjdkr|d |d
|d
< |d < || _|d }t| t
jj|r<|j r
isinstancer# r) KeyErrorospathr! r$ value __class__r r%
get_labelsr= r existsr' timetorchloadfeaturesloggerinfor+ Zget_dev_examplesr, Zget_test_examplesZget_train_examplesr save) r r. r/ r0 r1 r2 Zcached_features_filer= Z lock_pathstartZexamplesr r r __init__O s`
(
zGlueDataset.__init__c C s
t | jS )N)lenrN )r r r r __len__ s zGlueDataset.__len__)returnc C s
| j | S )N)rN )r ir r r __getitem__ s zGlueDataset.__getitem__c C s | j S )N)r= )r r r r rI s zGlueDataset.get_labels)r r r r r r. r# r> r r rN r) r* r r r& r rS rU rX rI r r r r r- F s
*Er- )!rE rK r? Z dataclassesr r enumr typingr r r rL Ztorch.utils.datar Zfilelockr Ztokenization_utils_baser utilsr Zprocessors.gluer r r Zprocessors.utilsr
get_loggerr rO r r) r- r r r r s