/usr/local/lib/python3.6/site-packages/transformers/data/processors/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/transformers/data/processors/__pycache__/glue.cpython-36.pyc (24167B)
3
EgZ @ s d Z ddlZddlZddlmZ ddlmZ ddlmZm Z m
Z
ddl m Z ddl
mZmZ d d
l
mZmZmZ e r~ddlZejeZd Zd0e
ee d f e e e d
ddZe redfej j e e e ej j dddZ d1ee e e e d
ddZ G dd deZ G dd deZ!G dd deZ"G dd d e"Z#G d d d eZ$G d d d eZ%G d!d" d"eZ&G d#d$ d$eZ'G d%d& d&eZ(G d'd( d(eZ)G d)d* d*eZ*d+dd+d+d d+d+d+d+d, Z+e$e"e#e!e%e&e'e(e)e*d-
Z,d.d.d.d.d.d/d.d.d.d.d-
Z-dS )2z GLUE processors and helpers N)asdict)Enum)ListOptionalUnion )PreTrainedTokenizer)is_tf_availablelogging )
DataProcessor InputExample
InputFeaturesu This {0} will be removed from the library soon, preprocessing should be handled with the 🤗 Datasets library. You can have a look at this example script for pointers: https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.pyztf.data.Dataset)examples tokenizer
max_lengthc C sZ t jtjdt t rFt| tjj rF|dkr6t
dt | |||dS t | |||||dS )a=
Loads a data file into a list of `InputFeatures`
Args:
examples: List of `InputExamples` or `tf.data.Dataset` containing the examples.
tokenizer: Instance of a tokenizer that will tokenize the examples
max_length: Maximum example length. Defaults to the tokenizer's max_len
task: GLUE task
label_list: List of labels. Can be obtained from the processor using the `processor.get_labels()` method
output_mode: String indicating the output mode. Either `regression` or `classification`
Returns:
If the `examples` input is a `tf.data.Dataset`, will return a `tf.data.Dataset` containing the task-specific
features. If the input is a list of `InputExamples`, will return a list of task-specific `InputFeatures` which
can be fed to the model.
functionNzWWhen calling glue_convert_examples_to_features from TF, the task parameter is required.)r task)r r
label_list output_mode)
warningswarnDEPRECATION_WARNINGformat
FutureWarningr
isinstancetfdataDataset
ValueError%_tf_glue_convert_examples_to_features"_glue_convert_examples_to_features)r r r r r r r" K/usr/local/lib/python3.6/site-packages/transformers/data/processors/glue.py!glue_convert_examples_to_features) s r$ )r r r returnc s t | fdd| D } t| |||d |dkr:tjntj} fdd}|j}tjjj|dd |D |fd d |D tj g fS )
zb
Returns:
A `tf.data.Dataset` containing the task-specific features.
c s g | ]} j j|qS r" )Ztfds_map get_example_from_tensor_dict).0example) processorr" r#
Z s z9_tf_glue_convert_examples_to_features..)r r zsts-bc 3 s< x6 D ].} dd t | j D }|jd}||fV qW d S )Nc S s i | ]\}}|d k r||qS )Nr" )r' kvr" r" r#
` s zF_tf_glue_convert_examples_to_features..gen..label)r itemspop)exdr. )featuresr" r# gen^ s
z2_tf_glue_convert_examples_to_features..genc S s i | ] }t j|qS r" )r Zint32)r' r+ r" r" r# r- h s z9_tf_glue_convert_examples_to_features..c S s i | ]}t jd g|qS )N)r TensorShape)r' r+ r" r" r# r- i s )
glue_processorsr$ r Zfloat32Zint64Zmodel_input_namesr r Zfrom_generatorr5 )r r r r Z
label_typer4 Z input_namesr" )r3 r) r# r N s
r c sl |d kr|j }|d k rlt| }|d krF|j }tjd| d| d krlt| tjd d| dd t|D ttt t
d f dfdd fd d
| D }|d d
| D |d d
d g }xHt t | D ]8 fdd D } t
f | d| i}
|j|
qW xNt| d d D ]:\} tjd tjd| j tjd| q*W |S )NzUsing label list z
for task zUsing output mode c S s i | ]\}}||qS r" r" )r' ir. r" r" r# r- s z6_glue_convert_examples_to_features..)r( r% c s> | j d krd S dkr | j S dkr2t| j S td S )Nclassification
regression)r. floatKeyError)r( ) label_mapr r" r# label_from_example s
z>_glue_convert_examples_to_features..label_from_examplec s g | ] } |qS r" r" )r' r( )r= r" r# r* s z6_glue_convert_examples_to_features..c S s g | ]}|j |jfqS r" )text_atext_b)r' r( r" r" r# r* s r T)r paddingZ
truncationc s i | ]} | |qS r" r" )r' r+ )batch_encodingr7 r" r# r- s r. z*** Example ***zguid: z
features: )Zmodel_max_lengthr6
get_labelsloggerinfoglue_output_modes enumerater
r intr: rangelenr appendguid) r r r r r r r) labelsr3 inputsfeaturer( r" )rA r7 r= r< r r# r! m s8
r! c @ s e Zd ZdZdZdS )
OutputModer8 r9 N)__name__
__module__ __qualname__r8 r9 r" r" r" r# rP s rP c sP e Zd ZdZ fddZdd Zdd Zdd Zd
d Zd d
Z dd Z
Z S )
MrpcProcessorz/Processor for the MRPC data set (GLUE version).c s$ t j|| tjtjdt d S )Nr) )super__init__r r r r r )selfargskwargs) __class__r" r# rV s zMrpcProcessor.__init__c C s> t |d j |d j jd|d j jdt|d j S )zSee base class.idx sentence1zutf-8 sentence2r. )r
numpydecodestr)rW tensor_dictr" r" r# r& s
z*MrpcProcessor.get_example_from_tensor_dictc C s6 t jdtjj|d | j| jtjj|ddS )zSee base class.z LOOKING AT z train.tsvtrain)rD rE ospathjoin_create_examples _read_tsv)rW data_dirr" r" r# get_train_examples s z MrpcProcessor.get_train_examplesc C s | j | jtjj|ddS )zSee base class.zdev.tsvdev)rf rg rc rd re )rW rh r" r" r# get_dev_examples s z MrpcProcessor.get_dev_examplesc C s | j | jtjj|ddS )zSee base class.ztest.tsvtest)rf rg rc rd re )rW rh r" r" r# get_test_examples s z MrpcProcessor.get_test_examplesc C s ddgS )zSee base class.01r" )rW r" r" r# rC s zMrpcProcessor.get_labelsc
C sp g }xft |D ]Z\}}|dkr q| d| }|d }|d }|dkrJdn|d } |jt|||| d qW |S )z5Creates examples for the training, dev and test sets.r -r rl N)rL r> r? r. )rG rK r
)
rW linesset_typer r7 linerL r> r? r. r" r" r# rf s z MrpcProcessor._create_examples) rQ rR rS __doc__rV r& ri rk rm rC rf
__classcell__r" r" )rZ r# rT s rT c sP e Zd ZdZ fddZdd Zdd Zdd Zd
d Zd d
Z dd Z
Z S )
MnliProcessorz3Processor for the MultiNLI data set (GLUE version).c s$ t j|| tjtjdt d S )Nr) )rU rV r r r r r )rW rX rY )rZ r" r# rV s zMnliProcessor.__init__c C s> t |d j |d j jd|d j jdt|d j S )zSee base class.r[ Zpremisezutf-8Z
hypothesisr. )r
r^ r_ r` )rW ra r" r" r# r& s
z*MnliProcessor.get_example_from_tensor_dictc C s | j | jtjj|ddS )zSee base class.z train.tsvrb )rf rg rc rd re )rW rh r" r" r# ri s z MnliProcessor.get_train_examplesc C s | j | jtjj|ddS )zSee base class.zdev_matched.tsvZ dev_matched)rf rg rc rd re )rW rh r" r" r# rk s z MnliProcessor.get_dev_examplesc C s | j | jtjj|ddS )zSee base class.ztest_matched.tsvZ test_matched)rf rg rc rd re )rW rh r" r" r# rm s z MnliProcessor.get_test_examplesc C s
dddgS )zSee base class.Z
contradiction
entailmentZneutralr" )rW r" r" r# rC s zMnliProcessor.get_labelsc
C sv g }xlt |D ]`\}}|dkr q| d|d }|d }|d }|jdrPdn|d } |jt|||| d qW |S )
z5Creates examples for the training, dev and test sets.r rp rl Nr )rL r> r? r. )rG
startswithrK r
)
rW rr rs r r7 rt rL r> r? r. r" r" r# rf s z MnliProcessor._create_examples) rQ rR rS ru rV r& ri rk rm rC rf rv r" r" )rZ r# rw s rw c s0 e Zd ZdZ fddZdd Zdd Z ZS )MnliMismatchedProcessorz>Processor for the MultiNLI Mismatched data set (GLUE version).c s$ t j|| tjtjdt d S )Nr) )rU rV r r r r r )rW rX rY )rZ r" r# rV s z MnliMismatchedProcessor.__init__c C s | j | jtjj|ddS )zSee base class.zdev_mismatched.tsvZdev_mismatched)rf rg rc rd re )rW rh r" r" r# rk
s z(MnliMismatchedProcessor.get_dev_examplesc C s | j | jtjj|ddS )zSee base class.ztest_mismatched.tsvZtest_mismatched)rf rg rc rd re )rW rh r" r" r# rm s z)MnliMismatchedProcessor.get_test_examples)rQ rR rS ru rV rk rm rv r" r" )rZ r# r} s r} c sP e Zd ZdZ fddZdd Zdd Zdd Zd
d Zd d
Z dd Z
Z S )
ColaProcessorz/Processor for the CoLA data set (GLUE version).c s$ t j|| tjtjdt d S )Nr) )rU rV r r r r r )rW rX rY )rZ r" r# rV s zColaProcessor.__init__c C s0 t |d j |d j jddt|d j S )zSee base class.r[ sentencezutf-8Nr. )r
r^ r_ r` )rW ra r" r" r# r& s
z*ColaProcessor.get_example_from_tensor_dictc C s | j | jtjj|ddS )zSee base class.z train.tsvrb )rf rg rc rd re )rW rh r" r" r# ri & s z ColaProcessor.get_train_examplesc C s | j | jtjj|ddS )zSee base class.zdev.tsvrj )rf rg rc rd re )rW rh r" r" r# rk * s z ColaProcessor.get_dev_examplesc C s | j | jtjj|ddS )zSee base class.ztest.tsvrl )rf rg rc rd re )rW rh r" r" r# rm . s z ColaProcessor.get_test_examplesc C s ddgS )zSee base class.rn ro r" )rW r" r" r# rC 2 s zColaProcessor.get_labelsc C s~ |dk}|r|dd }|r dnd}g }xPt |D ]D\}}| d| }|| } |rXdn|d }
|jt|| d|
d q2W |S )z5Creates examples for the training, dev and test sets.rl r Nr rp )rL r> r? r. )rG rK r
) rW rr rs test_mode
text_indexr r7 rt rL r> r. r" r" r# rf 6 s z ColaProcessor._create_examples) rQ rR rS ru rV r& ri rk rm rC rf rv r" r" )rZ r# r~ s r~ c sP e Zd ZdZ fddZdd Zdd Zdd Zd
d Zd d
Z dd Z
Z S )
Sst2Processorz0Processor for the SST-2 data set (GLUE version).c s$ t j|| tjtjdt d S )Nr) )rU rV r r r r r )rW rX rY )rZ r" r# rV H s zSst2Processor.__init__c C s0 t |d j |d j jddt|d j S )zSee base class.r[ r zutf-8Nr. )r
r^ r_ r` )rW ra r" r" r# r& L s
z*Sst2Processor.get_example_from_tensor_dictc C s | j | jtjj|ddS )zSee base class.z train.tsvrb )rf rg rc rd re )rW rh r" r" r# ri U s z Sst2Processor.get_train_examplesc C s | j | jtjj|ddS )zSee base class.zdev.tsvrj )rf rg rc rd re )rW rh r" r" r# rk Y s z Sst2Processor.get_dev_examplesc C s | j | jtjj|ddS )zSee base class.ztest.tsvrl )rf rg rc rd re )rW rh r" r" r# rm ] s z Sst2Processor.get_test_examplesc C s ddgS )zSee base class.rn ro r" )rW r" r" r# rC a s zSst2Processor.get_labelsc
C sx g }|dkrdnd}x^t |D ]R\}}|dkr0q | d| }|| }|dkrRdn|d } |jt||d| d q W |S )z5Creates examples for the training, dev and test sets.rl r r rp N)rL r> r? r. )rG rK r
)
rW rr rs r r r7 rt rL r> r. r" r" r# rf e s z Sst2Processor._create_examples) rQ rR rS ru rV r& ri rk rm rC rf rv r" r" )rZ r# r E s r c sP e Zd ZdZ fddZdd Zdd Zdd Zd
d Zd d
Z dd Z
Z S )
StsbProcessorz0Processor for the STS-B data set (GLUE version).c s$ t j|| tjtjdt d S )Nr) )rU rV r r r r r )rW rX rY )rZ r" r# rV v s zStsbProcessor.__init__c C s> t |d j |d j jd|d j jdt|d j S )zSee base class.r[ r\ zutf-8r] r. )r
r^ r_ r` )rW ra r" r" r# r& z s
z*StsbProcessor.get_example_from_tensor_dictc C s | j | jtjj|ddS )zSee base class.z train.tsvrb )rf rg rc rd re )rW rh r" r" r# ri s z StsbProcessor.get_train_examplesc C s | j | jtjj|ddS )zSee base class.zdev.tsvrj )rf rg rc rd re )rW rh r" r" r# rk s z StsbProcessor.get_dev_examplesc C s | j | jtjj|ddS )zSee base class.ztest.tsvrl )rf rg rc rd re )rW rh r" r" r# rm s z StsbProcessor.get_test_examplesc C s dgS )zSee base class.Nr" )rW r" r" r# rC s zStsbProcessor.get_labelsc
C st g }xjt |D ]^\}}|dkr q| d|d }|d }|d }|dkrNdn|d } |jt|||| d qW |S )
z5Creates examples for the training, dev and test sets.r rp ry rl Nr )rL r> r? r. r{ )rG rK r
)
rW rr rs r r7 rt rL r> r? r. r" r" r# rf s z StsbProcessor._create_examples) rQ rR rS ru rV r& ri rk rm rC rf rv r" r" )rZ r# r s s r c sP e Zd ZdZ fddZdd Zdd Zdd Zd
d Zd d
Z dd Z
Z S ) QqpProcessorz.Processor for the QQP data set (GLUE version).c s$ t j|| tjtjdt d S )Nr) )rU rV r r r r r )rW rX rY )rZ r" r# rV s zQqpProcessor.__init__c C s> t |d j |d j jd|d j jdt|d j S )zSee base class.r[ Z question1zutf-8Z question2r. )r
r^ r_ r` )rW ra r" r" r# r& s
z)QqpProcessor.get_example_from_tensor_dictc C s | j | jtjj|ddS )zSee base class.z train.tsvrb )rf rg rc rd re )rW rh r" r" r# ri s z QqpProcessor.get_train_examplesc C s | j | jtjj|ddS )zSee base class.zdev.tsvrj )rf rg rc rd re )rW rh r" r" r# rk s z QqpProcessor.get_dev_examplesc C s | j | jtjj|ddS )zSee base class.ztest.tsvrl )rf rg rc rd re )rW rh r" r" r# rm s z QqpProcessor.get_test_examplesc C s ddgS )zSee base class.rn ro r" )rW r" r" r# rC s zQqpProcessor.get_labelsc
C s |dk}|rdnd}|r dnd}g }xt |D ]v\}}|dkr@q.| d|d } y$|| }
|| } |rldn|d } W n tk
r w.Y nX |jt| |
| | d
q.W |S ) z5Creates examples for the training, dev and test sets.rl r r rq r rp NrB )rL r> r? r. )rG
IndexErrorrK r
)
rW rr rs r Zq1_indexZq2_indexr r7 rt rL r> r? r. r" r" r# rf s z QqpProcessor._create_examples) rQ rR rS ru rV r& ri rk rm rC rf rv r" r" )rZ r# r s r c sP e Zd ZdZ fddZdd Zdd Zdd Zd
d Zd d
Z dd Z
Z S )
QnliProcessorz/Processor for the QNLI data set (GLUE version).c s$ t j|| tjtjdt d S )Nr) )rU rV r r r r r )rW rX rY )rZ r" r# rV s zQnliProcessor.__init__c C s> t |d j |d j jd|d j jdt|d j S )zSee base class.r[ Zquestionzutf-8r r. )r
r^ r_ r` )rW ra r" r" r# r& s
z*QnliProcessor.get_example_from_tensor_dictc C s | j | jtjj|ddS )zSee base class.z train.tsvrb )rf rg rc rd re )rW rh r" r" r# ri s z QnliProcessor.get_train_examplesc C s | j | jtjj|ddS )zSee base class.zdev.tsvrj )rf rg rc rd re )rW rh r" r" r# rk s z QnliProcessor.get_dev_examplesc C s | j | jtjj|ddS )zSee base class.ztest.tsvrl )rf rg rc rd re )rW rh r" r" r# rm s z QnliProcessor.get_test_examplesc C s ddgS )zSee base class.rx not_entailmentr" )rW r" r" r# rC s zQnliProcessor.get_labelsc
C st g }xjt |D ]^\}}|dkr q| d|d }|d }|d }|dkrNdn|d } |jt|||| d qW |S ) z5Creates examples for the training, dev and test sets.r rp r r rl N)rL r> r? r. r{ )rG rK r
)
rW rr rs r r7 rt rL r> r? r. r" r" r# rf s z QnliProcessor._create_examples) rQ rR rS ru rV r& ri rk rm rC rf rv r" r" )rZ r# r s r c sP e Zd ZdZ fddZdd Zdd Zdd Zd
d Zd d
Z dd Z
Z S ) RteProcessorz.Processor for the RTE data set (GLUE version).c s$ t j|| tjtjdt d S )Nr) )rU rV r r r r r )rW rX rY )rZ r" r# rV s zRteProcessor.__init__c C s> t |d j |d j jd|d j jdt|d j S )zSee base class.r[ r\ zutf-8r] r. )r
r^ r_ r` )rW ra r" r" r# r&
s
z)RteProcessor.get_example_from_tensor_dictc C s | j | jtjj|ddS )zSee base class.z train.tsvrb )rf rg rc rd re )rW rh r" r" r# ri s z RteProcessor.get_train_examplesc C s | j | jtjj|ddS )zSee base class.zdev.tsvrj )rf rg rc rd re )rW rh r" r" r# rk s z RteProcessor.get_dev_examplesc C s | j | jtjj|ddS )zSee base class.ztest.tsvrl )rf rg rc rd re )rW rh r" r" r# rm s z RteProcessor.get_test_examplesc C s ddgS )zSee base class.rx r r" )rW r" r" r# rC s zRteProcessor.get_labelsc
C st g }xjt |D ]^\}}|dkr q| d|d }|d }|d }|dkrNdn|d } |jt|||| d qW |S ) z5Creates examples for the training, dev and test sets.r rp r r rl N)rL r> r? r. r{ )rG rK r
)
rW rr rs r r7 rt rL r> r? r. r" r" r# rf # s z RteProcessor._create_examples) rQ rR rS ru rV r& ri rk rm rC rf rv r" r" )rZ r# r s r c sP e Zd ZdZ fddZdd Zdd Zdd Zd
d Zd d
Z dd Z
Z S )
WnliProcessorz/Processor for the WNLI data set (GLUE version).c s$ t j|| tjtjdt d S )Nr) )rU rV r r r r r )rW rX rY )rZ r" r# rV 4 s zWnliProcessor.__init__c C s> t |d j |d j jd|d j jdt|d j S )zSee base class.r[ r\ zutf-8r] r. )r
r^ r_ r` )rW ra r" r" r# r& 8 s
z*WnliProcessor.get_example_from_tensor_dictc C s | j | jtjj|ddS )zSee base class.z train.tsvrb )rf rg rc rd re )rW rh r" r" r# ri A s z WnliProcessor.get_train_examplesc C s | j | jtjj|ddS )zSee base class.zdev.tsvrj )rf rg rc rd re )rW rh r" r" r# rk E s z WnliProcessor.get_dev_examplesc C s | j | jtjj|ddS )zSee base class.ztest.tsvrl )rf rg rc rd re )rW rh r" r" r# rm I s z WnliProcessor.get_test_examplesc C s ddgS )zSee base class.rn ro r" )rW r" r" r# rC M s zWnliProcessor.get_labelsc
C st g }xjt |D ]^\}}|dkr q| d|d }|d }|d }|dkrNdn|d } |jt|||| d qW |S ) z5Creates examples for the training, dev and test sets.r rp r r rl N)rL r> r? r. r{ )rG rK r
)
rW rr rs r r7 rt rL r> r? r. r" r" r# rf Q s z WnliProcessor._create_examples) rQ rR rS ru rV r& ri rk rm rC rf rv r" r" )rZ r# r 1 s r r ) colamnlimrpczsst-2zsts-bqqpqnlirtewnli)
r r zmnli-mmr zsst-2zsts-br r r r r8 r9 )NNNN)NNNN).ru rc r Z dataclassesr enumr typingr r r Ztokenization_utilsr utilsr r
r r
r Z
tensorflowr
get_loggerrQ rD r rH r$ r` r r r r! rP rT rw r} r~ r r r r r r Zglue_tasks_num_labelsr6 rF r" r" r" r# s
$ 1/./..4../