/usr/local/lib/python3.6/site-packages/transformers/utils/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/transformers/utils/__pycache__/fx.cpython-36.pyc (17976B)
3
EgZ @ s d dl Z d dlZd dlZd dlZd dlmZ d dlmZmZm Z m
Z
m Z m Z m
Z
mZ d dlZd dlmZ d dlmZ d dlmZmZmZmZmZ d dlmZ dd lm Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z, dd
l-m.Z. dd l/m0Z0m1Z1 dd l2m3Z3 e,j4e5Z6d7e
e) e ee7e e7 f e e
e* d
ddZ8dddddddddddg Z9g Z:x:e9D ]2Z;e
e8f e; ne:j>e8e; qnW e(gZ?e@eAeBeCe:e? dd d ZDG d d d eZEed ef ed ef d!d"d#ZFe7e e7ed ef f ed ef d$d%d&ZGeEe7e7eHd'd(d)ZIe e7ed ef f d*d+d,ZJd8eKeKe e eK d/d0d1ZLG d2d3 d3eZMd9e*e e e7 ed4d5d6ZNdS ): N)
ModuleType)AnyCallableDictIterableListOptionalTypeUnion)version)nn)Graph GraphModuleNodeProxyTracer)Argument )CONFIG_MAPPINGMODEL_FOR_CAUSAL_LM_MAPPING&MODEL_FOR_IMAGE_CLASSIFICATION_MAPPINGMODEL_FOR_MASKED_LM_MAPPING!MODEL_FOR_MULTIPLE_CHOICE_MAPPING*MODEL_FOR_NEXT_SENTENCE_PREDICTION_MAPPING MODEL_FOR_PRETRAINING_MAPPING$MODEL_FOR_QUESTION_ANSWERING_MAPPING&MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING)MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING&MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING
MODEL_MAPPINGGPT2DoubleHeadsModelPretrainedConfigPreTrainedModelXLNetForQuestionAnsweringlogging)
get_values)TORCH_FX_REQUIRED_VERSIONis_torch_fx_available)importlib_metadata)
model_namesupported_tasksreturnc C sx t | }ttttttttt t
t d }|d kr4|j }t
|trD|g}g }x*|D ]"}|| j|d }|rN|j| qNW |S )N) defaultZ pretrainingznext-sentence-predictionz masked-lmz causal-lmz
seq2seq-lmzmultiple-choicezquestion-answeringzsequence-classificationztoken-classificationzimage-classification)r r r r r r r r r r r r keys
isinstancestrgetappend)r) r* Zmodel_config_classZ task_mappingZ
model_classestask model_class r4 ?/usr/local/lib/python3.6/site-packages/transformers/utils/fx.py!_generate_supported_model_classes8 s,
r6 ZalbertZbertZ
distilbertZ
mobilebertZelectraz
megatron-bertZgpt2ZgptjZgpt_neoZt5Zrobertac C s | j S )N)__name__)cr4 r4 r5 x s r9 )keyc sx e Zd ZdZdeee d fdd
Zedd Z dd Z
d
d Z fd d
Z dd Z
fddZd fdd Z ZS )HFProxyz
Proxy that is able to provide the proper ranks, shapes and boolean values during symbolic tracing by implementing
the dim, size and __bool__ methods. It can be easily extended by either adding new methods or extending the
existing ones.
N)nodetracerc sN t j||d t| drJ| jd k rJ| jjj| _t| jjj j| _d | _ d S )N)r= r= )
super__init__hasattrr= rootdevicenext
parametersdtypecache)selfr< r= ) __class__r4 r5 r? s
zHFProxy.__init__c C s | j S )N)size)rG r4 r4 r5 shape s z
HFProxy.shapec C s d S )Nr4 )rG r: valuer4 r4 r5 __setitem__ s zHFProxy.__setitem__c C s dS )NFr4 )rG r: r4 r4 r5 __contains__ s zHFProxy.__contains__c s2 | j d k r| j |kS t|tr"dS t j|S d S )NT)rF r. r; r> __eq__)rG other)rH r4 r5 rN s
zHFProxy.__eq__c C s
| |k S )Nr4 )rG rO r4 r4 r5 __ne__ s zHFProxy.__ne__c sR | j d k rFt| j tr | j S t| j tjttfr:t| j S t j | S t j | S )N)
rF r. inttorchSizelisttuplelenr> __len__)rG )rH r4 r5 rW s
zHFProxy.__len__c s t j||||d}| j|_|S )N)argskwargs)r> __torch_function__rF )rG Z orig_methodtypesrX rY proxy)rH r4 r5 rZ s zHFProxy.__torch_function__)N)NN)r7
__module__ __qualname____doc__r r r r? propertyrJ rL rM rN rP rW rZ
__classcell__r4 r4 )rH r5 r; | s
r; .)funcr+ c s t j fdd}|S )zcWrapper that marks func as a leaf function, meaning that it will not be traced through by HFTracer.c s
| |S )Nr4 )rX rY )rb r4 r5 wrapper s z"_function_to_leaf..wrapper) functoolswraps)rb rc r4 )rb r5 _function_to_leaf s rf ) func_namemappingr+ c s t j fdd}|S )Nc s | |S )Nr4 )rX rY )rg rh r4 r5 rc s z&_function_leaf_getter..wrapper)rd re )rg rh rc r4 )rg rh r5 _function_leaf_getter s ri )r\ method_name
cache_name return_proxyc sJ t tj|tj fdd}||_|j| | j}t| || dS )z
Helper function that sets a recorded torch.Tensor method as a HFProxy method that will use the recorded values
during symbolic tracing.
c sD t | d jj }|jd}r@| d jd | |d}||_|S |S )Nr )rX rY )getattrr= rA poprZ rF )rX rY rF resr\ )rk original_methodrl r4 r5 method s
z-_create_recorded_proxy_method..methodN) rm rR Tensorrd re r7 __get__rH setattr)r\ rj rk rl rq Z bound_methodr4 )rk rp rl r5 _create_recorded_proxy_method s
ru )original_methodsc C s( x"| j D ]\}}ttj|| q
W dS )z]Helper function that resets the monkey patched torch.Tensor methods to their original values.N)itemsrt rR rr )rv namerq r4 r4 r5 _reset_tensor_methods s ry
)lowhighforbidden_valuesc C s6 |d kr g }t j| |}x||kr0t j| |}qW |S )N)randomrandint)r| r} r~ rK r4 r4 r5 _generate_random_int s
r c s e Zd ZdZddddZddlmZ eddd d
d hejd hiZ e
ff df fd
d Z e e
dddZd8eedddZee edddZee
e
ee dddZeee
dd d Zee
ee ee
ejf d d d Zd9eee
eee
d"d#d$Z d%d& Z e d'd(d)Z d:eeee
e f eee
e!d* fd+d,
Z"e#j$e
d-d.d/Z%e#j$e
d-d0d1Z&e#j$e
ed2 fd3d4 Z'e e(d5 fd6d7 Z) Z*S );HFTracerz
Tracer that is able to symbolically trace models from the library. To do that, it uses the HFProxy instead of the
regular PyTorch torch.fx.Proxy.
FT)__bool__rI Zdimr )modeling_utilsZarangezerosZonesZ full_likeZeyeZ*create_extended_attention_mask_for_decoderc s i | _ x0| jj D ]"\}}x|D ]}| j|| q W qW t j|||d t svtjt jd}t
d| dt dd | _ d | _
d S )N)autowrap_modulesautowrap_functionsenable_cpatchingrR z6Found an incompatible version of torch. Found version z, but only version z is supported.)_leaf_functions_register_FUNCTIONS_TO_AUTOWRAPrw _register_leaf_functionr> r? r' r parser( ImportErrorr& prev_modulerecorded_methods)rG r r r modulenamesrx Z
torch_version)rH r4 r5 r? s
zHFTracer.__init__)r rx c C s, t ||}t|}t|_|||f| j|< dS )z@Registers the function called name in module as a leaf function.N)rm rf r7 r] r )rG r rx orig_func patched_funcr4 r4 r5 r s
z HFTracer._register_leaf_function)rA restorec C sv xp| j D ]f}| j | \}}}|r>|jjjj| t||| q||jjj|< t||jjj}t|_t||| qW dS )z-Patches leaf functions specifically for root.N) r rH forward __globals__rn rt ri r7 r] )rG rA r rx r r r Z leaf_getterr4 r4 r5 _patch_leaf_functions_for_root s z'HFTracer._patch_leaf_functions_for_root)
module_idsr+ c C sR t j }xD|rL|dkrdS |jjdd}t||krD| j|drDdS |j}q
W dS )z
Finds out if the method (that is being recorded) is called inside a leaf module, this allows to not record
outputs that will not be encountered by the tracer.
NFrG zNot used anywayT)inspect currentframef_localsr0 idis_leaf_modulef_back)rG r r r r4 r4 r5 _method_is_called_in_leaf_module) s
z)HFTracer._method_is_called_in_leaf_module)modelrj rk r c s. t tj|tj fdd}|S )z[Helper function that wraps a torch.Tensor method to record its outputs during forward pass.c sL j r| |S t s*t g t }| |}|j| |S )N)r r@ rt rm r1 )rX rY rF ro )rk rq r r rG r4 r5 wrapped? s
z:HFTracer._wrap_method_for_model_recording..wrapped)rm rR rr rd re )rG r rj rk r r r4 )rk rq r r rG r5 _wrap_method_for_model_recording9 s z)HFTracer._wrap_method_for_model_recording)r method_namesc C s i }i }t dd |j D }x|D ]}d| }|||< ttj|sZtjd| d q$ttj|||< ttj|| j |||| |dkr$tjj
|d< ttjdt ttj| q$W ||fS )z
Helper function that patches torch.Tensor methods (specified by the method_names list) to record model
inference before symbolic tracing.
c s s | ]}t |V qd S )N)r ).0modr4 r4 r5 S s zLHFTracer._monkey_patch_tensor_methods_for_model_recording..Zcache_z"torch.Tensor has no method called z, skipping patching.rI rJ ) setmodulesr@ rR rr loggerinform rt r rJ r` )rG r r cache_namesrv r rj rk r4 r4 r5 0_monkey_patch_tensor_methods_for_model_recordingL s$
z9HFTracer._monkey_patch_tensor_methods_for_model_recording)r
input_namerJ r+ c C s` |j }|j}i }|d kr |d }|ttkrFtj|tj|d|d< n|tttfkrtj|tj|d|d< tj|tj|d|d< n|tt tt
tt krtj|tj|d|d< nR|tt tt
tttttttfk rtj|tj|d|d< nt| dnPd|ks d|kr8tj|tj|d||< n$||jjg }tj|tj|d||< |S )
z4Generates dummy input for model inference recording.labelsstart_positions
end_positionsr )rE rB z not supported yet.maskids)r r r )rH rB r% r rR r longr r# r r r r r r r r r NotImplementedErrorconfigZ hidden_sizefloat) rG r r rJ r3 rB Z inputs_dict
batch_sizeZshape_with_hidden_sizer4 r4 r5 _generate_dummy_inputg s>
z HFTracer._generate_dummy_inputN)r input_namesr c s |dkr| j }t }t }||g} jttkrHtddd}|jd| i }x |D ]} |j| j | | qRW | j |\}
} | | _ f | t
| fdd|
j D | _ dS )z
Records torch.Tensor method outputs (specified by method_names) that will then be used during symbolic tracing.
Nr )r| r} c s i | ]\}}t |r||qS r4 )r@ )r rj rk )r r4 r5
s z#HFTracer.record..)
_DEFAULT_METHODS_TO_RECORDr rH r% r insertupdater r rv ry rw r ) rG r r r r Zsequence_lengthrJ Z num_choicesinputsr r rv r4 )r r5 record s"
zHFTracer.recordc C s t |tjjrRxB| jj D ]4\}}||kr||krF| jd|f i ||< || S qW t |tjrxB| jj D ]4\}}||krj||kr| jd|f i ||< || S qjW |S )NZget_attr) r. rR r ParameterrA Znamed_parametersZ create_proxyrr Z
named_buffers)rG attrZattr_valZparameter_proxy_cachenpr4 r4 r5 _module_getattr s zHFTracer._module_getattr)r< c C sD t || }| jr@x.| jj D ] \}}| j| }t|||| q W |S )N)r; r rw r ru )rG r< r rj rk rl r4 r4 r5 r\ s
zHFTracer.proxy)rA
concrete_argsr r+ c s |d kr i }t j|j}|jj |j }| j|||d dd | jj D }| jj t
dd |D | j | t j
||d| _| j |dd t| j x8| jjD ],}|jdkr|j|krf |_q| jj| qW | jS ) N)r c S s g | ]\}}}|qS r4 r4 )r _Zpatchedr4 r4 r5
s z"HFTracer.trace..c S s g | ] }t |qS r4 )r )r fr4 r4 r5 r s )r T)r placeholder)r signaturer rD r- r r valuesZ_autowrap_function_idsr r r r> tracegraphry rv ZnodesoptargetrX Z
erase_node)rG rA r r sigr r r< )rH r4 r5 r s"
zHFTracer.trace)r r+ c C sX d}|j jj }| d| }x&t| j|rD| d| }|d7 }q W | jj|| |S )zb
Helper method which tries to insert a module that was not declared as submodule.
r r r )rH r7 lowerr@ rA Z
add_module)rG r idxmod_namepathr4 r4 r5 _insert_module_as_submodule s z$HFTracer._insert_module_as_submodulec C s t | drT| jrT| jj|}|dkr.| j|}|dkrJtd|j d|| _|S x(| jj D ]\}}||kr`|| _|S q`W | j|}|dkrtd|j d|| _|S dS )ag
Helper method to find the qualified name of `mod` in the Module hierarchy of `root`. For example, if `root` has
a submodule named `foo`, which has a submodule named `bar`, passing `bar` into this function will return the
string "foo.bar".
Args:
mod (str): The `Module` to retrieve the qualified name for.
submodule_pathsNz
Module named z is not installed as a submodulezModule ) r@ r r0 r NameErrorZ _get_namer rA Z
named_modules)rG r r r r r4 r4 r5 path_of_module s"
zHFTracer.path_of_module)mmodule_qualified_namer+ c s |j jd}| o t j||S )Nztorch.nn.modules.loss)r]
startswithr> r )rG r r Zis_loss_module)rH r4 r5 r ! s zHFTracer.is_leaf_module)ar+ c s& t |trt jt|S t j|S )N)r. ranger>
create_argrT )rG r )rH r4 r5 r % s
zHFTracer.create_arg)F)N)NN)+r7 r] r^ r_ r Z transformersr rR ZModuleUtilsMixinr mathr? r r/ r r" boolr r rQ r r r r r rr r r r r r r\ r r
r r Moduler r r r r ra r4 r4 )rH r5 r s.
* $ r )r r r+ c s dkr| j j tj| j} fdd|jj D }t| tsldj dd tD }t
d| j j d| t
}|j| |d }tjj| |}|S )
ap
Performs symbolic tracing on the model.
Args:
model ([`PretrainedModel`]):
The model to trace.
input_names (`List[str]`, *optional*):
The names of the inputs of the traced model. If unset, model.dummy_inputs.keys() are used instead.
Returns:
`torch.fx.GraphModule`: A GraphModule constructed by recording operations seen while tracing the model.
Example:
```python
from transformers.utils.fx import symbolic_trace
traced_model = symbolic_trace(model, input_names=["input_ids", "attention_mask", "token_type_ids"])
```
Nc s i | ]}|j kr|j|j qS r4 )rx r, )r r )r r4 r5 r H s z"symbolic_trace..z, c s s | ] }|j V qd S )N)r7 )r clsr4 r4 r5 r K s z!symbolic_trace..zModel z) is not supported yet, supported models: )r )Z dummy_inputsr- r r r rD r r. _SUPPORTED_MODELSjoinr rH r7 r r rR Zfxr )r r r r Zsupported_model_namesr= Z traced_graphZtracedr4 )r r5 symbolic_trace+ s
r )N)rz r{ N)N)Ord r r r r[ r typingr r r r r r r r
rR packagingr r Ztorch.fxr
r r r r Z
torch.fx.noder r r r r r r r r r r r r r r! r" r# r$ Z models.autor% utilsr& r' Zutils.versionsr(
get_loggerr7 r r/ r6 Z(_REGULAR_SUPPORTED_MODEL_NAMES_AND_TASKSZ_REGULAR_SUPPORTED_MODELSitemr. dictextendZ_SPECIAL_SUPPORTED_MODELSrU sortedrT r r r; rf ri r ru ry rQ r r r r4 r4 r4 r5 s^ ( L
3
* C