/usr/local/lib/python3.6/site-packages/transformers/models/auto/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/transformers/models/auto/__pycache__/dynamic.cpython-36.pyc (7174B)
3
Egh& @ s d Z ddlZddlZddlZddlZddlZddlmZ ddlm Z m
Z
m Z ddl m
Z
mZmZmZmZ ddlmZ ejeZdd Ze eejf d
d d Zd
d Zdd Z de eejf eee
e eejf e e e
e eef e
e e ef e
e e d
ddZ dS )z?Utilities to dynamically load model and tokenizer from the Hub. N)Path)DictOptionalUnion )HF_MODULES_CACHE TRANSFORMERS_DYNAMIC_MODULE_NAME cached_path
hf_bucket_urlis_offline_mode)loggingc C sH t tjkrdS tjjt tjt dd tt d } | j sD| j dS )z_
Creates the cache directory for modules with an init, and adds it to the Python path.
NT)exist_okz __init__.py) r syspathappendosmakedirsr existstouch) init_path r J/usr/local/lib/python3.6/site-packages/transformers/models/auto/dynamic.pyinit_hf_modules& s
r )namec C sP t tt| }|jj s&t|j tj|dd |d }|j sL|j dS )zF
Creates a dynamic module in the cache directory for modules.
T)r
z __init__.pyN) r r r parentr create_dynamic_moduler r r )r Zdynamic_module_pathr r r r r 5 s
r c C s t | ddd}|j }W dQ R X tjd|tjd}|tjd|tjd7 }dd |D }tt|}g }x:|D ]2}ytj| W qn t k
r |j
| Y qnX qnW t |d
krt d d j | d
dj | ddS )zi
Check if the current Python environment contains all the libraries that are imported in a file.
rzutf-8)encodingNz^\s*import\s+(\S+)\s*$)flagsz^\s*from\s+(\S+)\s+importc S s$ g | ] }|j d s|jd d qS ).r )
startswithsplit).0impr r r
P s z!check_imports..r z\This modeling file requires the following packages that were not found in your environment: z, z. Run `pip install `)
openreadrefindall MULTILINElistset importlib
import_module ImportErrorr lenjoin)filenamefcontentZimportsZmissing_packagesr# r r r
check_importsD s
r6 c C s$ |j tjjd}tj|}t|| S )zY
Import a module on the cache directory for modules and extract a class from it.
r )replacer r sepr. r/ getattr)
class_name module_pathmoduler r r get_class_in_moduleb s
r= F)
pretrained_model_name_or_path module_filer: cache_dirforce_downloadresume_downloadproxiesuse_auth_tokenrevisionlocal_files_onlyc
K sd t r| rtjd d} t| } tjj| rBtjj| |} d} n t| ||dd} | j dtjj
} yt | ||||| |d}
W n. t k
r tj
d| d | d
Y nX t|
ttjj
| }t| tt| }| dkr|}tj|
|| nNt|
j}|j d d g|jd
}d
j|d }|| j sDtj|
|| tjj||j d d }t||S )a*
Extracts a class from a module file, present in the local folder or repository of a model.
Calling this function will execute the code in the module file found locally or downloaded from the Hub. It should
therefore only be called on trusted repos.
Args:
pretrained_model_name_or_path (`str` or `os.PathLike`):
This can be either:
- a string, the *model id* of a pretrained model configuration hosted inside a model repo on
huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced
under a user or organization name, like `dbmdz/bert-base-german-cased`.
- a path to a *directory* containing a configuration file saved using the
[`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`.
module_file (`str`):
The name of the module file containing the class to look for.
class_name (`str`):
The name of the class to import in the module.
cache_dir (`str` or `os.PathLike`, *optional*):
Path to a directory in which a downloaded pretrained model configuration should be cached if the standard
cache should not be used.
force_download (`bool`, *optional*, defaults to `False`):
Whether or not to force to (re-)download the configuration files and override the cached versions if they
exist.
resume_download (`bool`, *optional*, defaults to `False`):
Whether or not to delete incompletely received file. Attempts to resume the download if such a file exists.
proxies (`Dict[str, str]`, *optional*):
A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',
'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request.
use_auth_token (`str` or *bool*, *optional*):
The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
when running `transformers-cli login` (stored in `~/.huggingface`).
revision(`str`, *optional*, defaults to `"main"`):
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
identifier allowed by git.
local_files_only (`bool`, *optional*, defaults to `False`):
If `True`, will only try to load the tokenizer configuration from local files.
Passing `use_auth_token=True` is required when you want to use a private model.
Returns:
`type`: The class, dynamically imported from the module.
Examples:
```python
# Download module *modeling.py* from huggingface.co and cache then extract the class *MyBertModel* from this
# module.
cls = get_class_from_dynamic_module("sgugger/my-bert-model", "modeling.py", "MyBertModel")
```z+Offline mode: forcing local_files_only=TrueTlocalN)r3 rE Zmirror/)r@ rA rC rB rF rD zCould not locate the z inside r z.py _)r loggerinfostrr r isdirr2 r
r7 r8 r EnvironmentErrorerrorr6 r r r r shutilcopyr r! r r= )r> r? r: r@ rA rB rC rD rE rF kwargsZmodule_file_or_url submoduleZresolved_module_fileZfull_submoduleZsubmodule_path module_nameZresolved_module_file_nameZmodule_name_partsZ final_moduler r r get_class_from_dynamic_modulek sH J
rV )NFFNNNF) __doc__r. r r) rQ r pathlibr typingr r r Z
file_utilsr r r r
r utilsr
get_logger__name__rK r rM PathLiker r6 r= boolrV r r r r s*