/usr/local/lib/python3.6/site-packages/transformers/utils/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/transformers/utils/__pycache__/hub.cpython-36.pyc (31649B)
3
Eg @ sB d Z ddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddl Z ddl m
Z
ddlmZ ddlmZ ddlmZ ddlmZmZmZmZmZmZ ddlm Z dd l m Z dd
l m Z m!Z! ddl"Z"dd l#m$Z$ dd l%m&Z&m'Z'm(Z(m)Z)m*Z* dd
l+m,Z, ddl-m.Z. ddl/m0Z0m1Z1 ddl2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8 e1j9e:Z;ej
e3krrdndZ?dd Z@ejAdejBjCejAdddZDejBjCeDd ZEejBjFejAd ejBjCejAddd ZGejBjCeGd ZHejBjIeEr6ejBjIeH r6d ej e3kZVeVrd+nd,ZWeWZXejd?d@ZcdAdB Zddjeee]df e]dCdDdEZeG dFdG dGe,ZfG dHdI dIe,ZgG dJdK dKe,ZhdLdM Zidke]eeee]e]f dNdOdPZjdle]eee]df eebe]df ee] dRdSdTZkdmee]ejlf e]eee]ejlf ebebeee]e]f eeebe]f ee] ebdU dVdWZmdnee]ejlf e]ee] ee] eee]e]f eeebe]f dXdYdZZndoee]ejlf ee] eeebe]f ebee] d[d\d]Zod^d_ ZpG d`da daZqdpe]ee] ee] dbdcddZrdS )qz?
Hub utilities: utilities related to download and cache models
N)contextmanager)partial)sha256)Path)BinaryIODictListOptionalTupleUnion)urlparse)uuid4)ZipFile
is_zipfile)FileLock)HfFolder
Repository create_repolist_repo_fileswhoami) HTTPError)tqdm ) __version__logging)ENV_VARS_TRUE_VALUES _tf_version_torch_versionis_tf_availableis_torch_available is_training_run_on_sagemakerZTRANSFORMERS_OFFLINE0TFc C s t S )N)_is_offline_mode r# r# @/usr/local/lib/python3.6/site-packages/transformers/utils/hub.pyis_offline_mode; s r% Z
TORCH_HOMEXDG_CACHE_HOMEz~/.cacheZtorchZ transformersZHF_HOMEZ huggingface PYTORCH_PRETRAINED_BERT_CACHEPYTORCH_TRANSFORMERS_CACHETRANSFORMERS_CACHEa In Transformers v4.0.0, the default path to cache downloaded models changed from '~/.cache/torch/transformers' to '~/.cache/huggingface/transformers'. Since you don't seem to have overridden and '~/.cache/torch/transformers' is a directory that exists, we're moving it to '~/.cache/huggingface/transformers' to avoid redownloading models you have already in the cache. You should only see this message once.HF_MODULES_CACHEmodulesZtransformers_modulesDISABLE_TELEMETRYz3https://s3.amazonaws.com/models.huggingface.co/bertzhttps://cdn.huggingface.coZHUGGINGFACE_CO_STAGINGNOz#https://moon-staging.huggingface.cozhttps://huggingface.co HUGGINGFACE_CO_RESOLVE_ENDPOINTzUsing the environment variable `HUGGINGFACE_CO_RESOLVE_ENDPOINT` is deprecated and will be removed in Transformers v5. Use `HF_ENDPOINT` instead.Z HF_ENDPOINTz)/{model_id}/resolve/{revision}/{filename}c C s t | }|jdkS )Nhttphttps)r/ r0 )r scheme)url_or_filenameparsedr# r# r$
is_remote_urlr s r4 )model_idfilename subfolderrevisionreturnc C sz |dk r| d| }|r^|d kr*t dd| k}|rJ| d| d| S | d| d| S |dkrjd}tj| ||dS )
a#
Resolve a model identifier, a file name, and an optional revision id, to a huggingface.co-hosted url, redirecting
to Cloudfront (a Content Delivery Network, or CDN) for large files.
Cloudfront is replicated over the globe so downloads are way faster for the end user (and it also lowers our
bandwidth costs).
Cloudfront aggressively caches files by default (default TTL is 24 hours), however this is not an issue here
because we migrated to a git-based versioning system on huggingface.co, so we now store the files on S3/Cloudfront
in a content-addressable way (i.e., the file name is its hash). Using content-addressable filenames means cache
can't ever be stale.
In terms of client-side caching from this library, we base our caching on the objects' ETag. An object' ETag is:
its sha1 if stored in git, or its sha256 if stored in git-lfs. Files cached locally from transformers before v3.5.0
are not shared with those new files, because the cached file's name contains a hash of the url (which changed).
N/tunabfsuzTThe Tuna and BFSU mirrors are no longer available. Try removing the mirror argument.-main)r5 r8 r6 )r; r< )
ValueErrorHUGGINGFACE_CO_PREFIXformat)r5 r6 r7 r8 mirrorZ
legacy_formatr# r# r$
hf_bucket_urlw s rC )urletagr9 c C sN | j d}t|j }|r8|j d}|dt|j 7 }| jdrJ|d7 }|S )a
Convert `url` into a hashed filename in a repeatable way. If `etag` is specified, append its hash to the url's,
delimited by a period. If the url ends with .h5 (Keras HDF5 weights) adds '.h5' to the name so that TF 2.0 can
identify it as a HDF5 file (see
https://github.com/tensorflow/tensorflow/blob/00fad90125b18b80fe054de1055770cfb8fe4ba3/tensorflow/python/keras/engine/network.py#L1380)
zutf-8.z.h5)encoder hexdigestendswith)rD rE Z url_bytesr6 Z
etag_bytesr# r# r$ url_to_filename s
rJ c C s |dkr t }t|tr t|}tjj|| }tjj|sHtd| d|d }tjj|sltd| dt |dd}t
j |}W dQ R X |d }|d }||fS ) z
Return the url and etag (which may be `None`) stored for *filename*. Raise `EnvironmentError` if *filename* or its
stored metadata do not exist.
Nzfile z
not foundz.jsonzutf-8)encodingrD rE ) r)
isinstancer strospathjoinexistsEnvironmentErroropenjsonload)r6 cache_dir
cache_path meta_path meta_filemetadatarD rE r# r# r$ filename_to_url s
r[ )rV r9 c C s | dkrt } nt| tr t| } g }xtj| D ]~}|jdr0tjj| |}t |ddP}t
j |}|d }|d }|jdrtjj |j
dd }|j|||f W dQ R X q0W |S ) a
Returns a list of tuples representing model binaries that are cached locally. Each tuple has shape `(model_url,
etag, size_MB)`. Filenames in `cache_dir` are use to get the metadata for each model, only urls ending with *.bin*
are added.
Args:
cache_dir (`Union[str, Path]`, *optional*):
The cache directory to search for models within. Will default to the transformers cache if unset.
Returns:
List[Tuple]: List of tuples each with shape `(model_url, etag, size_MB)`
Nz.jsonzutf-8)rK rD rE z.bing .A)r) rL r rM rN listdirrI rO rP rS rT rU getsizestripappend) rV Z
cached_modelsfilerX rY rZ rD rE Zsize_MBr# r# r$ get_cached_models s
ra )
user_agentuse_auth_tokenr9 c
C s |dkr t }t| tr t| } t|tr0t|}t rJ| rJtjd d} t| rlt| ||||||| d}
nBt j
j | r~| }
n0t | j
dkrtd| dntd| d |rt|
rtj|
r|
S t j
j|
\} } | jd
d d }
t j
j| |
}t j
j|r"t j|r"| r"|S |
d
}t| tj|dd t j| t|
rt |
d}|j | |j W dQ R X n:tj|
rtj |
}|j | |j ntd|
dW dQ R X |S |
S )aj
Given something that might be a URL (or might be a local path), determine which. If it's a URL, download the file
and cache it, and return the path to the cached file. If it's already a local path, make sure the file exists and
then return the path
Args:
cache_dir: specify a cache directory to save the file to (overwrite the default cache dir).
force_download: if True, re-download the file even if it's already cached in the cache dir.
resume_download: if True, resume the download if incompletely received file is found.
user_agent: Optional string or dict that will be appended to the user-agent on remote requests.
use_auth_token: Optional string or boolean to use as Bearer token for remote files. If True,
will get token from ~/.huggingface.
extract_compressed_file: if True and the path point to a zip or tar file, extract the compressed
file in a folder along the archive.
force_extract: if True when extract_compressed_file is True and the archive was already extracted,
re-extract the archive and override the folder where it was extracted.
Return:
Local path (string) of file or if networking is off, last version of file cached on disk.
Raises:
In case of non-recoverable file (non-existent or inaccessible url + no cache on disk).
Nz+Offline mode: forcing local_files_only=TrueT)rV force_downloadproxiesresume_downloadrb rc local_files_only zfile z
not foundzunable to parse z as a URL or as a local pathrF r= z
-extractedz.lock)
ignore_errorsrzArchive format of z could not be identified) r) rL r rM r% loggerinfor4 get_from_cacherN rO rQ r r1 rR r? r tarfile
is_tarfilesplitreplacerP isdirr\ r shutilrmtreemakedirsr
extractallcloserS )r2 rV rd re rf rb Zextract_compressed_fileZ
force_extractrc rg Z output_path
output_dirZ output_fileZoutput_extract_dir_nameZoutput_path_extracted lock_pathzip_fileZtar_filer# r# r$ cached_path s^ #
"
r{ c C s y2t jtjd j } | d }| d jdd }W n tk
rN d }d }Y nX tjtjdd}d|krndnd }d
tjkrtjd
jdd nd }tjd d tjd
d tjddtjdd||||d}|S )NZECS_CONTAINER_METADATA_URIZImage:r ZSM_FRAMEWORK_PARAMSz{}Z*sagemaker_distributed_dataparallel_enabledTFZTRAINING_JOB_ARN ZSM_FRAMEWORK_MODULEZ
AWS_REGIONZ SM_NUM_GPUSr Z SM_NUM_CPUS)Z sm_frameworkZ sm_regionZ
sm_number_gpuZ
sm_number_cpuZsm_distributed_trainingZsm_deep_learning_containerZ sm_deep_learning_container_tagZ
sm_account_id) requestsgetrN environrT rp Exceptionloadsgetenv)Z
instance_dataZdlc_container_usedZdlc_tagZsagemaker_paramsZruns_distributed_trainingZ
account_idZsagemaker_objectr# r# r$ define_sagemaker_informationP s&
"
r )rb r9 c C s dt dtjj d dt }t r4|dt 7 }t rH|dt 7 }t rT|d S t
r||ddj d d
t j
D 7 }tjjd d j tkr|d
7 }t| tr|ddj dd
| j
D 7 }nt| tr|d| 7 }|S )zF
Formats a user-agent string with basic info about a request.
z
transformers/z ; python/r z
; session_id/z; torch/z
; tensorflow/z; telemetry/offz; c s s | ]\}}| d | V qdS )r: Nr# ).0kvr# r# r$ v s z"http_user_agent..ZTRANSFORMERS_IS_CIrh z ; is_ci/truec s s | ]\}}| d | V qdS )r: Nr# )r r r r# r# r$ r { s )r sysversionrp
SESSION_IDr r r r r, r rP r itemsrN r r upperr rL dictrM )rb Zuar# r# r$ http_user_agentj s "
"
r c @ s e Zd ZdZdS )RepositoryNotFoundErrorz
Raised when trying to access a hf.co URL with an invalid repository name, or with a private repo name the user does
not have access to.
N)__name__
__module__ __qualname____doc__r# r# r# r$ r s r c @ s e Zd ZdZdS )EntryNotFoundErrorzfRaised when trying to access a hf.co URL with a valid repository and revision but an invalid filename.N)r r r r r# r# r# r$ r s r c @ s e Zd ZdZdS )RevisionNotFoundErrorzYRaised when trying to access a hf.co URL with a valid repository but an invalid revision.N)r r r r r# r# r# r$ r s r c C sl d| j kr`| j d }|dkr.td| j n2|dkrHtd| j n|dkr`td| j | j dS ) zb
Internal version of `request.raise_for_status()` that will refine a potential HTTPError.
z X-Error-CodeZ RepoNotFoundz0404 Client Error: Repository Not Found for url: Z
EntryNotFoundz+404 Client Error: Entry Not Found for url: ZRevisionNotFoundz.404 Client Error: Revision Not Found for url: N)headersr rD r r raise_for_status)requestZ
error_coder# r# r$ _raise_for_status s
r )rD temp_filer c
C s t j|}|dkr"d| d|d< tj| d||d}t| |jjd}|dk r\|t| nd}td dd
||d d }x0|jd
d
D ] } | r|j t
| |j | qW |j dS )z8
Download remote file. Do not gobble up errors.
r zbytes=r= RangeT)streamre r zContent-LengthNBi Z Downloading)unitZ
unit_scaleZ unit_divisortotalinitialdesc)
chunk_size)
copydeepcopyr~ r r r intr iter_contentupdatelenwriterw )
rD r re resume_sizer rj content_lengthr progresschunkr# r# r$ http_get s&
r
)rD rb rc r9 c sX |dkr t }t|tr t|}tj|dd dt|i} t|trRd| | d< n*|r|tj }
|
dkrnt dd|
| d< | } d} |sDylt
j | | d||d }
t |
|
j
jd
p|
j
jd } | dkrtd d
|
j kodkn r|
j
d } W nL t
jjt
jjtttfk
r Y n$ tt
jjt
jjfk
rB Y nX t| | }tjj ||}| dkrtjj |rx|S dd t j tj ||j!dd d D }t"|dkrtjj ||d) S |rt#dnt$dtjj |r| r|S |d }t%|H tjj |r"| r"|S |rl|d t&dd fd d }|}tjj rftj' j(}nd}nt)t*j+d |dd }d}| .}t,j-| d |j. t/| |||| d! W dQ R X t,j-d"| d#| tj0|j.| tj1d$}tj1| tj2|d$| @ t,j-d%| | | d&}|d' }t3|d(}t4j5|| W dQ R X W dQ R X |S )*ar
Given a URL, look for the corresponding file in the local cache. If it's not there, download it. Then return the
path to the cached file.
Return:
Local path (string) of file or if networking is off, last version of file cached on disk.
Raises:
In case of non-recoverable file (non-existent or inaccessible url + no cache on disk).
NT)exist_okz
user-agentzBearer
authorizationzIYou specified use_auth_token=True, but a huggingface token was not found.F)r allow_redirectsre timeoutz
X-Linked-EtagETagz\Distant resource does not have an ETag, we won't be able to reliably ensure reproducibility.i, i ZLocationc S s( g | ] }|j d r|j d r|qS )z.jsonz.lock)rI )r r` r# r# r$
s z"get_from_cache..rF r z.*r zCannot find the requested files in the cached path and outgoing traffic has been disabled. To enable model look-ups and downloads online, set 'local_files_only' to False.zConnection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.z.lockz .incompletezio.BufferedWriter)r9 c 3 s t d } | V W d Q R X d S )Nab)rS )f)incomplete_pathr# r$ _resumable_file_manager6 s z/get_from_cache.._resumable_file_managerwb)modedirdeletezB not found in cache or force_download set to True, downloading to )re r r zstoring z
in cache at i zcreating metadata file for )rD rE z.jsonw)6r) rL r rM rN ru r r get_tokenrR r~ headr r r OSError status_code
exceptionsSSLError
ProxyErrorr r r r ConnectionErrorTimeoutrJ rO rP rQ fnmatchfilterr\ rp r FileNotFoundErrorr? r r statst_sizer tempfileNamedTemporaryFilerk rl namer rq umaskchmodrS rT dump)rD rV rd re Z etag_timeoutrf rb rc rg r tokenZurl_to_downloadrE rj r6 rW Zmatching_filesry r Ztemp_file_managerr r r metarX rY r# )r r$ rm s
$
rm ) path_or_repor6 rV rd rf re rc r8 rg c
C s t r| rtjd d}t| } tjj| rPtjj| |} tjj| rL| S dS t | ||dd} yt
| ||||||d} W nZ t k
r t | dY n: t
k
r t | d| dY n t k
r dS X | S ) a@
Tries to locate a file in a local folder and repo, downloads and cache it if necessary.
Args:
path_or_repo (`str` or `os.PathLike`):
This can be either:
- a string, the *model id* of a model repo on huggingface.co.
- a path to a *directory* potentially containing the file.
filename (`str`):
The name of the file to locate in `path_or_repo`.
cache_dir (`str` or `os.PathLike`, *optional*):
Path to a directory in which a downloaded pretrained model configuration should be cached if the standard
cache should not be used.
force_download (`bool`, *optional*, defaults to `False`):
Whether or not to force to (re-)download the configuration files and override the cached versions if they
exist.
resume_download (`bool`, *optional*, defaults to `False`):
Whether or not to delete incompletely received file. Attempts to resume the download if such a file exists.
proxies (`Dict[str, str]`, *optional*):
A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',
'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request.
use_auth_token (`str` or *bool*, *optional*):
The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
when running `transformers-cli login` (stored in `~/.huggingface`).
revision (`str`, *optional*, defaults to `"main"`):
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
identifier allowed by git.
local_files_only (`bool`, *optional*, defaults to `False`):
If `True`, will only try to load the tokenizer configuration from local files.
Passing `use_auth_token=True` is required when you want to use a private model.
Returns:
`Optional[str]`: Returns the resolved file (to the cache folder if downloaded from a repo) or `None` if the
file does not exist.
Examples:
```python
# Download a tokenizer configuration from huggingface.co and cache.
tokenizer_config = get_file_from_repo("bert-base-uncased", "tokenizer_config.json")
# This model does not have a tokenizer config so the result will be None.
tokenizer_config = get_file_from_repo("xlm-roberta-base", "tokenizer_config.json")
```z+Offline mode: forcing local_files_only=TrueTN)r6 r8 rB )rV rd re rf rg rc a is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.z is not a valid git identifier (branch name, tag name or commit id) that exists for this model name. Check the model page at 'https://huggingface.co/z' for available revisions.)r% rk rl rM rN rO rr rP isfilerC r{ r rR r )
r r6 rV rd rf re rc r8 rg Z
resolved_filer# r# r$ get_file_from_repo\ s4 =
r )r r6 r8 rB re rc c C s0 t jj| r"t jjt jj| |S t| |||d}dt i}t|trVd| |d< n*|rt j
}|dkrrt dd| |d< t j
||d|dd } y t| d
S tk
r }
z tj|
t | d W Y dd}
~
X nR tk
r }
z tj|
t | d W Y dd}
~
X n t jk
r* dS X dS )
aV
Checks if a repo contains a given file wihtout downloading it. Works for remote repos and local folders.
This function will raise an error if the repository `path_or_repo` is not valid or if `revision` does not exist for
this repo, but will return False for regular connection errors.
)r6 r8 rB z
user-agentzBearer r NzIYou specified use_auth_token=True, but a huggingface token was not found.Fr )r r re r TzE is not a local folder or a valid repository name on 'https://hf.co'.z is not a valid git identifier (branch name, tag name or commit id) that exists for this model name. Check the model page at 'https://huggingface.co/{path_or_repo}' for available revisions.)rN rO rr r rP rC r rL rM r r rR r~ r r r rk errorr r ) r r6 r8 rB re rc rD r r rj er# r# r$ has_file s0
r )r r8 rc rg r9 c s t | } tjj| rNg }x0tj| D ]"\ }}|j fdd|D q$W |S t sX|r\g S t|t rl|}n|dkr~tj }nd}yt
| ||dS t k
r } zt | d|W Y dd}~X nX dS )a|
Gets the list of files inside `path_or_repo`.
Args:
path_or_repo (`str` or `os.PathLike`):
Can be either the id of a repo on huggingface.co or a path to a *directory*.
revision (`str`, *optional*, defaults to `"main"`):
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
identifier allowed by git.
use_auth_token (`str` or *bool*, *optional*):
The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
when running `transformers-cli login` (stored in `~/.huggingface`).
local_files_only (`bool`, *optional*, defaults to `False`):
Whether or not to only rely on local files and not to attempt to download any files.
This API is not optimized, so calling it a lot may result in connection errors.
Returns:
`List[str]`: The list of files available in `path_or_repo`.
c s g | ]}t jj |qS r# )rN rO rP )r r )rO r# r$ r s z%get_list_of_files..TN)r8 r zQ is not a local path or a model identifier on the model Hub. Did you make a typo?)
rM rN rO rr walkextendr% rL r r r r r? ) r r8 rc rg Z
list_of_filesZ dir_namesZ
file_namesr r r# )rO r$ get_list_of_files s&
r c C sf t jjt jj| dsdS tjdj | d}|jdkr:dS tjdj tjtjdd| d j }||j kS )
zM
Checks if the folder in `repo_path` is a local clone of `repo_url`.
z.gitFz
git branch)cwdr z
git remote -vTzutf-8)stderrstdoutcheckrK r )
rN rO rQ rP
subprocessrunrp
returncodePIPEr ) repo_pathrepo_urlZtest_gitremotesr# r# r$ is_local_clone1 s
r c @ s e Zd ZdZdee ee eee ee ee eeeef edddZe deee eeeeef eddd Z
e dee ee ee eeeeef e d
d d Z
e de ee ed
ddZdS )PushToHubMixinzW
A Mixin containing the functionality to push a model or tokenizer to the hub.
NF)repo_path_or_namer use_temp_dircommit_message organizationprivaterc r9 c
K s |r<|dkr4|dkrd}t |j} | j| |||d}tj }| j|||||d}
| j| t| drt| dr|t |jd} | j| | j f | | j
|
|d} |rt j | | S ) u
Upload the {object_files} to the 🤗 Model Hub while synchronizing a local clone of the repo in
`repo_path_or_name`.
Parameters:
repo_path_or_name (`str`, *optional*):
Can either be a repository name for your {object} in the Hub or a path to a local folder (in which case
the repository will have the name of that local folder). If not specified, will default to the name
given by `repo_url` and a local directory with that name will be created.
repo_url (`str`, *optional*):
Specify this in case you want to push to an existing repository in the hub. If unspecified, a new
repository will be created in your namespace (unless you specify an `organization`) with `repo_name`.
use_temp_dir (`bool`, *optional*, defaults to `False`):
Whether or not to clone the distant repo in a temporary directory or in `repo_path_or_name` inside the
current working directory. This will slow things down if you are making changes in an existing repo
since you will need to clone the repo before every push.
commit_message (`str`, *optional*):
Message to commit while pushing. Will default to `"add {object}"`.
organization (`str`, *optional*):
Organization in which you want to push your {object} (you must be a member of this organization).
private (`bool`, *optional*):
Whether or not the repository created should be private (requires a paying subscription).
use_auth_token (`bool` or `str`, *optional*):
The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
when running `transformers-cli login` (stored in `~/.huggingface`). Will default to `True` if
`repo_url` is not specified.
Returns:
`str`: The url of the commit of your {object} in the given repository.
Examples:
```python
from transformers import {object_class}
{object} = {object_class}.from_pretrained("bert-base-cased")
# Push the {object} to your namespace with the name "my-finetuned-bert" and have a local clone in the
# *my-finetuned-bert* folder.
{object}.push_to_hub("my-finetuned-bert")
# Push the {object} to your namespace with the name "my-finetuned-bert" with no local clone.
{object}.push_to_hub("my-finetuned-bert", use_temp_dir=True)
# Push the {object} to an organization with the name "my-finetuned-bert" and have a local clone in the
# *my-finetuned-bert* folder.
{object}.push_to_hub("my-finetuned-bert", organization="huggingface")
# Make a change to an existing repo that has been cloned locally in *my-finetuned-bert*.
{object}.push_to_hub("my-finetuned-bert", repo_url="https://huggingface.co/sgugger/my-finetuned-bert")
```
NT)r r rc )r r r r rc historycreate_model_card)rx Z
model_name)r )
r r _get_repo_url_from_namer mkdtemp_create_or_get_repoZsave_pretrainedhasattrr r _push_to_hubrs rt )
selfr r r r r r rc Zmodel_card_kwargs repo_namerepoZbase_model_card_argsrD r# r# r$ push_to_hubN s0 @
zPushToHubMixin.push_to_hub)r r r rc r9 c C sF t |tr|}n"|r.tj }|d kr2tdnd }t|| ||d ddS )NzYou must login to the Hugging Face hub on this computer by typing `transformers-cli login` and entering your credentials to use `use_auth_token=True`. Alternatively, you can pass your own token as the `use_auth_token` argument.T)r r Z repo_typer )rL rM r r r? r )r r r rc r r# r# r$ r s
z&PushToHubMixin._get_repo_url_from_name)r r r r rc r9 c C s |d kr|d krt d|d kr,|d kr,d}|d krB|jdd }|d krttjj| rtt|j}| j||||d}tjj|stj| t |||d}|j
|S )Nz:You need to specify a `repo_path_or_name` or a `repo_url`.Tr: r )r r rc )Z
clone_fromrc r ) r? rp rN rO rQ r r r ru r Zgit_pull)clsr r r r rc r r r# r# r$ r s
z"PushToHubMixin._create_or_get_repo)r r r9 c C s8 |d kr,d| j krd}nd| j kr(d}nd}|j|dS )N Tokenizerz
add tokenizerZConfigz
add configz add model)r )r r )r r r r# r# r$ r s
zPushToHubMixin._push_to_hub)NNFNNNN)NNN)NNNNN)N)r r r r r rM boolr r staticmethodr classmethodr r r r# r# r# r$ r I s. 0\ $r )r5 r r c C sD |d krt j }|d kr2t|d }| d| S | d| S d S )Nr r: )r r r )r5 r r usernamer# r# r$ get_full_repo_name s r )NNN)N)N)N) NFNFNFFNF)N)Nr N)NFNr FNNF)NFFNNNF)NNNN)NNF)NN)sr r r iorT rN rs r r rn r warnings
contextlibr functoolsr hashlibr pathlibr typingr r r r r
r urllib.parser uuidr
zipfiler r r~ Zfilelockr Zhuggingface_hubr r r r r Zrequests.exceptionsr Ztransformers.utils.loggingr rh r r Z import_utilsr r r r r r
get_loggerr rk r r r r" r% r rO rP Ztorch_cache_homeZold_default_cache_path
expanduserZ
hf_cache_homeZdefault_cache_pathrr warningmover' r( r) r* Z TRANSFORMERS_DYNAMIC_MODULE_NAMEhexr r, ZS3_BUCKET_PREFIXZCLOUDFRONT_DISTRIB_PREFIXZ
_staging_modeZ_default_endpointr. warn
FutureWarningr@ r4 rM rC rJ r[ ra r r{ r r r r r r r rm PathLiker r r r r r r# r# r# r$ s
"
# [ Tb D. $6 7