Name	Size	Mode	Actions
add_new_model.cpython-36.pyc	6531	0644	editdlrm
add_new_model_like.cpython-36.pyc	42688	0644	editdlrm
convert.cpython-36.pyc	4943	0644	editdlrm
download.cpython-36.pyc	1660	0644	editdlrm
env.cpython-36.pyc	2605	0644	editdlrm
lfs.cpython-36.pyc	7244	0644	editdlrm
run.cpython-36.pyc	3372	0644	editdlrm
serving.cpython-36.pyc	6748	0644	editdlrm
train.cpython-36.pyc	4589	0644	editdlrm
transformers_cli.cpython-36.pyc	1261	0644	editdlrm
user.cpython-36.pyc	8453	0644	editdlrm
__init__.cpython-36.pyc	794	0644	editdlrm

3 �Eg�@s�dZd
dl
Z
d
dlZd
dlZd
dlZd
dlZd
dlmZ
d
dlm Z
d
dl mZmZm Z
d
dlZddlmZ
dd lmZ
eje�
Zd ZGdd�de�ZGd d�d�Zed�
dd�Ze ed�
dd�ZGdd�de �ZGdd�d�ZdS)a Implementation of a custom transfer agent for the transfer type "multipart" for git-lfs. Inspired by: github.com/cbartz/git-lfs-swift-transfer-agent/blob/master/git_lfs_swift_transfer.py Spec is: github.com/git-lfs/git-lfs/blob/master/docs/custom-transfers.md To launch debugger while developing: ``` [lfs "customtransfer.multipart"] path = /path/to/transformers/.env/bin/python args = -m debugpy --listen 5678 --wait-for-client /path/to/transformers/src/transformers/commands/transformers_cli.py lfs-multipart-upload ```�N)
�ArgumentParser)
�AbstractContextManager)�Dict�List�Optional�)
�logging�
)
�BaseTransformersCLICommandzlfs-multipart-uploadc@s"eZ
dZd
Zeed�
dd��
ZdS)�LfsCommandsu� Implementation of a custom transfer agent for the transfer type "multipart" for git-lfs. This lets users upload large files >5GB 🔥. Spec for LFS custom transfer agent is: https://github.com/git-lfs/git-lfs/blob/master/docs/custom-transfers.md This introduces two commands to the CLI: 1. $ transformers-cli lfs-enable-largefiles This should be executed once for each model repo that contains a model file >5GB. It's documented in the error message you get if you just try to git push a 5GB file without having enabled it before. 2. $ transformers-cli lfs-multipart-upload This command is called by lfs directly and is not meant to be called by the user. )
�parserc
CsP|jd
dd�}
|
j
dtdd�
|
jdd�d �

|jtd d�}|jdd�d �

dS)Nzlfs-enable-largefileszeDeprecated: use `huggingface-cli` instead. Configure your repository to enable upload of files > 5GB.)
�help�pathz/Local path to repository you want to configure.)�typer c

Sst|�
S)
N)
�LfsEnableCommand)
�args�r�C/usr/local/lib/python3.6/site-packages/transformers/commands/lfs.py�>sz1LfsCommands.register_subcommand..)
�funczgDeprecated: use `huggingface-cli` instead. Command will get called by git-lfs, do not call it directly.c

Sst|�
S)
N)
�LfsUploadCommand)
rrrrrEs)� add_parser�add_argument�str�set_defaults�LFS_MULTIPART_UPLOAD_COMMAND)rZ enable_parserZ upload_parserrrr�register_subcommand6s




zLfsCommands.register_subcommandN)�__name__� __module__�__qualname__�__doc__�staticmethodrrrrrrr$s
rc@seZ
dZd
d�Zdd�ZdS)rcCs |
|_dS)
N)
r)�selfrrrr�__init__Is
zLfsEnableCommand.__init__c
Csptj
d
�

tjj|jj�
}
tjj|
�
s6td�

td�

t j dj�d|
d�
t j dt��j�d|
d�
td�

dS) NzcManaging repositories through transformers-cli is deprecated. Please use `huggingface-cli` instead.z)This does not look like a valid git repo.r z=git config lfs.customtransfer.multipart.path transformers-cliT)�check�cwdz-git config lfs.customtransfer.multipart.args z Local repo set up for largefiles) �warnings�warn�osr�abspathr�isdir�print�exit� subprocess�run�splitr)r"Z local_pathrrrr.Ls









zLfsEnableCommand.runN)rrrr#r.rrrrrHs
r)
�msgc

Cs(tj
|�
d
}tjj|�

tjj�
dS)z-Write out the message in Line delimited JSON.�
N)�json�dumps�sys�stdout�write�flush)
r0rrr� write_msg_s

r8)
�returnc
CsVtj
tjj�j��
}d
|jd�
|jd�
fkr0dS|jd�
d krRtjd�

tj d�

|S) z$Read Line delimited JSON from stdin.� terminater�eventN�download�uploadzReceived unexpected messager )r<r=) r2�loadsr4�stdin�readline�strip�get�logger�criticalr,)
r0rrr�read_msgfs

rEc@sLeZ
dZd
Zeeed�dd�Zdd�Zdd�Zdd d�
Z dd �Z dd�ZdS)� FileSlicezq File-like object that only reads a slice of a file Inspired by stackoverflow.com/a/29838711/593036 )�filepath� seek_from� read_limitcCs|
|_||_
||_d
|_dS)Nr
)rGrHrI�n_seen)r"rGrHrIrrrr#|s



zFileSlice.__init__c

Cs t|j
d
�|_|jj|j�

|S)N�rb)�openrG�
f�seekrH)
r"rrr� __enter__�s


zFileSlice.__enter__c
Cs$tj
|jj��
j}
t|j|
|j�S)
N)r(�fstatrM�fileno�st_size�minrIrH)r"�total_lengthrrr�__len__�s

zFileSlice.__len__r cCsP|j|j
krd
S|j
|j}|jj|
dkr.|nt|
|��
}|jt|�
7_|S)N�r
)rJrIrM�readrS�len)r"�
nZremaining_amount�datarrrrW�s





zFileSlice.readc

cs|jdd�
V
dS)N�i)
rYii@)
rW)
r"rrr�__iter__�s
zFileSlice.__iter__c

Gs|jj
�
dS)
N)rM�close)r"rrrr�__exit__�s
zFileSlice.__exit__N�����)
r_)rrrr r�intr#rOrUrWr\r^rrrrrFus rFc@seZ
dZd
d�Zdd�ZdS)rcCs |
|_dS)
N)
r)r"rrrrr#�s
zLfsUploadCommand.__init__c
Csh
tj
tjj�j��
}
|
jd
�
dko.|
jd�
dksLtdddd�i
�

tjd �

ti�

�
xt �}|dkrptjd �

|d}|d}|d d}|d d}t |jd�
�
}t|j ��
}g} x~t|�
D]r\} }t|| ||d��R}tj||d�} | j�
| j| jjd�
| d d��

td|| d ||d��

WdQRXq�Wtj||| d�d�} | j�
td|d��

qXWdS)Nr;�initZ operationr=�error� zWrong lfs init operation)�code�messager r
�oidr�action�href�header� chunk_size)rHrI)
rZ�etag)rkZ partNumber�progress)r;rfZ bytesSoFarZbytesSinceLast)rf�parts)
r2Zcomplete)r;rf)r2r>r4r?r@rArBr8r,rEr`�pop�list�values� enumeraterF�requests�put�raise_for_status�append�headers�post)r"Zinit_msgr0rfrGZcompletion_urlrirjZpresigned_urlsrm�
iZ presigned_urlrZ�
rrrrr.�sD




 















zLfsUploadCommand.runN)rrrr#r.rrrrr�s
r)r r2r(r-r4r&�argparser� contextlibr�typingrrrrr�utilsr�r � get_loggerrrCrrrr8rErFrrrrr� s&







 $%