/usr/local/lib/python3.6/site-packages/fsspec/implementations/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/fsspec/implementations/__pycache__/cached.cpython-36.pyc (23030B)
3
8%Eg7l @ s d dl Z d dlZd dlZd dlZd dlZd dlZd dlZd dlmZm Z d dl
m Z m Z d dl
mZ d dlmZ d dlmZmZ d dlmZ d dlmZ d d lmZ ejd
ZG d d d e Z G d
d de Z G dd de Z G dd dZ dd Z dS ) N)movermtree)AbstractFileSystem
filesystem)_DEFAULT_CALLBACK)compr) BaseCache MMapCache)BlocksizeMismatchError)AbstractBufferedFile)infer_compressionfsspecc s e Zd ZdZd)Zd* fd d
Zd d Zd
d Zdd Zdd Z dd Z
dd Z dd Z d+dd Z
d d Zd d Z fd!d"Zd#d$ Zd%d& Zd'd( Z ZS ),CachingFileSystema4 Locally caching filesystem, layer over any other FS
This class implements chunk-wise local storage of remote files, for quick
access after the initial download. The files are stored in a given
directory with random hashes for the filenames. If no directory is given,
a temporary one is used, which should be cleaned up by the OS after the
process ends. The files themselves as sparse (as implemented in
MMapCache), so only the data which is accessed takes up space.
Restrictions:
- the block-size must be the same for each access of a given file, unless
all blocks of the file have already been read
- caching can only be applied to file-systems which produce files
derived from fsspec.spec.AbstractBufferedFile ; LocalFileSystem is also
allowed, for testing
blockcachecachedNTMP
F: c
s t jf |
|dk|dkA s&td|dkr:tj g} nt|trL|g} n|} tj| d
dd | _ |pni _
| _ | _ | _
| _| _t|tr|nt|jtr|jn|jd _ j |dk r|n t|f j
_ fdd } | _dS ) a
Parameters
----------
target_protocol: str (optional)
Target filesystem protocol. Provide either this or ``fs``.
cache_storage: str or list(str)
Location to store files. If "TMP", this is a temporary directory,
and will be cleaned up by the OS when this process ends (or later).
If a list, each location will be tried in the order given, but
only the last will be considered writable.
cache_check: int
Number of seconds between reload of cache metadata
check_files: bool
Whether to explicitly see if the UID of the remote file matches
the stored one before using. Warning: some file systems such as
HTTP cannot reliably give a unique hash of the contents of some
path, so be sure to set this option to False.
expiry_time: int
The time in seconds after which a local copy is considered useless.
Set to falsy to prevent expiry. The default is equivalent to one
week.
target_options: dict or None
Passed to the instantiation of the FS, if fs is None.
fs: filesystem instance
The target filesystem to run against. Provide this or ``protocol``.
same_names: bool (optional)
By default, target URLs are hashed, so that files from different backends
with the same basename do not conflict. If this is true, the original
basename is used.
compression: str (optional)
To decompress on download. Can be 'infer' (guess from the URL name),
one of the entries in ``fsspec.compression.compr``, or None for no
decompression.
NzKPlease provide one of filesystem instance (fs) or remote_protocol, not bothr T)exist_okr c s j jt j| S )N)fs_strip_protocoltype)path)self G/usr/local/lib/python3.6/site-packages/fsspec/implementations/cached.pyr y s z3CachingFileSystem.__init__.
._strip_protocol)super__init__
ValueErrortempfilemkdtemp
isinstancestrosmakedirsstoragekwargs cache_check check_filesexpiry compression
same_namesprotocoltarget_protocol
load_cacher r r )
r r/ Z
cache_storager) r* expiry_timeZtarget_optionsr r- r, r( r' r ) __class__)r r r * s. 0
zCachingFileSystem.__init__c C s t j| jd dd d S )Nr T)r r )r% r& r' )r r r r _mkcache s zCachingFileSystem._mkcachec C s g }x| j D ]}tjj|d}tjj|rt|dJ}tj|}x.|j D ]"}t |d t
rJt |d |d< qJW |j | W dQ R X q |j i q W | j
|pi g| _tj | _dS )z#Read set of stored blocks from filecacherbblocksN)r' r% r joinexistsopenpickleloadvaluesr# listsetappendr3 cached_filestime
last_cache)r r@ r' fnfZloaded_cached_filescr r r r0 s
z CachingFileSystem.load_cachec
C s t jj| jd d}| jd
}t jj|rt|d}tj|}W dQ R X x|j D ]\}}||krV|d dks|| d dkrd|d< n"|| d }|j
|d ||d< t |d || d |d< || d |d< qVW x*|j D ]\}}||kr|||< qW n|}d d
|j D }x2|j D ]&}t
|d tr$t|d |d< q$W tj \}} t|d }tj|| W dQ R X | j t| | || jd< tj | _dS )z#Save set of stored blocks from filer r4 r5 Nr6 TrA uidc S s i | ]\}}|j |qS r )copy).0kvr r r
s z0CachingFileSystem.save_cache..wbr r r )r% r r7 r' r@ r8 r9 r: r; itemsupdatemaxr< r# r> r= r! mkstempdumpr3 r rA rB )
r rC r4 rD r@ rI rE r6 fd2Zfn2r r r
save_cache s:
z CachingFileSystem.save_cachec C sP | j | jsdS tj | j | jk}tdd | jD }|sD| rL| j dS )z0Reload caches if time elapsed or any disappearedNc s s | ]}t jj|V qd S )N)r% r r8 )rH r' r r r s z1CachingFileSystem._check_cache..)r3 r) rA rB allr' r0 )r ZtimecondZ existcondr r r _check_cache s
z CachingFileSystem._check_cachec C s | j |}| j xt| j| jD ]~\}}||kr4q"|| j }| jr\|d | jj|kr\q"| j rzt
j
|d | j krzq"t j j
||d }t j j|r"||fS q"W dS )z Is path in cache and still validrF rA rC F)r rV zipr' r@ rG r* r ukeyr+ rA r% r r7 r8 )r r r' r4 detailrC r r r _check_file s
z CachingFileSystem._check_filec C s t | jd | j dS )zRemove all files and metadat from the cache
In the case of multiple cache locations, this clears only the last one,
which is assumed to be the read/write one.
r Nr )r r' r0 )r r r r clear_cache s z CachingFileSystem.clear_cachec C sd | j |}| j|}|s dS |\}}|j| jd rXtj| | jd j| | j nt ddS )zRemove cached version of given file
Deletes local copy of the given (remote) path. If it is found in a cache
location which is not the last, it is assumed to be read-only, and
raises PermissionError
Nr zT s z)CachingFileSystem._open..r r ) r r _openrZ loggerdebugr9 hash_namer- r% r r7 r' r> rA rX r@ r3 r, r r rj r
r Z _fetch_rangesizer4 rl rS ) r r rd re rf rg r( rY rC hashr6 compr )rl rD r r rn sh
zCachingFileSystem._openc C s t ||dS )N) same_name)rq )r r ru r r r rq X s zCachingFileSystem.hash_namec C s |j r
dS | j|j}| jd | }|d dk rPt|d |j |jkrPd|d< y tjd | j tjd W n> t
t fk
r tjd Y n t k
r tjd Y nX | d|_ dS )
z9Called when a file is closing, so store the set of blocksNr r6 Tz
going to saveZsavedz&Cache saving failed while closing filez-Cache save failed due to interpreter shutdownr )
closedr r r@ lenrj rr ro rp rS IOErrorOSError NameError)r rD rl r rE r r r rk [ s $
z"CachingFileSystem.close_and_updatec s d&kr fd d S d'kr"t d(kr8tt S d"krHtS tjd#}|jd$d } |krp| S |d k r |jkr|j S t|}t| }tj|stj | rt
|d% s|j d k r|j ||S |S t
j S d S ))Nr0 rn rS rk r __getattribute__
__reduce___make_local_detailsr9 catcat_fileget
read_blocktailheadrZ rV r3 r[ rb
local_file_paths_from_path
get_mapper open_many commit_manyrq __hash____eq__to_jsonc s t t j| |S )N)getattrr __get__)argskw)itemr r r rm s z4CachingFileSystem.__getattribute__..
__reduce_ex___cacher2 __dict__r __self__) r0 rn rS rk r r{ r| r} r9 r~ r r r r r rZ rV r3 r[ rb r3 r r r r r rq r r r )r )r )AttributeErrorr r objectr{ r r inspect
isfunctionisdatadescriptorhasattrr r r )r r dr clsm)r2 )r r r r{ o sh
z"CachingFileSystem.__getattribute__c C s~ | |kr dS t |t| s dS | j|jko|| j|jko|| j|jko|| j|jko|| j|jko|| j|jko|| j|jko|| j |j kS )zTest for equality.TF)
r# r r' r( r) r* r+ r, r- r/ )r otherr r r r s zCachingFileSystem.__eq__c C sX t t| jt t| jA t | jA t | jA t | jA t | jA t | j A t | j
A S )zCalculate hash.) rs tupler' r$ r( r) r* r+ r, r- r/ )r r r r r s
zCachingFileSystem.__hash__c C s t ddS )z[Calculate JSON representation.
Not implemented yet for CachingFileSystem.
z5CachingFileSystem JSON representation not implementedN)NotImplementedError)r r r r r s zCachingFileSystem.to_json)r r ) Nr r Fr NNFN)r5 NTN)__name__
__module__ __qualname____doc__r. r r3 r0 rS rV rZ r[ rb rn rq rk r{ r r r
__classcell__r r )r2 r r s8 K*
U B
r c @ sJ e Zd ZdZdZdZdd Zdd Zdd Zd
d e fd d
Z
dddZ dS )WholeFileCacheFileSystema
Caches whole remote files on first access
This class is intended as a layer over any other file system, and
will make a local copy of each file accessed, so that all subsequent
reads are local. This is similar to ``CachingFileSystem``, but without
the block-wise functionality and so can work even when sparse files
are not allowed. See its docstring for definition of the init
arguments.
The class still needs access to the remote store for listing files,
and may refresh cached files.
filecacheTc s dd D }dj kr"j nfdd|D S jr@tfdd|D }dd t||D }fddt||D }dd t||D }|rވjj|| fd d|D }jd jd d t||D j d
d fddt||D S )Nc S s g | ]
}|j qS r )r )rH Zofr r r
s z6WholeFileCacheFileSystem.open_many..rc c s g | ]}t j| jd dqS )F)rd rf )
LocalTempFiler rd )rH r )
open_filesr r r r s c s g | ]} j |qS r )rZ )rH sp)r r r r s c S s g | ]\}}|s|qS r r )rH pr r r r r s c s. g | ]&\}}t jj jd j| jqS )r r )r% r r7 r' rq r- )rH r r )r r r r s c S s g | ]\}}|s|qS r r )rH rC r r r r r s c s2 g | ]*}| j | jd tj jj|dqS )T)rh rC r6 rA rF )rq r- rA r rX )rH r )r r r r s r c S s i | ]\}}||qS r r )rH r rY r r r rK
s z6WholeFileCacheFileSystem.open_many..c S s t | tr| d S | S )Nr )r# r )rC r r r firstpart s z5WholeFileCacheFileSystem.open_many..firstpartc s* g | ]"\}}t |r |n|jd qS ))rd )r9 rd )rH Zfn0Zfn1)r r r r r s r )
rd r3 r, r rW r r r@ rN rS )r r pathsr` ZdownpathZdownfn0ZdownfnZ newdetailr )r r r r r s,
z"WholeFileCacheFileSystem.open_manyc C s& | j jdd |D dd |D d S )Nc S s g | ]
}|j qS r )rC )rH rD r r r r s z8WholeFileCacheFileSystem.commit_many..c S s g | ]
}|j qS r )r )rH rD r r r r s )r put)r r r r r r s z$WholeFileCacheFileSystem.commit_manyc C s^ | j || j}tjj| jd |}||dtj | jj|d}|| j d |< t
j d| |S )Nr T)rh rC r6 rA rF zCopying %s to local cacher r ) rq r- r% r r7 r' rA r rX r@ ro rp )r r rs rC rY r r r r} s z,WholeFileCacheFileSystem._make_local_detailsFraisec K s | j |||jdd d}g }g }g } x^|D ]V}
| j|
} | s\| j|
} |j|
|j| nt| trj| nd | f\} } | j| q*W |r| jj|| | j i }
|j t
| x2t || D ]$\}
} t | dj
|
|
< |jd qW t|trt
|dkr|dkr|
|d }
|
S )Nmaxdepth) recursiver r5 r Fr )Z expand_pathr rZ r} r? r# r r rS Zset_sizerw rW r9 readZrelative_updater$ )r r r Zon_errorcallbackr( r ZgetpathsZ
storepathsfnsr rY rC outr r r r~ + s0
$ z WholeFileCacheFileSystem.catr5 c
K s | j |}d|kr*| jj|fd|i|S | j|}|r|\}}|d |d }}|dkrtjd| t||}|jd|_|S t d| n
| j
|}||d< | j | j r`| jj|f|}t|d } t
|trtd
|jj|j|_| j d krt|n| j }
t|
|d d
}d} x,| rHt|dd} |j| } | j| q W W d Q R X W d Q R X n| jj|| | j | j||S )Nrc rd rC r6 TzOpening local copy of %srh z?Attempt to open partially cached file %sas a wholly cached filerL r ri r5 )rd rj i i P )r r rn rZ ro rp r9 r rh r r} r3 r, r# r r r4 fetcherrr r r r r writerS )
r r rd r( rY rC ra r6 rD f2rt datablockr r r rn O sB
&z WholeFileCacheFileSystem._openN)r5 ) r r r r r. r r r r} r r~ rn r r r r r s 0 r c sJ e Zd ZdZdZdZ fddZdd Zdd Zd
d Z dd
dZ
Z S )SimpleCacheFileSystema Caches whole remote files on first access
This class is intended as a layer over any other file system, and
will make a local copy of each file accessed, so that all subsequent
reads are local. This implementation only copies whole files, and
does not keep any metadata about the download time or file details.
It is therefore safer to use in multi-threaded/concurrent situations.
This is the only of the caching filesystems that supports write: you will
be given a real local open file, and upon close and commit, it will be
uploaded to the target filesystem; the writability or the target URL is
not checked until that time.
Z simplecacheTc sb |j }xdD ] }d||< qW t jf | x(| jD ] }tjj|s4tj|dd q4W i g| _d S )Nr) r1 r* FT)r )r) r1 r* ) rG r r r' r% r r8 r& r@ )r r( r keyr' )r2 r r r s
z SimpleCacheFileSystem.__init__c C sH | j | j|| j}x,| jD ]"}tjj||}tjj|r |S q W d S )N)rV rq r- r' r% r r7 r8 )r r shar' rC r r r rZ s z!SimpleCacheFileSystem._check_filec C s d S )Nr )r r r r rS s z SimpleCacheFileSystem.save_cachec C s d S )Nr )r r r r r0 s z SimpleCacheFileSystem.load_cacher5 c K sD | j |}d|kr t| ||dS | j|}|r8t||S | j|| j}tjj| j d |}t
j d| ||d< | j | j
r*| jj|f|}t|d|}t|trtd|jj|j|_| j
dkrt|n| j
}t| |d d}d
} x*| rt|d d}
|j|
} |j| qW W d Q R X W d Q R X n| jj|| | j||S )Nrc )rd r zCopying %s to local cacherd rL r ri r5 Trj r r r r i i P ) r r rZ r9 rq r- r% r r7 r' ro rp r3 r, r rn r# r r r4 r rr r r r r r r ) r r rd r( rC r rD r rt r r r r r rn s2
$zSimpleCacheFileSystem._open)r5 ) r r r r r. r r rZ rS r0 rn r r r )r2 r r s
r c @ s^ e Zd ZdZdddZdd Zd
d Zd d
Zdd Zdd Z dd Z
e dd Z dd Z
dS )r z8A temporary local file, which will be uploaded on commitNrL Tr c C sd |r|| _ t||| _ntj \}| _ t||| _|| _|rH| jj| || _|| _d| _ || _
d S )NF) rC r9 fhr! rP rd seekr r rv rf )r r r rC rd rf r fdr r r r s zLocalTempFile.__init__c C s t | j| j| jd| j| j ffS )Nzrb+)r r r rC rf tell)r r r r r| s zLocalTempFile.__reduce__c C s | j S )N)r )r r r r __enter__ s zLocalTempFile.__enter__c C s | j d S )N)rl )r exc_typeexc_valexc_tbr r r __exit__ s zLocalTempFile.__exit__c C s" | j j d| _| jr | j d S )NT)r rl rv rf commit)r r r r rl s
zLocalTempFile.closec C s | j j tj| j d S )N)r rl r% r] rC )r r r r discard s
zLocalTempFile.discardc C s | j j| j| j d S )N)r r rC r )r r r r r s zLocalTempFile.commitc C s t | jjtr| jjS | jS d S )N)r# r namer$ rC )r r r r r s zLocalTempFile.namec C s t | j|S )N)r r )r r r r r __getattr__ s zLocalTempFile.__getattr__)NrL Tr )r r r r r r| r r rl r r propertyr r r r r r r s
r c C s( |rt jj| }ntj| j j }|S )N)r% r basenamehashlibsha256encode hexdigest)r ru rs r r r rq s rq )!r r loggingr% r: r! rA shutilr r r
r r Zfsspec.callbacksr Zfsspec.compressionr Z fsspec.corer r Zfsspec.exceptionsr
Z fsspec.specr Z fsspec.utilsr getLoggerro r r r r rq r r r r s0
G +O7