/usr/local/lib/python3.6/site-packages/fsspec/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/fsspec/__pycache__/caching.cpython-36.pyc (13325B)
3
8%Eg[A @ s d dl Z d dlZd dlZd dlZd dlZd dlZejdZG dd deZ G dd de Z
G dd de Z G d d
d
e Z G d d d e Z
G d
d de ZG dd de ZG dd de Ze e
ee e
e eedZdS ) NZfsspecc @ s$ e Zd ZdZdZdd Zdd ZdS ) BaseCacheag Pass-though cache: doesn't keep anything, calls every time
Acts as base class for other cachers
Parameters
----------
blocksize: int
How far to read ahead in numbers of bytes
fetcher: func
Function of the form f(start, end) which gets bytes from remote as
specified
size: int
How big this file is
nonec C s || _ || _|| _d S )N) blocksizefetchersize)selfr r r r 8/usr/local/lib/python3.6/site-packages/fsspec/caching.py__init__ s zBaseCache.__init__c C s< |d kr d}|d kr| j }|| j ks,||kr0dS | j||S )Nr )r r )r startstopr r r _fetch" s zBaseCache._fetchN)__name__
__module__ __qualname____doc__namer
r r r r r r s r c sF e Zd ZdZdZd fdd Zdd Zdd Zd
d Zd d
Z Z
S ) MMapCachezmemory-mapped sparse file cache
Opens temporary file, which is filled blocks-wise when data is requested.
Ensure there is enough disc space in the temporary location.
This cache method might only work on posix
mmapNc s8 t j||| |d kr t n|| _|| _| j | _d S )N)superr
setblockslocation _makefilecache)r r r r r r ) __class__r r r
7 s zMMapCache.__init__c C s dd l }dd l}| jdkr t S | jd ks:tjj| j r| jd krV|j }t | _
nt j | jd}|j
| jd |jd |j nt j | jd}|j |j | jS )Nr zwb+ 1zrb+)r tempfiler bytearrayr ospathexists
TemporaryFiler r ioopenseekwriteflushfileno)r r r fdr r r r = s
zMMapCache._makefilec s t jd| d| |d kr"d}|d kr0 j}| jksB||krFdS | j }| j } fddt||d D }xl|r|jd}| j }t| j j}t jd| d | d| j|| j||< j j
| qxW j|| S )
NzMMap cache fetching -r r c s g | ]}| j kr|qS r )r ).0i)r r r
] s z$MMapCache._fetch..r zMMap get block #z () loggerdebugr r rangepopminr r r add) r r endZ start_blockZ end_blockZneedr. Zsstartsendr )r r r S s$
zMMapCache._fetchc C s | j j }|d= |S )Nr )__dict__copy)r stater r r __getstate__j s
zMMapCache.__getstate__c C s | j j| | j | _d S )N)r8 updater r )r r: r r r __setstate__p s zMMapCache.__setstate__)NN) r r r r r r
r r r; r=
__classcell__r r )r r r , s r c s, e Zd ZdZdZ fddZdd Z ZS )ReadAheadCachea! Cache which reads only when we get beyond a block of data
This is a much simpler version of BytesCache, and does not attempt to
fill holes in the cache or keep fragments alive. It is best suited to
many small reads in a sequential order (e.g., reading lines from a file).
readaheadc s& t j||| d| _d| _d| _d S )Nr r )r r
r r r6 )r r r r )r r r r
s zReadAheadCache.__init__c C s |d kr d}|d ks || j kr$| j }|| j ks6||kr:dS || }|| jkrp|| jkrp| j|| j || j S | j| ko| jk n r| j|| j d }|t|8 }| j}nd}t| j || j }| j||| _|| _| jt| j | _|| jd | S )Nr r )r r r6 r lenr4 r r )r r r6 lpartr r r r s& zReadAheadCache._fetch)r r r r r r
r r> r r )r r r? v s r? c s, e Zd ZdZdZ fddZdd Z ZS )FirstChunkCachezCaches the first block of a file only
This may be useful for file types where the metadata is stored in the header,
but is randomly accessed.
firstc s t j||| d | _d S )N)r r
r )r r r r )r r r r
s zFirstChunkCache.__init__c C s |pd}|p| j }|| jk r| jd krh|| jkrX| jd|}|d | j | _||d S | jd| j| _| j|| }|| jkr|| j| j|7 }|S | j||S d S )Nr )r r r r )r r r6 datarC r r r r s
zFirstChunkCache._fetch)r r r r r r
r r> r r )r r rD s rD c sb e Zd ZdZdZd fdd Zdd Zdd Zd
d Zd d
Z dd Z
fddZ dd Z Z
S )
BlockCachea
Cache holding memory as a set of blocks.
Requests are only ever made `blocksize` at a time, and are
stored in an LRU cache. The least recently accessed block is
discarded when more than `maxblocks` are stored.
Parameters
----------
blocksize : int
The number of bytes to store in each block.
Requests are only ever made for `blocksize`, so this
should balance the overhead of making a request against
the granularity of the blocks.
fetcher : Callable
size : int
The total size of the file being cached.
maxblocks : int
The maximum number of blocks to cache for. The maximum memory
use for this cache is then ``blocksize * maxblocks``.
Z
blockcache c s< t j||| tj|| | _|| _tj|| j| _ d S )N)
r r
mathceilnblocks maxblocks functools lru_cache _fetch_block_fetch_block_cached)r r r r rL )r r r r
s zBlockCache.__init__c C s dj | j| j| jS )Nz.)formatr r rK )r r r r __repr__ s zBlockCache.__repr__c C s
| j j S )z
The statistics on the block cache.
Returns
----------
NamedTuple
Returned directly from the LRU Cache used internally.
)rP
cache_info)r r r r rS s zBlockCache.cache_infoc C s | j }|d= |S )NrP )r8 )r r: r r r r; s zBlockCache.__getstate__c C s& | j j| tj|d | j| _d S )NrL )r8 r< rM rN rO rP )r r: r r r r= s zBlockCache.__setstate__c C sx |d kr d}|d kr| j }|| j ks,||kr0dS || j }|| j }x t||d D ]}| j| qTW | j||||dS )Nr r r )start_block_numberend_block_number)r r r2 rP _read_cache)r r r6 rT rU block_numberr r r r s
zBlockCache._fetchc sN || j kr tdj|| j || j }|| j }tjd| t j||}|S )z=
Fetch the block of data for `block_number`.
z;'block_number={}' is greater than the number of blocks ({})z BlockCache fetching block %d)rK
ValueErrorrQ r r0 infor r )r rW r r6 Zblock_contents)r r r rO s
zBlockCache._fetch_blockc
C s || j }|| j }||kr2| j|}||| S g }|j| j||d x&t|d |D ]} |j| j| q^W |j| j|d| dj|S dS )z
Read from our block cache.
Parameters
----------
start, end : int
The start and end byte positions.
start_block_number, end_block_number : int
The start and end block numbers.
Nr r )r rP appendr2 join)
r r r6 rT rU start_posend_posblockoutrW r r r rV ! s
zBlockCache._read_cache)rH )r r r r r r
rR rS r; r= r rO rV r> r r )r r rG s rG c s6 e Zd ZdZdZd
fdd Zdd Zdd Z ZS )
BytesCacheaK Cache which holds data in a in-memory bytes object
Implements read-ahead by the block size, for semi-random reads progressing
through the file.
Parameters
----------
trim: bool
As we read more data, whether to discard the start of the buffer when
we are more than a blocksize ahead of it.
bytesTc s, t j||| d| _d | _d | _|| _d S )Nr )r r
r r r6 trim)r r r r rb )r r r r
S s
zBytesCache.__init__c C s& |d kr d}|d kr| j }|| j ks,||kr0dS | jd k rx|| jkrx| jd k rx|| jk rx|| j }| j||| | S | jrt| j || j }n|}||ks|| j krdS | jd ks|| jk r| jd ks|| jkr| j||| _|| _n|| jk r@| j| | jkr | j||| _|| _n | j|| j}|| _|| j | _n^|| jkr| j| j kr\nB|| j | jkr| j||| _|| _n| j| j|}| j| | _| jt| j | _|| j }| j||| | }| jr"| j| j | jd }|dkr"| j| j| 7 _| j| j| d | _|S )Nr r r ) r r r6 r r r4 r rA rb )r r r6 offsetZbendnewr_ numr r r r Z sZ
zBytesCache._fetchc C s
t | jS )N)rA r )r r r r __len__ s zBytesCache.__len__)T) r r r r r r
r rf r> r r )r r r` D s
=r` c s. e Zd ZdZdZd fdd Zdd Z ZS ) AllBytesz!Cache entire contents of the fileallNc s0 t j||| |d kr&| jd| j}|| _d S )Nr )r r
r r rF )r r r r rF )r r r r
s zAllBytes.__init__c C s | j || S )N)rF )r r r6 r r r r s zAllBytes._fetch)NNNN)r r r r r r
r r> r r )r r rg s rg c s6 e Zd ZdZdZi df fdd Z fddZ ZS )KnownPartsOfAFilea
Cache holding known file parts.
Parameters
----------
blocksize: int
How far to read ahead in numbers of bytes
fetcher: func
Function of the form f(start, end) which gets bytes from remote as
specified
size: int
How big this file is
data: dict
A dictionary mapping explicit `(start, stop)` file-offset tuples
with known bytes.
strict: bool, default True
Whether to fetch reads that go beyond a known byte-range boundary.
If `False`, any read that ends outside a known part will be zero
padded. Note that zero padding will not be used for reads that
begin outside a known byte-range.
partsTc s t t| j||| || _|rtt|j }|d g}|j|d g} xv|dd D ]f\}
} |d \} }
|
|
kr| | f|d< | d |j|
| f7 < qV|j|
| f | j|j|
| f qVW t t
|| | _ n|| _ d S )Nr r rk rk ) r ri r
strictsortedlistkeysr3 rZ dictziprF )r r r r rF rl _Z old_offsetsoffsetsr r r
Zstart0Zstop0)r r r r
s
zKnownPartsOfAFile.__init__c s d}x| j j D ]\\}}}|| ko.|k n r|| }|||| | }| j sp|| koj|kn r|d|| t| 7 }|S |}P qW | jd krtd||f dtjd||f d tj d| d| |t
j || S )Nr z&Read is outside the known file parts: z. z%. IO/caching performance may be poor!z!KnownPartsOfAFile cache fetching r, ) rF itemsrl rA r rX warningswarnr0 r1 r r )r r r
r_ Zloc0loc1rF off)r r r r s
zKnownPartsOfAFile._fetch)r r r r r r
r r> r r )r r ri s ri )r r ra r@ r^ rE rh rj )rM r% loggingrI r! rv getLoggerr0 objectr r r? rD rG r` rg ri cachesr r r r s.
!J* WV