/usr/local/lib/python3.6/site-packages/fsspec/implementations/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/fsspec/implementations/__pycache__/dask.cpython-36.pyc (4656B)
3
8%EgY @ sx d dl Z d dlmZmZ d dlmZ d dlmZ d dlm Z m
Z
d dl m Z dd Z
G d d
d
e
ZG d d d e ZdS )
N)Client_get_global_client)
get_worker)
filesystem)AbstractBufferedFileAbstractFileSystem)infer_storage_optionsc C s( | d krt S t| tr | S t| S d S )N)r
isinstancer )client r E/usr/local/lib/python3.6/site-packages/fsspec/implementations/dask.py _get_client
s
r
c sp e Zd ZdZd fdd Zedd Zdd Zd d
Zd d Z d
d Z
dd Z dd Z dddZ
dd Z ZS )DaskWorkerFileSystema) View files accessible to a worker as any other remote file-system
When instances are run on the worker, uses the real filesystem. When
run on the client, they call the worker to provide information or data.
**Warning** this implementation is experimental, and read-only for now.
Nc sP t jf | |d k|d kA s&td|| _|| _d | _|| _|| _| j d S )NzKPlease provide one of filesystem instance (fs) or target_protocol, not both) super__init__
ValueErrortarget_protocoltarget_optionsworkerr
fs_determine_worker)selfr r r r
kwargs) __class__r r r s z DaskWorkerFileSystem.__init__c C s: t | }d|kr2d|kr2d|d d|d iS i S d S )Nhostportr
:)r )pathsor r r _get_kwargs_from_urls- s z*DaskWorkerFileSystem._get_kwargs_from_urlsc C sh y0t d| _| jd kr.t| jf| jp(i | _W n2 tk
rb d| _t| j| _t j
| | _ Y nX d S )NTF) r r r r r r r r
r
daskZdelayedrfs)r r r r r 5 s
z&DaskWorkerFileSystem._determine_workerc O s, | j r| jj|| n| jj||j d S )N)r r mkdirr! compute)r argsr r r r r" B s zDaskWorkerFileSystem.mkdirc O s, | j r| jj|| n| jj||j d S )N)r r rmr! r# )r r$ r r r r r% H s zDaskWorkerFileSystem.rmc O s, | j r| jj|| n| jj||j d S )N)r r copyr! r# )r r$ r r r r r& N s zDaskWorkerFileSystem.copyc O s, | j r| jj|| n| jj||j d S )N)r r mvr! r# )r r$ r r r r r' T s zDaskWorkerFileSystem.mvc O s* | j r| jj||S | jj||j S d S )N)r r lsr! r# )r r$ r r r r r( Z s zDaskWorkerFileSystem.lsrbTc K sD | j r$| jj|f||||d|S tf | |||||d|S d S )N)mode
block_size
autocommit
cache_options)r r r* r+ r, r- )r r _openDaskFile)r r r* r+ r, r- r r r r r. ` s zDaskWorkerFileSystem._openc
C sN | j r4| j||}|j| |j|| S Q R X n| jj||||j S d S )N)r r. seekreadr! fetch_ranger# )r r r* startendfr r r r2 } s
z DaskWorkerFileSystem.fetch_range)NNNN)r) NTN)__name__
__module__ __qualname____doc__r staticmethodr r r" r% r& r' r( r. r2
__classcell__r r )r r r s
r c s8 e Zd Zd fdd Zd ddZdd Zd d
Z ZS )
r/ r) c s" |dkrt dt jf | d S )Nr) z1Remote dask files can only be opened in "rb" mode)r r r )r r* r )r r r r s zDaskFile.__init__Fc C s d S )Nr )r finalr r r
_upload_chunk s zDaskFile._upload_chunkc C s dS )zCreate remote file/uploadNr )r r r r _initiate_upload s zDaskFile._initiate_uploadc C s | j j| j| j||S )z*Get the specified set of bytes from remote)r r2 r r* )r r3 r4 r r r _fetch_range s zDaskFile._fetch_range)r) )F)r6 r7 r8 r r= r> r? r; r r )r r r/ s
r/ )r Zdistributed.clientr r Zdistributed.workerr Zfsspecr Z fsspec.specr r Z fsspec.utilsr r
r r/ r r r r
s
r