/usr/local/lib/python3.6/site-packages/fsspec/implementations/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/fsspec/implementations/__pycache__/dbfs.cpython-36.pyc (13351B)
3
8%Eg:9 @ sd d dl Z d dlZd dlZd dlmZ d dlmZ G dd deZG dd deZ G dd d eZ
dS )
N)AbstractFileSystem)AbstractBufferedFilec s e Zd ZdZ fddZ ZS )DatabricksExceptionz<
Helper class for exceptions raised in this module.
c s t j| || _|| _dS )z Create a new DatabricksExceptionN)super__init__
error_codemessage)selfr r ) __class__ E/usr/local/lib/python3.6/site-packages/fsspec/implementations/dbfs.pyr s z DatabricksException.__init__)__name__
__module__ __qualname____doc__r
__classcell__r r )r
r r
s r c s e Zd ZdZ fddZd!ddZd"ddZd#d d
Zd$d d
Zd%ddZ d&ddZ
dd Z d'ddZ dd Z
dd Zd d Zd( fd d Z ZS ))DatabricksFileSystemz
Get access to the Databricks filesystem implementation over HTTP.
Can be used inside and outside of a databricks cluster.
c sB || _ || _tj | _| jjjdd| j i t jf | dS )a
Create a new DatabricksFileSystem.
Parameters
----------
instance: str
The instance URL of the databricks cluster.
For example for an Azure databricks cluster, this
has the form adb-
..azuredatabricks.net.
token: str
Your personal token. Find out more
here: https://docs.databricks.com/dev-tools/api/latest/authentication.html
AuthorizationzBearer N) instancetokenrequestsSessionsessionheadersupdater r )r r r kwargs)r
r r r s
z DatabricksFileSystem.__init__Tc C s | j |}|sy| jddd|id}W n: tk
r` } z |jdkrLt|j|W Y dd}~X nX |d }dd |D }|| j|< |r|S d
d |D S ) a"
List the contents of the given path.
Parameters
----------
path: str
Absolute path
detail: bool
Return not only the list of filenames,
but also additional information on file sizes
and types.
getlistpath)methodendpointjsonRESOURCE_DOES_NOT_EXISTNfilesc S s, g | ]$}|d |d rdnd|d dqS )r is_dir directoryfile file_size)nametypesizer ).0or r r
M s z+DatabricksFileSystem.ls..c S s g | ] }|d qS )r( r )r+ r, r r r r- X s )Z_ls_from_cache _send_to_apir r FileNotFoundErrorr dircache)r r Zdetailoutrer# r r r ls3 s
zDatabricksFileSystem.lsc C s |sZy(| j ddd|id td| dW n, tk
rX } z|jdkrHW Y dd}~X nX y| j d d
d|id W n: tk
r } z |jd krt|j|W Y dd}~X nX | j| j| dS ) aE
Create a given absolute path and all of its parents.
Parameters
----------
path: str
Absolute path to create
exist_ok: bool
If false, checks if the folder
exists before creating it (and raises an
Exception if this is the case)
r z
get-statusr )r r r! zPath z already existsr" NpostmkdirsRESOURCE_ALREADY_EXISTS)r. FileExistsErrorr r r invalidate_cache_parent)r r exist_okr3 r r r makedirsZ s
z DatabricksFileSystem.makedirsc K s |st | j|f| dS )a!
Create a given absolute path and all of its parents.
Parameters
----------
path: str
Absolute path to create
create_parents: bool
Whether to create all parents or not.
"False" is not implemented so far.
N)NotImplementedErrorr6 )r r Zcreate_parentsr r r r mkdir{ s zDatabricksFileSystem.mkdirFc C s y| j dd||dd W nT tk
rn } z8|jdkrF| j||d n|jdkrZt|j|W Y dd}~X nX | j| j| dS )z
Remove the file or folder at the given absolute path.
Parameters
----------
path: str
Absolute path what to remove
recursive: bool
Recursively delete all files in a folder.
r5 delete)r recursive)r r r! ZPARTIAL_DELETEZIO_ERRORN)r. r r rmOSErrorr r9 r: )r r r@ r3 r r r rA s
zDatabricksFileSystem.rmNc C s |rt |rt y| jdd||dd W nP tk
rz } z4|jdkrRt|jn|jdkrft|j|W Y dd}~X nX | j| j| | j| j| dS )a
Move a source to a destination path.
A note from the original [databricks API manual]
(https://docs.databricks.com/dev-tools/api/latest/dbfs.html#move).
When moving a large number of files the API call will time out after
approximately 60s, potentially resulting in partially moved data.
Therefore, for operations that move more than 10k files, we strongly
discourage using the DBFS REST API.
Parameters
----------
source_path: str
From where to move (absolute path)
destination_path: str
To where to move (absolute path)
recursive: bool
Not implemented to far.
maxdepth:
Not implemented to far.
r5 move) source_pathdestination_path)r r r! r" r7 N) r= r. r r r/ r r8 r9 r: )r rD rE r@ Zmaxdepthr3 r r r mv s"
zDatabricksFileSystem.mvrbdefaultc K s t | |f||d|S )z
Overwrite the base class method to make sure to create a DBFile.
All arguments are copied from the base method.
Only the default blocksize is allowed.
)mode
block_size)DatabricksFile)r r rI rJ r r r r _open s zDatabricksFileSystem._openc C s |dkr| j j}n |dkr$| j j}ntd| tjjd| j d|}|||d}y |j W nV t j
k
r } z8y|j j }W n t
k
r |Y nX tf |W Y dd}~X nX |j S )a
Send the given json to the DBFS API
using a get or post request (specified by the argument `method`).
Parameters
----------
method: str
Which http method to use for communication; "get" or "post".
endpoint: str
Where to send the request to (last part of the API URL)
json: dict
Dictionary of information to send
r5 r zDo not understand method zhttps://z/api/2.0/dbfs/)r! N)r r5 r
ValueErrorurllibparseurljoinr raise_for_statusr HTTPErrorresponser! Exceptionr ) r r r r! Z session_callurlr2 r3 Zexception_jsonr r r r. s
z!DatabricksFileSystem._send_to_apic C s^ y | j dd||dd}|d S tk
rX } z |jdkrDt|j|W Y dd}~X nX dS )a{
Internal function to create a handle, which can be used to
write blocks of a file to DBFS.
A handle has a unique identifier which needs to be passed
whenever written during this transaction.
The handle is active for 10 minutes - after that a new
write transaction needs to be created.
Make sure to close the handle after you are finished.
Parameters
----------
path: str
Absolute path for this file.
overwrite: bool
If a file already exist at this location, either overwrite
it or raise an exception.
r5 create)r overwrite)r r r! handler7 N)r. r r r8 r )r r rW r2 r3 r r r _create_handle s
z#DatabricksFileSystem._create_handlec C sX y| j ddd|id W n: tk
rR } z |jdkr>t|j|W Y dd}~X nX dS )z
Close a handle, which was opened by :func:`_create_handle`.
Parameters
----------
handle: str
Which handle to close.
r5 closerX )r r r! r" N)r. r r r/ r )r rX r3 r r r
_close_handle& s
z"DatabricksFileSystem._close_handlec C s~ t j|j }y| jdd||dd W nP tk
rx } z4|jdkrPt|jn|jdkrdt|j|W Y dd}~X nX dS )a
Upload data to an already opened file handle
(opened by :func:`_create_handle`).
The maximal allowed data size is 1MB after
conversion to base64.
Remember to close the handle when you are finished.
Parameters
----------
handle: str
Which handle to upload data to.
data: bytes
Block of data to add to the handle.
r5 z add-block)rX data)r r r! r" ZMAX_BLOCK_SIZE_EXCEEDEDN) base64 b64encodedecoder. r r r/ r rM )r rX r\ r3 r r r _add_data7 s
z DatabricksFileSystem._add_datac C s y*| j dd|||| dd}tj|d S tk
rz } z4|jdkrRt|jn|jd
krft|j|W Y d d }~X nX d S ) a
Download data in bytes from a given absolute path in a block
from [start, start+length].
The maximum number of allowed bytes to read is 1MB.
Parameters
----------
path: str
Absolute path to download data from
start: int
Start position of the block
end: int
End position of the block
r read)r offsetlength)r r r! r\ r" INVALID_PARAMETER_VALUEMAX_READ_SIZE_EXCEEDEDN)rd re )r. r] b64decoder r r/ r rM )r r startendr2 r3 r r r _get_dataU s
z DatabricksFileSystem._get_datac s2 |d kr| j j n| j j|d t j| d S )N)r0 clearpopr r9 )r r )r
r r r9 s s z%DatabricksFileSystem.invalidate_cache)T)T)T)F)FN)rG rH )T)N)r
r r r r r4 r< r> rA rF rL r. rY r[ r` ri r9 r r r )r
r r s
'
!
,
)
r c sH e Zd ZdZdZd fd
d Zd d
ZdddZdd Zdd Z Z
S )rK zH
Helper class for files referenced in the DatabricksFileSystem.
rG rH T readaheadNc sX |dks|dkr| j }|| j ks.td| t j||f|||||pHi d| dS )zr
Create a new instance of the DatabricksFile.
The blocksize needs to be the default one.
NrH z,Only the default block size is allowed, not )rI rJ
autocommit
cache_type
cache_options)DEFAULT_BLOCK_SIZEAssertionErrorr r ) r fsr rI rJ rp rq rr r )r
r r r s
zDatabricksFile.__init__c C s | j j| j| _dS )z(Internal function to start a file uploadN)ru rY r rX )r r r r _initiate_upload s z DatabricksFile._initiate_uploadFc sn | j jd | j j fdd| jt D }x |D ]}| jj| j|d q8W |rj| jj| jd dS dS )z.)rX r\ )rX TN) bufferseekgetvalue_to_sized_blockslenru r` rX r[ )r finalZ data_chunks
data_chunkr )r\ r
_upload_chunk s
z DatabricksFile._upload_chunkc C s@ d}|| }x.| j |D ] \}}|| jj| j||d7 }qW |S )z-Internal function to download a block of data )r rg rh )rz ru ri r )r rg rh Z
return_buffer total_lengthZ chunk_startZ chunk_endr r r _fetch_range s zDatabricksFile._fetch_rangec c s: x4t d|| jD ]"}|}t||| j }||fV qW dS )zFHelper function to split a range from 0 to total_length into bloksizesr N)range blocksizemin)r r r} Z
data_startZdata_endr r r rz s z DatabricksFile._to_sized_blocksi i )rG rH Tro N)F) r
r r r rs r rv r~ r rz r r r )r
r rK { s
rK ) r] rN r Zfsspecr Z fsspec.specr rT r r rK r r r r s
f