FM / __pycache_

/usr/local/lib/python3.6/site-packages/fsspec/implementations/__pycache__/

/usr/local/lib/python3.6/site-packages/fsspec/implementations/__pycache__

Name	Size	Mode	Actions
arrow.cpython-36.pyc	7328	0644	edit dl rm
cached.cpython-36.pyc	23030	0644	edit dl rm
dask.cpython-36.pyc	4656	0644	edit dl rm
dbfs.cpython-36.pyc	13351	0644	edit dl rm
dvc.cpython-36.pyc	2622	0644	edit dl rm
ftp.cpython-36.pyc	10444	0644	edit dl rm
git.cpython-36.pyc	3750	0644	edit dl rm
github.cpython-36.pyc	7404	0644	edit dl rm
hdfs.cpython-36.pyc	7093	0644	edit dl rm
http.cpython-36.pyc	20752	0644	edit dl rm
jupyter.cpython-36.pyc	4131	0644	edit dl rm
libarchive.cpython-36.pyc	6005	0644	edit dl rm
local.cpython-36.pyc	11912	0644	edit dl rm
memory.cpython-36.pyc	6697	0644	edit dl rm
reference.cpython-36.pyc	16142	0644	edit dl rm
sftp.cpython-36.pyc	6104	0644	edit dl rm
smb.cpython-36.pyc	10591	0644	edit dl rm
tar.cpython-36.pyc	4502	0644	edit dl rm
webhdfs.cpython-36.pyc	14712	0644	edit dl rm
zip.cpython-36.pyc	3276	0644	edit dl rm
__init__.cpython-36.pyc	156	0644	edit dl rm

Edit: /usr/local/lib/python3.6/site-packages/fsspec/implementations/__pycache__/webhdfs.cpython-36.pyc (14712B)

3 8%Eg�:�@s�ddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl Z ddlmZm Z ddlmZejd�ZGdd �d e �ZGd d�de�ZdS)�N)�suppress)�quote�)�AbstractBufferedFile�AbstractFileSystem)�infer_storage_options�webhdfscseZdZdZeej��Zdd d�Zd?dd�Ze dd��Zedd��Ze dd��Zdd�Zd@dd�Zdd�Zdd�Zd d!�ZdAd"d#�Zd$d%�Zd&d'�Zd(d)�ZdBd*d+�Zd,d-�Zd.d/�ZdCd0d1�Zd2d3�ZdDd4d5�Z d6d7�Z!d8d9�Z"d:d;�Z#�Z$S)E�WebHDFSa| Interface to HDFS over HTTP using the WebHDFS API. Supports also HttpFS gateways. Three auth mechanisms are supported: insecure: no auth is done, and the user is assumed to be whoever they say they are (parameter `user`), or a predefined value such as "dr.who" if not given spnego: when kerberos authentication is enabled, auth is negotiated by requests_kerberos https://github.com/requests/requests-kerberos . This establishes a session based on existing kinit login and/or specified principal/password; parameters are passed with ``kerb_kwargs`` token: uses an existing Hadoop delegation token from another secured service. Indeed, this client can also generate such tokens when not insecure. Note that tokens expire, but can be renewed (by a previously specified user) and may allow for proxying. r�webHDFS��FNc s�|jr dSt�jf| �dj| r$dnd||d�|_||_|p>i|_i|_|pNi|_|dk r||dk sj|dk rrt d��||jd<|dk r�||jd<|dk r�||jd <|r�|dk r�t d ��|j �dS)a� Parameters ---------- host: str Name-node address port: int Port for webHDFS kerberos: bool Whether to authenticate with kerberos for this connection token: str or None If given, use this token on every call to authenticate. A user and user-proxy may be encoded in the token and should not be also given user: str or None If given, assert the user name to connect with proxy_to: str or None If given, the user has the authority to proxy, and this value is the user in who's name actions are taken kerb_kwargs: dict Any extra arguments for HTTPKerberosAuth, see ``_ data_proxy: dict, callable or None If given, map data-node addresses. This can be necessary if the HDFS cluster is behind a proxy, running on Docker or otherwise has a mismatch between the host-names given by the name-node and the address by which to refer to them from the client. If a dict, maps host names `host->data_proxy[host]`; if a callable, full URLs are passed, and function must conform to `url->data_proxy(url)`. use_https: bool Whether to connect to the Name-node using HTTPS instead of HTTP kwargs Nz%{protocol}://{host}:{port}/webhdfs/v1�https�http)�protocol�host�portz_If passing a delegation token, must not set user or proxy_to, as these are encoded in the tokenZ delegationz user.nameZdoaszJIf using Kerberos auth, do not specify the user, this is handled by kinit.)�_cached�super�__init__�format�url�kerb�kerb_kwargs�pars�proxy� ValueError�_connect)�selfrrZkerberos�token�userZproxy_torZ data_proxyZ use_https�kwargs)� __class__��H/usr/local/lib/python3.6/site-packages/fsspec/implementations/webhdfs.pyr+s,. zWebHDFS.__init__cCs0tj�|_|jr,ddlm}|f|j�|j_dS)Nr)�HTTPKerberosAuth)�requests�Session�sessionrZrequests_kerberosr#r�auth)rr#r!r!r"rvs zWebHDFS._connect�getTc Ks�|jt|pd�}|j�}|j|j�|j�|d<tjd||�|jj |j�||||d�} | j dkr�y$| j�} | d d}| d d}Wntt fk r�Yn@X|dkr�t|��n,|dkr�t|��n|dkr�t|��nt|��| j�| S)N��opzsending %s with %s)�methodr�params�data�allow_redirects��ZRemoteException�message� exception�IllegalArgumentException�UnsupportedOperationException�SecurityException�AccessControlException�FileNotFoundException)r/r0r1r2r3)r6r7)r8r9)r:)rr�copy�updater�upper�logger�debugr&�request�status_code�jsonr�KeyError�PermissionError�FileNotFoundError�RuntimeError�raise_for_status) rr*r+�pathr-�redirectrr�args�out�err�msg�expr!r!r"�_call}s6 z WebHDFS._call�rbc Ks$|p|j}t|||||j|||d�S)a^ Parameters ---------- path: str File location mode: str 'rb', 'wb', etc. block_size: int Client buffer size for read-ahead or write buffer autocommit: bool If False, writes to temporary file that only gets put in final location upon commit replication: int Number of copies of file on the cluster, write mode only permissions: str or int posix permissions, write mode only kwargs Returns ------- WebHDFile instance )�mode� block_size�tempdir� autocommit�replication�permissions)� blocksize� WebHDFilerS)rrHrQrRrTrUrVrr!r!r"�_open�s! z WebHDFS._opencCs |dj�|d<|d|d<|S)N�type�length�size)�lower)�infor!r!r"� _process_info�szWebHDFS._process_infocCst|�dS)NrH)r)�clsrHr!r!r"�_strip_protocol�szWebHDFS._strip_protocolcCs:t|�}|jdd�|jdd�d|kr6|jd�|d<|S)NrHr�usernamer)r�pop)ZurlpathrKr!r!r"�_get_kwargs_from_urls�szWebHDFS._get_kwargs_from_urlscCs,|jd|d�}|j�d}||d<|j|�S)NZ GETFILESTATUS)rH� FileStatus�name)rOrBr_)rrHrKr^r!r!r"r^�szWebHDFS.infocCsz|jd|d�}|j�dd}x0|D](}|j|�|jd�d|d|d<q$W|rdt|dd �d �Stdd�|D��SdS) NZ LISTSTATUS)rHZFileStatusesre�/Z pathSuffixrfcSs|dS)Nrfr!)�ir!r!r"��szWebHDFS.ls..)�keycss|]}|dVqdS)rfNr!)�.0r^r!r!r"� �szWebHDFS.ls..)rOrBr_�rstrip�sorted)rrHZdetailrK�infosr^r!r!r"�ls�s z WebHDFS.lscCs|jd|d�}|j�dS)z8Total numbers of files, directories and bytes under pathZGETCONTENTSUMMARY)rHZContentSummary)rOrB)rrHrKr!r!r"�content_summary�szWebHDFS.content_summarycCsb|jd|dd�}d|jkrJ|j|jd�}|jj|�}|j�|j�dS|j�|j�dSdS)z/Checksum info of file, giving method and resultZGETFILECHECKSUMF)rHrI�LocationZFileChecksumN)rO�headers�_apply_proxyr&r(rGrB)rrHrK�location�out2r!r!r"�ukey�s zWebHDFS.ukeycCs|jd�}|j�dS)zGet user's home directoryZGETHOMEDIRECTORY�Path)rOrB)rrKr!r!r"�home_directory�s zWebHDFS.home_directorycCsB|r|jd|d�}n |jd�}|j�d}|dkr:td��|dS)z�Retrieve token which can give the same authority to other uses Parameters ---------- renewer: str or None User who may use this token; if None, will be current user ZGETDELEGATIONTOKEN)�renewer�TokenNz1No token available for this user/security contextZ urlString)rOrBr)rrzrK�tr!r!r"�get_delegation_tokens zWebHDFS.get_delegation_tokencCs|jdd|d�}|j�dS)z/Make token live longer. Returns new expiry timeZRENEWDELEGATIONTOKEN�put)r+r�long)rOrB)rrrKr!r!r"�renew_delegation_tokenszWebHDFS.renew_delegation_tokencCs|jdd|d�dS)z Stop the token from being usefulZCANCELDELEGATIONTOKENr~)r+rN)rO)rrr!r!r"�cancel_delegation_tokenszWebHDFS.cancel_delegation_tokencCs|jdd||d�dS)aSet the permission at path Parameters ---------- path: str location to set (file or directory) mod: str or int posix epresentation or permission, give as oct string, e.g, '777' or 0o777 Z SETPERMISSIONr~)r+rHZ permissionN)rO)rrH�modr!r!r"�chmodsz WebHDFS.chmodcCs>i}|dk r||d<|dk r$||d<|jdd|d�|��dS)zChange owning user and/or groupN�owner�group�SETOWNERr~)r+rH)r�)rO)rrHr�r�rr!r!r"�chown+sz WebHDFS.chowncCs|jd|d|d�dS)a9 Set file replication factor Parameters ---------- path: str File location (not for directories) replication: int Number of copies of file on the cluster. Should be smaller than number of data nodes; normally 3 on most systems. ZSETREPLICATIONr~)rHr+rUN)rO)rrHrUr!r!r"�set_replication4szWebHDFS.set_replicationcKs|jdd|d�dS)NZMKDIRSr~)r+rH)rO)rrHrr!r!r"�mkdirBsz WebHDFS.mkdircCs(|dkr|j|�rt|��|j|�dS)NF)�exists�FileExistsErrorr�)rrH�exist_okr!r!r"�makedirsEszWebHDFS.makedirscKs|jdd||d�dS)NZRENAMEr~)r+rH�destination)rO)rZpath1Zpath2rr!r!r"�mvJsz WebHDFS.mvcKs|jdd||rdndd�dS)N�DELETE�delete�true�false)r+rH� recursive)rO)rrHr�rr!r!r"�rmMs z WebHDFS.rmcKs|j|�dS)N)r�)rrHrr!r!r"�rm_fileUszWebHDFS.rm_filecKs�|j|��}dj|j|�dtjd��g�}y4|j|d��}tj||�WdQRX|j||�Wn4tk r�t t ��|j|�WdQRX�YnXWdQRXdS)Nrgz.tmp.��wb)�open�join�_parent�secretsZ token_hex�shutil�copyfileobjr�� BaseExceptionrrEr�)rZlpathZrpathrZlstreamZ tmp_fnameZrstreamr!r!r"�cp_fileXs zWebHDFS.cp_filecCsL|jrt|j�r|j|�}n,|jrHx$|jj�D]\}}|j||d�}q.W|S)N�)r�callable�items�replace)rru�k�vr!r!r"rtfszWebHDFS._apply_proxy)rr )rFNNNNNF)r(NNT)rPNTNN)F)N)NN)F)F)%�__name__� __module__�__qualname__�__doc__�str�tempfile� gettempdirrSrrrrOrY�staticmethodr_�classmethodrardr^rprqrwryr}r�r�r�r�r�r�r�r�r�r�r�rt� __classcell__r!r!)r r"r sNA # & r csJeZdZdZ�fdd�Zddd�Zdd�Zd d �Zdd�Zd d�Z �Z S)rXz"A file living in HDFS over webHDFScs�t�j||f|�|j�}|jdd�dkr6|jdd�|jdd�dkrR|jdd�|jdd�|_|jd�}|jdd�dkr�|j|_tjj |t tj��|_dS)NrVrUi�rSrTF) rrr;r(rcrVrH�target�osr�r��uuid�uuid4)r�fsrHrrS)r r!r"rss zWebHDFile.__init__FcCs,|jjj|j|jj�ddid�}|j�dS)z�Write one part of a multi-block file upload Parameters ========== final: bool This is the last block, so should complete file, if self.autocommit is True. zcontent-typezapplication/octet-stream)r-rsT)r�r&�postru�buffer�getvaluerG)r�finalrKr!r!r"� _upload_chunk�s zWebHDFile._upload_chunkcCs�|jj�}d|jkrd\}}nd\}}d|d<|jj|||jfdd i|��}|jj|jd �}d|jkr�|jjj |dd id�}|j �|jdd�|_dS)zCreate remote file/upload�a�APPEND�POST�CREATE�PUTr�� overwriterIFrr�wzcontent-typezapplication/octet-stream)rsN)r�r�)r�r�) rr;rQr�rOrHrtrsr&r~rGr�ru)rrr*r+rKrurvr!r!r"�_initiate_upload�s zWebHDFile._initiate_uploadcCs�t|d�}t|j|�}||ks(||jkr,dS|jjd|j|||dd�}|j�d|jkr�|jd}|jjj |jj |��}|jS|jSdS)Nr�ZOPENF)rH�offsetr[rIrr)�max�minr\r�rOrHrGrsr&r(rt�content)r�start�endrKrurvr!r!r"�_fetch_range�s zWebHDFile._fetch_rangecCs|jj|j|j�dS)N)r�r�rHr�)rr!r!r"�commit�szWebHDFile.commitcCs|jj|j�dS)N)r�r�rH)rr!r!r"�discard�szWebHDFile.discard)F)r�r�r�r�rr�r�r�r�r�r�r!r!)r r"rXps rX)�loggingr�r�r�r�r�� contextlibr�urllib.parserr$�specrr�utilsr� getLoggerr>r rXr!r!r!r"�s ^