/usr/local/lib/python3.6/site-packages/datasets/filesystems/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/datasets/filesystems/__pycache__/hffilesystem.cpython-36.pyc (3226B)
3
<%Eg @ s\ d dl mZ d dlmZ d dlZd dlmZ d dlmZ ddlm Z m
Z
G dd d eZ dS )
)
PurePosixPath)OptionalN)AbstractFileSystem) DatasetInfo )"get_authentication_headers_for_url
hf_hub_urlc sf e Zd ZdZdZdZdee ee d fdd
Z dd Z
deed d d
Z dd Z dddZ
ZS ) HfFileSystemz/Interface to files in a Hugging face repository ZhfN) repo_infotokenc s& t j| f| || _|| _d| _dS )a
The file system can be instantiated using a huggingface_hub.hf_api.DatasetInfo object,
and can be used to list and open files from a Hugging Face dataset repository with fsspec.
Args:
repo_info (:obj:``DatasetInfo``, `optional`):
Dataset repository info from huggingface_hub.HfApi().dataset_info(...)
token (:obj:``str``, `optional`):
Hugging Face token. Will default to the locally saved token if not provided.
N)super__init__r r dir_cache)selfr r kwargs) __class__ K/usr/local/lib/python3.6/site-packages/datasets/filesystems/hffilesystem.pyr s zHfFileSystem.__init__c C sf | j d krbi | _ xP| jjD ]D}|jd dd| j |j< | j jdd tt|jjd d D qW d S )Nfile)namesizetypec S s" i | ]}t |d ddt |qS )N directory)r r r )str).0dr r r
1 s z*HfFileSystem._get_dirs.. )r r ZsiblingsZ rfilenameupdatelistr parents)r Zhf_filer r r _get_dirs& s
zHfFileSystem._get_dirsrb)pathmodec K sP t | jts td| j t| jj|| jjd}tj||t || j
ddj S )Nz;Open is only implemented for dataset repositories, but got )revision)Zuse_auth_token)r& headers)
isinstancer r NotImplementedErrorr idshafsspecopenr r )r r% r& r urlr r r _open7 s zHfFileSystem._openc K s2 | j | j|}|| jkr&| j| S t|d S )N)r# Z_strip_protocolr FileNotFoundError)r r% r r r r infoF s
zHfFileSystem.infoFc K s | j t|jd}i }x>| jj D ]0\}}t|jd}|j}||kr&||t|< q&W t|j }|rn|S tt dd |D S d S )N/c s s | ]}|d V qdS )r Nr )r fr r r [ s z"HfFileSystem.ls..)
r# r stripr itemsparentr r! valuessorted) r r% Zdetailr pathspr4 rootoutr r r lsN s zHfFileSystem.ls)NN)r$ )F)__name__
__module__ __qualname____doc__Z root_markerprotocolr r r r r# r0 r2 r?
__classcell__r r )r r r s r ) pathlibr typingr r- r Zhuggingface_hub.hf_apir Zutils.file_utilsr r r r r r r s