FM / __pycache_

/usr/local/lib/python3.6/site-packages/datasets/__pycache__/

/usr/local/lib/python3.6/site-packages/datasets/__pycache__

Name	Size	Mode	Actions
arrow_dataset.cpython-36.pyc	189850	0644	edit dl rm
arrow_reader.cpython-36.pyc	22483	0644	edit dl rm
arrow_writer.cpython-36.pyc	22110	0644	edit dl rm
builder.cpython-36.pyc	52779	0644	edit dl rm
combine.cpython-36.pyc	5508	0644	edit dl rm
config.cpython-36.pyc	5181	0644	edit dl rm
dataset_dict.cpython-36.pyc	83319	0644	edit dl rm
data_files.cpython-36.pyc	30153	0644	edit dl rm
fingerprint.cpython-36.pyc	18979	0644	edit dl rm
info.cpython-36.pyc	17359	0644	edit dl rm
inspect.cpython-36.pyc	18583	0644	edit dl rm
iterable_dataset.cpython-36.pyc	62932	0644	edit dl rm
keyhash.cpython-36.pyc	3341	0644	edit dl rm
load.cpython-36.pyc	62922	0644	edit dl rm
metric.cpython-36.pyc	23217	0644	edit dl rm
naming.cpython-36.pyc	2571	0644	edit dl rm
search.cpython-36.pyc	31339	0644	edit dl rm
splits.cpython-36.pyc	22522	0644	edit dl rm
streaming.cpython-36.pyc	4119	0644	edit dl rm
table.cpython-36.pyc	79612	0644	edit dl rm
__init__.cpython-36.pyc	2282	0644	edit dl rm

Edit: /usr/local/lib/python3.6/site-packages/datasets/__pycache__/streaming.cpython-36.pyc (4119B)

3 <%Eg��@s�ddlZddlZddlmZddlmZmZmZddlm Z m Z mZmZm Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZddlmZddl m!Z!ee"�Z#er�dd l$m%Z%deee&e'fd �dd�Z(d d�dd�Z)dS)�N)�wraps)� TYPE_CHECKING�Optional�Union�)� xbasename�xdirname� xet_parse�xgetsize�xglob�xisdir�xisfile�xjoin�xlistdir�xopen�xpandas_read_csv�xpandas_read_excel�xPath�xrelpath�xsio_loadmat�xsplit� xsplitext�xwalk�xxml_dom_minidom_parse)� get_logger)�patch_submodule)�get_imports)�DatasetBuilder)�use_auth_tokencs�tj|�}t|d�r|jrdS�fdd�}t|d|t��j�t|d|t��j�t|d|t��j�t|d|t ��j�t|d t �j�t|d t�j�t|dt�j�t|dt �j�t|d t�j�t|dt�j�t|d|t��j�t|d|t��j�t|d|t��j�t|dt�j�t|d|t�dgd�j�t|dtdgd�j�t|d|t�dgd�j�t|d|t��j�t|d|t��j�d|_dS)aExtend the module to support streaming. We patch some functions in the module to use `fsspec` to support data streaming: - We use `fsspec.open` to open and read remote files. We patch the module function: - `open` - We use the "::" hop separator to join paths and navigate remote compressed/archive files. We patch the module functions: - `os.path.join` - `pathlib.Path.joinpath` and `pathlib.Path.__truediv__` (called when using the "/" operator) The patched functions are replaced with custom functions defined to work with the :class:`~download.streaming_download_manager.StreamingDownloadManager`. Args: module_path: Path to the module to be extended. use_auth_token: Whether to use authentication token. �_patched_for_streamingNcs t��fdd��}d|_|S)Ncs�|d�i|��S)Nr�)�args�kwargs)�functionrr �.wrap_auth..wrapper� wrap_auth)rZ_decorator_name_)r#r%)r)r#r$r&@sz.extend_module_for_streaming..wrap_auth�openz os.listdirzos.walkz glob.globzos.path.joinzos.path.dirnamezos.path.basenamezos.path.relpathz os.path.splitzos.path.splitextz os.path.isdirzos.path.isfilezos.path.getsizezpathlib.Pathzpandas.read_csv�__version__)�attrszpandas.read_excelzscipy.io.loadmatzxml.etree.ElementTree.parsezxml.dom.minidom.parseT)� importlib� import_module�hasattrrrr�startrrrrrrrrrrr r rrrrr r)�module_pathr�moduler&r )rr$�extend_module_for_streaming's0 r0r)�buildercCs|t|j|jd�|jjd�sxxZttj|j��D]F}|ddkr.|d}dj|jj d�dd�|g�}t||jd�q.WdS) a{Extend the dataset builder module and the modules imported by it to support streaming. Args: builder (:class:`DatasetBuilder`): Dataset builder instance. use_auth_token (``str`` or :obj:`bool`, optional): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub. If True, will get token from `"~/.huggingface"`. )rz datasets.rZinternalr�.N��) r0� __module__r� startswithr�inspect�getfile� __class__�join�split)r1ZimportsZinternal_import_nameZinternal_module_namer r r$�$extend_dataset_builder_for_streamingbs r;)N)*r*r6� functoolsr�typingrrrZ#download.streaming_download_managerrrr r rrr rrrrrrrrrrrrZ utils.loggingrZutils.patchingrZutils.py_utilsr�__name__�loggerr1r�str�boolr0r;r r r r$�sT;