/usr/local/lib/python3.6/site-packages/datasets/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/datasets/__pycache__/load.cpython-36.pyc (62922B)
3
<%EgR @ s~ d Z ddlZddlZddlZddlZddlZddlZddlZddlm Z ddl
m Z ddl m
Z
ddlmZmZmZmZmZmZmZmZ ddlZddlZddlmZmZ dd l m Z dd
l m Z dd l m!Z! dd l"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z* dd
l+m,Z,m-Z- ddl.m/Z/ ddl0m1Z1 ddl2m3Z3m4Z4m5Z5 ddl6m7Z7 ddl8m9Z9m:Z: ddl;m
m?Z? ddl@mAZA ddlBmCZCmDZDmEZEmFZF ddlGmHZH ddlImJZJ ddlKmLZLmMZMmNZNmOZOmPZPmQZQmRZRmSZSmTZTmUZU ddlVmWZW ddlXmYZY dd lZm[Z[ dd l\m]Z] dd l^m_Z_ e[e`ZaebeCjc d g Zde jedfefeee
eff d d!d"Zgdjeeee! eeA f d$d%d&Zheef efd'd(d)Ziefefd*d+d,Zjefefeefefefeff ee/ eeefeff d-d.d/Zkefefefefeeefeff eeefeff ee1 efd0d1d2Zlefeeefeff eeefeff efefefe1eefeff d3d4d5Zmdke%eeeneff eeefeff d6d7d8Zoe%eeeneff eeefeff d6d9d:Zpe G d;d< d<Zqe G d=d> d>ZrG d?d@ d@ZsG dAdB dBZtG dCdD dDesZuG dEdF dFetZvG dGdH dHetZwG dIdJ dJesZxG dKdL dLesZyG dMdN dNesZzG dOdP dPesZ{G dQdR dResZ|G dSdT dTesZ}G dUdV dVetZ~dlefeeefe_f ee/ ee1 eef eef eeeeefe$f eqdWdXdYZdmefeeefe_f ee/ ee1 eef erdZd[d\Zdnefeef eeeef eef enee/ ee1 eeefe_f eAd^ d_d`Zdoefeef eef eeefeef eefeefeef f f f eef ee7 ee/ ee1 eeefe_f eeeneff e!da dbdcZdpefeef eef eeefeef eefeefeef f f f eeefeHf eef ee7 ee/ ee1 eneen eneeefe_f eeeneff eeefeJf enee,e e-e?f dddedfZdqefeen ee e,f dgdhdiZdS )rzAccess datasets. N)Counter) dataclass)Path)DictListMappingOptionalSequenceTupleTypeUnion)HfApiHfFolder )config)Dataset)DatasetBuilder)DEFAULT_PATTERNS_ALL
DataFilesDict
DataFilesList'get_data_patterns_in_dataset_repositoryget_data_patterns_locally+get_metadata_patterns_in_dataset_repository get_metadata_patterns_locallysanitize_patterns) DatasetDictIterableDatasetDict)DownloadConfig) DownloadMode)StreamingDownloadManagerxglobxjoin)Features)extract_path_from_uriis_remote_filesystem) DatasetInfoDatasetInfosDict)IterableDataset)Metric)_EXTENSION_TO_MODULE_MODULE_SUPPORTS_METADATA_PACKAGED_DATASETS_MODULES_hash_python_lines)Split) TaskTemplate)
OfflineModeIsEnabled!_raise_if_offline_mode_is_enabled cached_path
head_hf_s3
hf_github_url
hf_hub_urlinit_hf_modulesis_relative_pathrelative_to_absolute_pathurl_or_path_join)FileLock)is_small_dataset)
get_logger) get_imports)Versionzip)namehf_modules_cachec C s^ t |}tjj|| }tj|dd tjjtjj|dsZttjj|dd W dQ R X |S )a]
Create a module with name `name` in which you can add dynamic modules
such as metrics or datasets. The module can be imported using its name.
The module is created in the HF_MODULE_CACHE directory by default (~/.cache/huggingface/modules) but it can
be overriden by specifying a path to another directory in `hf_modules_cache`.
T)exist_okz __init__.pywN)r5 ospathjoinmakedirsexistsopen)r? r@ dynamic_modules_path rJ 7/usr/local/lib/python3.6/site-packages/datasets/load.pyinit_dynamic_modulesW s
rL T)returnc C sx t j| }|rt}nt}d}xV|jj D ]H\}}tj|r(t||r(tj |rPq(|}tj
|}|dk r(||kr(P q(W |S )zImport a module at module_path and return its main class:
- a DatasetBuilder if dataset is True
- a Metric if dataset is False
N) importlib
import_moduler r( __dict__itemsinspectisclass
issubclass
isabstract getmodule) module_pathdatasetmoduleZ
main_cls_typeZmodule_main_clsr? objZ
obj_modulerJ rJ rK import_main_classi s
r[ )
file_pathsrM c
C s g }x<| D ]4}t jj|r4|jtt|jd q
|j| q
W g }x2|D ]*}t|dd}|j|j W dQ R X qLW t
|S )zt
Convert a list of scripts or text files provided in file_paths into a hashed filename in a repeatable way.
z
*.[pP][yY]zutf-8)encodingN) rC rD isdirextendlistr rglobappendrH readlinesr, )r\ Z to_use_files file_pathlinesfrJ rJ rK
files_to_hash s
rg )r?
resource_typec C sB t j r>t jr>yt| | d |dkd W n tk
r< Y nX dS )z1Update the download count of a dataset or metric.z.pyrX )filenamerX N)r HF_DATASETS_OFFLINEZHF_UPDATE_DOWNLOAD_COUNTSr2 Exception)r? rh rJ rJ rK increase_load_count s
rl )r? base_pathimportsdownload_configrM c C s g }g }|j }|jdkr d|_x|D ]\}}}} |dkrJ|j||f q&|| krttd| d| d| d| d |d krt||d
}
n|d kr|}
ntd t|
|d
} | dk rtjj| | } |j|| f q&W i } xR|D ]J\}
}yt j
|
}W q t k
r( |
| ks ||
kr$|| |
< Y qX qW | rt | dkrFdnd}t | dkr\dnd}t d| d| ddj| d| ddj| j
d |S )a
Download additional module for a module .py at URL (or local path) /.py
The imports must have been parsed first using ``get_imports``.
If some modules need to be installed with pip, an error is raised showing how to install them.
This function return the list of downloaded modules as tuples (import_name, module_file_path).
The downloaded modules can then be moved into an importable directory with ``_copy_script_and_other_resources_in_importable_dir``.
NzDownloading extra moduleslibraryz
Error in the z script, importing relative z module but z: is the name of the script. Please change relative import zl to another name and add a '# From: URL_OR_PATH' comment pointing to the original relative import file path.Zinternalz.pyZexternalzWrong import_type)ro r dependencies
dependencyZthemitzTo be able to use z$, you need to install the following z: z, z.
Please install z using 'pip install z' for instance')copy
download_descrb
ValueErrorr8 r1 rC rD rE rN rO ImportErrorlenvalues)r? rm rn ro
local_importsZlibrary_importsZ import_type import_name import_pathZ
sub_directoryZurl_or_filenameZlocal_import_pathZneeds_to_be_installedZlibrary_import_nameZlibrary_import_pathlibZ_depencencies_strZ _them_strrJ rJ rK _download_additional_modules sH
6r )r? importable_directory_pathsubdirectory_nameoriginal_local_pathr{ additional_files
download_moderM c ! C s, t jj||}t jj|| d }|d } t| |tjkrTt jj|rTtj| t j |dd t jj|d}
t jj|
st
|
d W dQ R X t j |dd t jj|d}
t jj|
st
|
d W dQ R X t jj|stj || t jj |d d } t jj| s:||d
} t
| dd d }
t
j| |
W dQ R X x|D ]\}}t jj|rt jj||d }t jj|stj || nFt jj|rt jj||}t jj|stj|| ntd
| q@W xL|D ]D\}}t jj||}t jj| s tj|| rtj || qW |S Q R X dS )a Copy a script and its required imports to an importable directory
Args:
name (str): name of the resource to load
importable_directory_path (str): path to the loadable folder in the dynamic modules directory
subdirectory_name (str): name of the subdirectory in importable_directory_path in which to place the script
original_local_path (str): local path to the resource script
local_imports (List[Tuple[str, str]]): list of (destination_filename, import_file_to_copy)
additional_files (List[Tuple[str, str]]): list of (destination_filename, additional_file_to_copy)
download_mode (Optional[DownloadMode]): download mode
Return:
importable_local_file: path to an importable module with importlib.import_module
z.pyz.lockT)rA z __init__.pyrB Nr z.json)zoriginal file pathzlocal file pathzutf-8)r] zError with local import at )rC rD rE r9 r FORCE_REDOWNLOADrG shutilrmtreerF rH copyfilesplitextjsondumpisfiler^ copytreeOSErrorfilecmpcmp)r? r r r r{ r r Zimportable_subdirectoryimportable_local_fileZ lock_pathZinit_file_path meta_pathmetaZ meta_filer| r} Zfull_path_local_import file_nameZ
original_pathZdestination_additional_pathrJ rJ rK 2_copy_script_and_other_resources_in_importable_dir sN
r )
local_pathr{ r rI module_namespacer? r rM c C s t jj|||jdd}t|jddd t|jd jdd t| gdd |D }t |j
dd
||| |||d
} t j d | d jt jj
|||jdd||j
dd g}
|
|fS )N/z--T)parentsrA z __init__.py)rA c S s g | ] }|d qS )r rJ ).0locrJ rJ rK
A s z+_create_importable_file..r )r? r r r r{ r r z#Created importable dataset file at .r )rC rD rE replacer mkdirparenttouchrg r splitloggerdebugbasename) r r{ r rI r r? r r hashr rW rJ rJ rK _create_importable_file5 s *r )data_files_listuse_auth_tokenrM c C s^ t dd | dtj D }|rZx8|j D ],\}}|tkrBt| S |dkr*t| |dS q*W dS )a Infer module (and builder kwargs) from list of data files.
Args:
data_files_list (DataFilesList): List of data files.
use_auth_token (bool or str, optional): Whether to use token or token to authenticate on the Hugging Face Hub
for private remote files.
Returns:
tuple[str, str]: Tuple with
- inferred module name
- builder kwargs
c s s* | ]"}t |jD ]}|d d V qqdS )r N)r suffixes)r filepathsuffixrJ rJ rK b s z.infer_module_for_data_files..Nr> )r )r r Z*DATA_FILES_MAX_NUMBER_FOR_MODULE_INFERENCE most_commonr) 'infer_module_for_data_files_in_archives)r r extensions_counterext_rJ rJ rK infer_module_for_data_filesR s r c C s g }d}xf| D ]^}t |jdr|d7 }|tjkr4P tt j|d}|dd t|d|dd tj D 7 }qW t d
d |D }|r|j
dd d }|t krt | S d S ) a Infer module (and builder kwargs) from list of archive data files.
Args:
data_files_list (DataFilesList): List of data files.
use_auth_token (bool or str, optional): Whether to use token or token to authenticate on the Hugging Face Hub
for private remote files.
Returns:
tuple[str, str]: Tuple with
- inferred module name
- builder kwargs
r z.zipr z**c S s g | ]}|j d d qS )z::r )r )r rf rJ rJ rK r s z;infer_module_for_data_files_in_archives..T) recursiver Nc s s* | ]"}t |jD ]}|d d V qqdS )r N)r r )r r r rJ rJ rK r s z:infer_module_for_data_files_in_archives..) strendswithr Z2GLOBBED_DATA_FILES_MAX_NUMBER_FOR_MODULE_INFERENCEr! r extractr Z3ARCHIVED_DATA_FILES_MAX_NUMBER_FOR_MODULE_INFERENCEr r r) )r r Zarchived_filesZarchive_files_counterr Z extractedr r rJ rJ rK r n s"
r c @ s e Zd ZU eeedS )
DatasetModuleN)__name__
__module__ __qualname__r rW r dictbuilder_kwargsrJ rJ rJ rK r s
r c @ s e Zd ZU eedS ) MetricModuleN)r r r r rW r rJ rJ rJ rK r s
r c @ s e Zd ZedddZdS )_DatasetModuleFactory)rM c C s t d S )N)NotImplementedError)selfrJ rJ rK
get_module s z _DatasetModuleFactory.get_moduleN)r r r r r rJ rJ rJ rK r s r c @ s e Zd ZedddZdS )_MetricModuleFactory)rM c C s t d S )N)r )r rJ rJ rK r s z _MetricModuleFactory.get_moduleN)r r r r r rJ rJ rJ rK r s r c @ sv e Zd ZdZdeeeeef ee ee ee dddZ
ee edddZ ee edd d
Z e
d d d
ZdS )GithubDatasetModuleFactoryz
Get the module of a dataset from GitHub (legacy).
The dataset script is downloaded from GitHub.
This class will eventually be removed and a HubDatasetModuleFactory will be used instead.
N)r? revisionro r rI c C sd || _ || _|r|j nt | _| jjdk r4d| j_|| _|| _| j jddksTt t
|dd d S )N r r rX )rh ) r? r ru r ro max_retriesr rI countAssertionErrorrl )r r? r ro r rI rJ rJ rK __init__ s z#GithubDatasetModuleFactory.__init__)r rM c C s< t | j| jd |d}| jj }|jd kr0d|_t||dS )Nz.py)rD r? r zDownloading builder script)ro )r3 r? ro ru rv r1 )r r rd ro rJ rJ rK download_loading_script s
z2GithubDatasetModuleFactory.download_loading_scriptc C sV t | jtj|d}| jj }|jd kr,d|_y t||dS tt fk
rP d S X d S )N)rD r? r zDownloading metadata)ro )
r3 r? r DATASETDICT_INFOS_FILENAMEro ru rv r1 FileNotFoundErrorConnectionError)r r
dataset_infosro rJ rJ rK download_dataset_infos_file s
z6GithubDatasetModuleFactory.download_dataset_infos_file)rM c C s
| j }y| j|}W nR tk
rf |d k s t | j| jd |dd}| jj }|jd kr2d|_t||dS )Nz.pyF)rD r? r rX zDownloading builder script)ro )r3 r? ro ru rv r1 )r r rd ro rJ rJ rK r s
z1GithubMetricModuleFactory.download_loading_script)rM c C s | j }y| j|}| j }W nR tk
rl |d k sBtjdd d k rF n"d}| j|}tjd| j d Y nX t|}t | jt
| jd|dd|| j d}| j r| j nt
}t||g |d | j| jd
\}}tj t||S ) Nr r z-Couldn't find a directory or a metric named 'zH' in this version. It was picked from the main branch on github instead.r F)rD r? r rX )r? rm rn ro metrics)r r{ r rI r r? r )r r r rC r r r r? r< r r3 ro rI rL r r rN r r )r r r rn r{ rI rW r rJ rJ rK r s8
z$GithubMetricModuleFactory.get_module)NNNN)r r r r r r r r= r r r r r r rJ rJ rJ rK r s 0 r c @ s@ e Zd ZdZd eee ee ee dddZe dddZ
dS )
LocalMetricModuleFactoryzRGet the module of a local metric. The metric script is loaded from a local script.N)rD ro r rI c C s. || _ t|j| _|pt | _|| _|| _d S )N)rD r stemr? r ro r rI )r rD ro r rI rJ rJ rK r F s
z!LocalMetricModuleFactory.__init__)rM c C sn t | j}t| jtt| jj|| jd}| jr6| jnt }t
| j|g |d| j| j d\}}t j
t||S )N)r? rm rn ro r )r r{ r rI r r? r )r< rD r r? r r r ro rI rL r r rN r r )r rn r{ rI rW r rJ rJ rK r S s"
z#LocalMetricModuleFactory.get_module)NNN) r r r r r r r r r r r rJ rJ rJ rK r C s
"r c @ s@ e Zd ZdZd eee ee ee dddZe dddZ
dS )
#LocalDatasetModuleFactoryWithScriptzTGet the module of a local dataset. The dataset script is loaded from a local script.N)rD ro r rI c C s. || _ t|j| _|pt | _|| _|| _d S )N)rD r r r? r ro r rI )r rD ro r rI rJ rJ rK r o s
z,LocalDatasetModuleFactoryWithScript.__init__)rM c C s t | jjtj }t| j}t| jtt | jj|| j d}|j
rTtjt|fgng }| j rd| j nt }t
| j|||d| j| jd\}}tj |tt | jjd}t|||S )N)r? rm rn ro r )r r{ r rI r r? r )r rm )r rD r r r r< r r? r ro is_filerI rL r r rN r r ) r r rn r{ r rI rW r r rJ rJ rK r | s(
z.LocalDatasetModuleFactoryWithScript.get_module)NNN) r r r r r r r r r r r rJ rJ rJ rK r l s
"r c @ sJ e Zd ZdZd eee eeeeef ee dddZ
e dddZ dS )
&LocalDatasetModuleFactoryWithoutScriptzGet the module of a dataset loaded from the user's data files. The dataset builder module to use is inferred
from the data files extensions.N)rD data_dir
data_filesr c C sF |r t jj|r td| || _t|j| _|| _|| _|| _ d S )Nz;`data_dir` must be relative to a dataset directory's root: )
rC rD isabsrw r r r? r r r )r rD r r r rJ rJ rK r s z/LocalDatasetModuleFactoryWithoutScript.__init__)rM c C s | j rtjj| j| j n| j}| jd k r2t| jn| j d k rDt|nt|}tj||t d}dd |j
D }t t t
t|j d dkrtd| tt|j \}}|std| j | jd ko|tko|tkrRy t|}W n tk
r d }Y nX |d k rRtj||d}x0|D ](} t|| | || j|j || < q&W t| \}
} | |tjj| j| jd |}tjjtjj| jt j rt tjj| jt j d
d } t j | }
W d Q R X tt|
|d < t!j"|
|d |d
< t#|
| |S )N)rm allowed_extensionsc S s i | ]\}}t ||qS rJ )r )r keyr rJ rJ rK
s zELocalDatasetModuleFactoryWithoutScript.get_module..r r z=Couldn't infer the same data file format for all splits. Got z)No data files or dataset script found in )rm )r r config_namerm zutf-8)r] r info)$r rC rD rE r r r r from_local_or_remoteALL_ALLOWED_EXTENSIONSrQ ry setr` r> rz rw nextiterr r* r r r origin_metadatar+ r r r r rH r loadr% from_dictr )r rm patternsr module_names module_namer metadata_patternsmetadata_filesr rW r rf r rJ rJ rK r sL
z1LocalDatasetModuleFactoryWithoutScript.get_module)NNN)
r r r r r r r r r r r r r rJ rJ rJ rK r s
, r c @ sP e Zd ZdZd eee eeeeef ee ee
dddZ e dddZ
dS )
PackagedDatasetModuleFactoryz`Get the dataset builder module from the ones that are packaged with the library: csv, json, etc.N)r? r r ro r c C s. || _ || _|| _|| _|| _t|dd d S )NrX )rh )r? r r ro r rl )r r? r r ro r rJ rJ rK r s z%PackagedDatasetModuleFactory.__init__)rM c
C s | j d k r tt| j j n tt j }| jd k r>t| jn| j d k rPt|nt|}tj|| j j
|d}| jd kr| j t kr|t
kry t|}W n tk
r d }Y nX |d k rtj|| j j
|d}x.|D ]&}t|| | || j|j ||< qW t| j \}}||d} t||| S )N)r rm )r r )r r r resolver r r r r ro r r? r* r r r r r r+ r )
r rm r r r r r rW r r rJ rJ rK r s. *
z'PackagedDatasetModuleFactory.get_module)NNNN)r r r r r r r r r r r r r r rJ rJ rJ rK r s 2
r c @ s^ e Zd ZdZd eeeeef ee eeeee f ee
ee dddZ e
dddZdS )
$HubDatasetModuleFactoryWithoutScriptz
Get the module of a dataset loaded from data files of a dataset repository.
The dataset builder module to use is inferred from the data files extensions.
N)r? r r r ro r c C sN || _ || _|| _|| _|p t | _|| _| j jddks>tt |dd d S )Nr r rX )rh )
r? r r r r ro r r r rl )r r? r r r ro r rJ rJ rK r s
z-HubDatasetModuleFactoryWithoutScript.__init__)rM c sn t jjtr$ jjr tj nd }n jj}ttjj j
j |rF|nddd} j d k rdt
j n
t| j}tj|| jtd} fdd|j D }tttt|j d dkrtd | tt|j \}}|st d
j
j d kr|t kr|t kryt | j}W n t k
r: d }Y nX |d k rt j|| jd } x0|D ](}
t ||
| ||
j!| j! ||
< q^W t"| \} } | | j
j#d d
t$ j
d j d j
d|} jj% }
|
j&d krd|
_&yjt't$ j
tj( j d jd}t)|dd}t*j+|}W d Q R X tt||d< t,j-||d |d< W n t k
r` Y nX t.| | |S )Nzno-tokeng Y@)r tokentimeout) dataset_inform r c s" i | ]\}}t | jjd |qS ))r )r ro r )r r r )r rJ rK r D s zCHubDatasetModuleFactoryWithoutScript.get_module..r r z=Couldn't infer the same data file format for all splits. Got z)No data files or dataset script found in )r rm r z--r )r )r r r rm repo_idzDownloading metadata)ro zutf-8)r] r r )/
isinstancero r boolr get_tokenr
r HF_ENDPOINTr r? r r r r r r Z from_hf_repor rQ ry r r` r> rz rw r r r r* r r r r r+ r r4 ru rv r1 r rH r r r% r r )r r Zhfh_dataset_infor r r r r r r r rW r ro r rf r rJ )r rK r . sn
z/HubDatasetModuleFactoryWithoutScript.get_module)NNNNN)r r r r r r r r= r r r r r r r rJ rJ rJ rK r s @ r c @ sj e Zd ZdZd
eeeeef ee ee ee dddZ
edddZ edd d
Z e
dd d ZdS )!HubDatasetModuleFactoryWithScriptz~Get the module of a dataset from a dataset repository. The dataset script comes from the script inside the dataset repository.N)r? r ro r rI c C sH || _ || _|pt | _|| _|| _| j jddks8tt|dd d S )Nr r rX )rh ) r? r r ro r rI r r rl )r r? r ro r rI rJ rJ rK r y s z*HubDatasetModuleFactoryWithScript.__init__)rM c C sH t | j| jjdd d | jd}| jj }|jd kr.zDataset z is not cached in c s& t | jjdd d j jS )Nr r z.pyr )r r? r statst_mtime) module_hash)r r rJ rK _get_modification_time s zECachedDatasetModuleFactory.get_module.._get_modification_time)r r z3Using the latest cached version of the module from z (last modified on z() since it couldn't be found locally at r z&, or remotely on the Hugging Face Hub.)r r r r )rI rL rC rD rE r? r r^ listdirr sortedtimectimer rj r r r r rN r r )r rI hashesr
r Z warning_msgrW r rJ )r r rK r s, .
z%CachedDatasetModuleFactory.get_module)N) r r r r r r r r r rJ rJ rJ rK r s r c @ s4 e Zd ZdZd eee dddZedddZdS )
CachedMetricModuleFactoryz
Get the module of a metric that has been loaded once already and cached.
The script that is loaded from the cache is the most recent one with a matching name.
N)r? rI c C s$ || _ || _| j jddks td S )Nr r )r? rI r r )r r? rI rJ rJ rK r s z"CachedMetricModuleFactory.__init__)rM c s j r j nt }tjj|dj tjj rDdd tj D nd }|sbtdj d| fdd}t ||dd }t
j d
tjj | d t j
|| d j d
djtjj|dj|jg}tj t||S )Nr c S s g | ]}t |d kr|qS )r )ry )r r rJ rJ rK r s z8CachedMetricModuleFactory.get_module..zMetric z is not cached in c s t | jd j jS )Nz.py)r r? r r )r )r r rJ rK r
s zDCachedMetricModuleFactory.get_module.._get_modification_time)r r z3Using the latest cached version of the module from z (last modified on z() since it couldn't be found locally at z&, or remotely on the Hugging Face Hub.r r )rI rL rC rD rE r? r^ r r r r r r
r r rN r r )r rI r r
r rW rJ )r r rK r s 0 z$CachedMetricModuleFactory.get_module)N) r r r r r r r r r rJ rJ rJ rK r s r )rD r ro r rI r r rM c ! K s |dkrt f |}t|p tj}d|_d|_|tjk|_ttdd | j t
j dj dd% }|j
dsp|d }t
jj| |} | tkrt| ||||dj S | j
|rt
jj| rt| ||d j S td
t| nt
jj| rt| ||d j S t
jj| r t| |||d j S t| r| jddkryt | jdd krft | ||||d
j S | jddkrt t j }
yJt |j!t"r|j!rt#j$ nd} n|j!} |
j%| || r| nddd} W n t&k
r }
zt |
t't(j)j*t(j)j+frt+d| dt,|
j- dndt.|
krPd| d}t|rH|d| d n|nFdt.|
krd| d}|r|d| d n|}t|d n|
W Y dd}
~
X nX |dd | j/D krt0| ||||d
j S t1| |||||d j S W n t&k
r } zyt2| |d j S t&k
r } zft |t'rHt+d | d | dt |trtd
t| d!| d"t,|j- d#| d|dW Y dd}~X nX W Y dd}~X nX ntd
t| d$dS )&av
Download/extract/cache a dataset module.
Dataset codes are cached inside the the dynamic modules cache to allow easy import (avoid ugly sys.path tweaks).
Args:
path (str): Path or name of the dataset.
Depending on ``path``, the dataset builder that is used comes from a generic dataset script (JSON, CSV, Parquet, text etc.) or from the dataset script (a python file) inside the dataset directory.
For local datasets:
- if ``path`` is a local directory (containing data files only)
-> load a generic dataset builder (csv, json, text etc.) based on the content of the directory
e.g. ``'./path/to/directory/with/my/csv/data'``.
- if ``path`` is a local dataset script or a directory containing a local dataset script (if the script has the same name as the directory):
-> load the dataset builder from the dataset script
e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'``.
For datasets on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``)
- if ``path`` is a dataset repository on the HF hub (containing data files only)
-> load a generic dataset builder (csv, text etc.) based on the content of the repository
e.g. ``'username/dataset_name'``, a dataset repository on the HF hub containing your data files.
- if ``path`` is a dataset repository on the HF hub with a dataset script (if the script has the same name as the directory)
-> load the dataset builder from the dataset script in the dataset repository
e.g. ``glue``, ``squad``, ``'username/dataset_name'``, a dataset repository on the HF hub containing a dataset script `'dataset_name.py'`.
revision (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset script to load:
- For datasets in the `huggingface/datasets` library on GitHub like "squad", the default version of the module is the local version of the lib.
You can specify a different version from your local version of the lib (e.g. "main" or "1.2.0") but it might cause compatibility issues.
- For community datasets like "lhoestq/squad" that have their own git repository on the Datasets Hub, the default version "main" corresponds to the "main" branch.
You can specify a different version that the default "main" by using a commit sha or a git tag of the dataset repository.
download_config (:class:`DownloadConfig`, optional): Specific download configuration parameters.
download_mode (:class:`DownloadMode`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode.
dynamic_modules_path (Optional str, defaults to HF_MODULES_CACHE / "datasets_modules", i.e. ~/.cache/huggingface/modules/datasets_modules):
Optional path to the directory in which the dynamic modules are saved. It must have been initialized with :obj:`init_dynamic_modules`.
By default the datasets and metrics are stored inside the `datasets_modules` module.
data_dir (:obj:`str`, optional): Directory with the data files. Used only if `data_files` is not specified,
in which case it's equal to passing `os.path.join(data_dir, "**")` as `data_files`.
data_files (:obj:`Union[Dict, List, str]`, optional): Defining the data_files of the dataset configuration.
**download_kwargs (additional keyword arguments): optional attributes for DownloadConfig() which will override
the attributes in download_config if supplied.
Returns:
DatasetModule
NTc S s | S )NrJ )xrJ rJ rK c s z(dataset_module_factory..r r z.py)r r ro r )r rI z"Couldn't find a dataset script at )r r r r )r ro r rI zno-tokeng Y@)r r r r zCouldn't reach 'z' on the Hub ()Z404z Dataset 'z' doesn't exist on the Hubz at revision ''Z401z. If the repo is private, make sure you are authenticated with `use_auth_token=True` after logging in with `huggingface-cli login`.c S s g | ]
}|j qS rJ )Z rfilename)r ZsiblingrJ rJ rK r s z*dataset_module_factory..)r r r ro r )rI z0Couln't reach the Hugging Face Hub for dataset 'z': z8 or any data file in the same directory. Couldn't find 'z"' on the Hugging Face Hub either: z: z( or any data file in the same directory.r )3r r REUSE_DATASET_IF_EXISTSextract_compressed_file
force_extractr Zforce_downloadr` filterr rC sepr r rD rE r+ r r r r r r7 r^ r r6 r r0 r r
r r r r r r r r rk r/ requests
exceptionsConnectTimeoutr typer r Zsiblingsr r r )rD r ro r rI r r download_kwargsri
combined_pathZhf_apir r emsge1e2rJ rJ rK dataset_module_factory" s :
&
&,r$ )rD r ro r rI rM c
K s |dkrt f |}t|p tj}d|_d|_ttdd | jtj dj
dd }|j dsd|d }tj j
| |}| j |rtj j| rt| ||dj S td t| ntj j|rt|||dj S t| o| jdd
kryt| ||||d j S tk
r } zjyt| |d j S tk
rp } z4t|tsD|dtd t| d
| ddW Y dd} ~ X nX W Y dd}~X nX ntd t| ddS )a
Download/extract/cache a metric module.
Metrics codes are cached inside the the dynamic modules cache to allow easy import (avoid ugly sys.path tweaks).
Args:
path (str): Path or name of the metric script.
- if ``path`` is a local metric script or a directory containing a local metric script (if the script has the same name as the directory):
-> load the module from the metric script
e.g. ``'./metrics/accuracy'`` or ``'./metrics/accuracy/accuracy.py'``.
- if ``path`` is a metric on the Hugging Face Hub (ex: `glue`, `squad`)
-> load the module from the metric script in the github repository at huggingface/datasets
e.g. ``'accuracy'`` or ``'rouge'``.
revision (Optional ``Union[str, datasets.Version]``):
If specified, the module will be loaded from the datasets repository at this version.
By default:
- it is set to the local version of the lib.
- it will also try to load it from the main branch if it's not available at the local version of the lib.
Specifying a version that is different from your local version of the lib might cause compatibility issues.
download_config (:class:`DownloadConfig`, optional): Specific download configuration parameters.
download_mode (:class:`DownloadMode`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode.
dynamic_modules_path (Optional str, defaults to HF_MODULES_CACHE / "datasets_modules", i.e. ~/.cache/huggingface/modules/datasets_modules):
Optional path to the directory in which the dynamic modules are saved. It must have been initialized with :obj:`init_dynamic_modules`.
By default, the datasets and metrics are stored inside the `datasets_modules` module.
**download_kwargs (additional keyword arguments): optional attributes for DownloadConfig() which will override
the attributes in download_config if supplied.
Returns:
MetricModule
NTc S s | S )NrJ )r rJ rJ rK r s z'metric_module_factory..r r z.py)r rI z!Couldn't find a metric script at r )r ro r rI )rI z
. Metric 'z/' doesn't exist on the Hugging Face Hub either.r r )r r r r r r` r r rC r r r rD rE r r r r r7 r6 r r rk r r )
rD r ro r rI r ri r r" r# rJ rJ rK metric_module_factory sF )
&
*r% F) rD r
process_id num_process cache_dir
experiment_idkeep_in_memoryro r r rM c
K sX t |p
t j}t| | ||dj} t| dd} | f ||||||d|
}
|
j|d |
S )ay Load a `datasets.Metric`.
Args:
path (``str``):
path to the metric processing script with the metric builder. Can be either:
- a local path to processing script or the directory containing the script (if the script has the same name as the directory),
e.g. ``'./metrics/rouge'`` or ``'./metrics/rogue/rouge.py'``
- a metric identifier on the HuggingFace datasets repo (list all available metrics with ``datasets.list_metrics()``)
e.g. ``'rouge'`` or ``'bleu'``
config_name (:obj:`str`, optional): selecting a configuration for the metric (e.g. the GLUE metric has a configuration for each subset)
process_id (:obj:`int`, optional): for distributed evaluation: id of the process
num_process (:obj:`int`, optional): for distributed evaluation: total number of processes
cache_dir (Optional str): path to store the temporary predictions and references (default to `~/.cache/huggingface/metrics/`)
experiment_id (``str``): A specific experiment id. This is used if several distributed evaluations share the same file system.
This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).
keep_in_memory (bool): Whether to store the temporary results in memory (defaults to False)
download_config (Optional ``datasets.DownloadConfig``: specific download configuration parameters.
download_mode (:class:`DownloadMode`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode.
revision (Optional ``Union[str, datasets.Version]``): if specified, the module will be loaded from the datasets repository
at this version. By default it is set to the local version of the lib. Specifying a version that is different from
your local version of the lib might cause compatibility issues.
Returns:
`datasets.Metric`
Example:
```py
>>> from datasets import load_metric
>>> accuracy = load_metric('accuracy')
>>> accuracy.compute(references=[1, 0], predictions=[1, 1])
{'accuracy': 0.5}
```
)r ro r F)rX )r r& r' r( r* r) )ro )r r r% rW r[ download_and_prepare)rD r r& r' r( r) r* ro r r Zmetric_init_kwargsZ
metric_moduleZ
metric_clsr rJ rJ rK load_metric< s 0 r, ) rD r? r r r( featuresro r r r rM c
s t |p
t j}| dk r.|r"|j nt }| |_t |||||d} t| j} | j}
|
j d|}|
j d|}|
j d} t
kr|dkrd d} fdd t D }|r|d
|d d 7 }t || f ||||||| d
|
|
}|S )aa Load a dataset builder from the Hugging Face Hub, or a local dataset. A dataset builder can be used to inspect general information that is required to build a dataset (cache directory, config, dataset info, etc.)
without downloading the dataset itself.
You can find the list of datasets on the Hub at https://huggingface.co/datasets or with ``datasets.list_datasets()``.
A dataset is a directory that contains:
- some data files in generic formats (JSON, CSV, Parquet, text, etc.)
- and optionally a dataset script, if it requires some code to read the data files. This is used to load any kind of formats or structures.
Note that dataset scripts can also download and read data files from anywhere - in case your data files already exist online.
Args:
path (:obj:`str`): Path or name of the dataset.
Depending on ``path``, the dataset builder that is used comes from a generic dataset script (JSON, CSV, Parquet, text etc.) or from the dataset script (a python file) inside the dataset directory.
For local datasets:
- if ``path`` is a local directory (containing data files only)
-> load a generic dataset builder (csv, json, text etc.) based on the content of the directory
e.g. ``'./path/to/directory/with/my/csv/data'``.
- if ``path`` is a local dataset script or a directory containing a local dataset script (if the script has the same name as the directory):
-> load the dataset builder from the dataset script
e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'``.
For datasets on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``)
- if ``path`` is a dataset repository on the HF hub (containing data files only)
-> load a generic dataset builder (csv, text etc.) based on the content of the repository
e.g. ``'username/dataset_name'``, a dataset repository on the HF hub containing your data files.
- if ``path`` is a dataset repository on the HF hub with a dataset script (if the script has the same name as the directory)
-> load the dataset builder from the dataset script in the dataset repository
e.g. ``glue``, ``squad``, ``'username/dataset_name'``, a dataset repository on the HF hub containing a dataset script `'dataset_name.py'`.
name (:obj:`str`, optional): Defining the name of the dataset configuration.
data_dir (:obj:`str`, optional): Defining the data_dir of the dataset configuration. If specified for the generic builders (csv, text etc.) or the Hub datasets and `data_files` is None,
the behavior is equal to passing `os.path.join(data_dir, **)` as `data_files` to reference all the files in a directory.
data_files (:obj:`str` or :obj:`Sequence` or :obj:`Mapping`, optional): Path(s) to source data file(s).
cache_dir (:obj:`str`, optional): Directory to read/write data. Defaults to "~/.cache/huggingface/datasets".
features (:class:`Features`, optional): Set the features type to use for this dataset.
download_config (:class:`~utils.DownloadConfig`, optional): Specific download configuration parameters.
download_mode (:class:`DownloadMode`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode.
revision (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset script to load:
- For datasets in the `huggingface/datasets` library on GitHub like "squad", the default version of the module is the local version of the lib.
You can specify a different version from your local version of the lib (e.g. "main" or "1.2.0") but it might cause compatibility issues.
- For community datasets like "lhoestq/squad" that have their own git repository on the Datasets Hub, the default version "main" corresponds to the "main" branch.
You can specify a different version that the default "main" by using a commit sha or a git tag of the dataset repository.
use_auth_token (``str`` or :obj:`bool`, optional): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If True, will get token from `"~/.huggingface"`.
**config_kwargs (additional keyword arguments): Keyword arguments to be passed to the :class:`BuilderConfig`
and used in the :class:`DatasetBuilder`.
Returns:
:class:`DatasetBuilder`
Passing `use_auth_token=True` is required when you want to access a private dataset.
Example:
```py
>>> from datasets import load_dataset_builder
>>> ds_builder = load_dataset_builder('rotten_tomatoes')
>>> ds_builder.info.features
{'label': ClassLabel(num_classes=2, names=['neg', 'pos'], id=None),
'text': Value(dtype='string', id=None)}
```
N)r ro r r r r r r z.Please specify the data files to load for the z dataset builder.c s g | ]}t | kr|qS rJ )r) )r extension)rD rJ rK r s z(load_dataset_builder..z9
For example `data_files={"train": "path/to/data/train/*.r z"}`)r( r r r r r- r )
r r ru r r r$ r[ rW r popr+ r) rw )rD r? r r r( r- ro r r r
config_kwargsZdataset_moduleZ builder_clsr r r error_msgZexample_extensionsbuilder_instancerJ )rD rK load_dataset_builder sB W
r3 )rD r? r r r r( r- ro r ignore_verificationsr*
save_infosr r task streamingrM c
K s t | tjj rtdt|p"tj}| p,| } tf | |||||||| |
d
|}|rb|j|dS | t k}|j
||| ||
d |
dk r|
n
t |j j
}
|j|| |
d}|dk r|j|}| r|j |S )a7% Load a dataset from the Hugging Face Hub, or a local dataset.
You can find the list of datasets on the Hub at https://huggingface.co/datasets or with ``datasets.list_datasets()``.
A dataset is a directory that contains:
- some data files in generic formats (JSON, CSV, Parquet, text, etc.)
- and optionally a dataset script, if it requires some code to read the data files. This is used to load any kind of formats or structures.
Note that dataset scripts can also download and read data files from anywhere - in case your data files already exist online.
This function does the following under the hood:
1. Download and import in the library the dataset script from ``path`` if it's not already cached inside the library.
If the dataset has no dataset script, then a generic dataset script is imported instead (JSON, CSV, Parquet, text, etc.)
Dataset scripts are small python scripts that define dataset builders. They define the citation, info and format of the dataset,
contain the path or URL to the original data files and the code to load examples from the original data files.
You can find some of the scripts here: https://github.com/huggingface/datasets/tree/main/datasets
You can find the complete list of datasets in the Datasets Hub at https://huggingface.co/datasets
2. Run the dataset script which will:
* Download the dataset file from the original URL (see the script) if it's not already available locally or cached.
* Process and cache the dataset in typed Arrow tables for caching.
Arrow table are arbitrarily long, typed tables which can store nested objects and be mapped to numpy/pandas/python generic types.
They can be directly accessed from disk, loaded in RAM or even streamed over the web.
3. Return a dataset built from the requested splits in ``split`` (default: all).
It also allows to load a dataset from a local directory or a dataset repository on the Hugging Face Hub without dataset script.
In this case, it automatically loads all the data files from the directory or the dataset repository.
Args:
path (:obj:`str`): Path or name of the dataset.
Depending on ``path``, the dataset builder that is used comes from a generic dataset script (JSON, CSV, Parquet, text etc.) or from the dataset script (a python file) inside the dataset directory.
For local datasets:
- if ``path`` is a local directory (containing data files only)
-> load a generic dataset builder (csv, json, text etc.) based on the content of the directory
e.g. ``'./path/to/directory/with/my/csv/data'``.
- if ``path`` is a local dataset script or a directory containing a local dataset script (if the script has the same name as the directory):
-> load the dataset builder from the dataset script
e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'``.
For datasets on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``)
- if ``path`` is a dataset repository on the HF hub (containing data files only)
-> load a generic dataset builder (csv, text etc.) based on the content of the repository
e.g. ``'username/dataset_name'``, a dataset repository on the HF hub containing your data files.
- if ``path`` is a dataset repository on the HF hub with a dataset script (if the script has the same name as the directory)
-> load the dataset builder from the dataset script in the dataset repository
e.g. ``glue``, ``squad``, ``'username/dataset_name'``, a dataset repository on the HF hub containing a dataset script `'dataset_name.py'`.
name (:obj:`str`, optional): Defining the name of the dataset configuration.
data_dir (:obj:`str`, optional): Defining the data_dir of the dataset configuration. If specified for the generic builders (csv, text etc.) or the Hub datasets and `data_files` is None,
the behavior is equal to passing `os.path.join(data_dir, **)` as `data_files` to reference all the files in a directory.
data_files (:obj:`str` or :obj:`Sequence` or :obj:`Mapping`, optional): Path(s) to source data file(s).
split (:class:`Split` or :obj:`str`): Which split of the data to load.
If None, will return a `dict` with all splits (typically `datasets.Split.TRAIN` and `datasets.Split.TEST`).
If given, will return a single Dataset.
Splits can be combined and specified like in tensorflow-datasets.
cache_dir (:obj:`str`, optional): Directory to read/write data. Defaults to "~/.cache/huggingface/datasets".
features (:class:`Features`, optional): Set the features type to use for this dataset.
download_config (:class:`~utils.DownloadConfig`, optional): Specific download configuration parameters.
download_mode (:class:`DownloadMode`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode.
ignore_verifications (:obj:`bool`, default ``False``): Ignore the verifications of the downloaded/processed dataset information (checksums/size/splits/...).
keep_in_memory (:obj:`bool`, default ``None``): Whether to copy the dataset in-memory. If `None`, the dataset
will not be copied in-memory unless explicitly enabled by setting `datasets.config.IN_MEMORY_MAX_SIZE` to
nonzero. See more details in the :ref:`load_dataset_enhancing_performance` section.
save_infos (:obj:`bool`, default ``False``): Save the dataset information (checksums/size/splits/...).
revision (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset script to load:
- For datasets in the `huggingface/datasets` library on GitHub like "squad", the default version of the module is the local version of the lib.
You can specify a different version from your local version of the lib (e.g. "main" or "1.2.0") but it might cause compatibility issues.
- For community datasets like "lhoestq/squad" that have their own git repository on the Datasets Hub, the default version "main" corresponds to the "main" branch.
You can specify a different version that the default "main" by using a commit sha or a git tag of the dataset repository.
use_auth_token (``str`` or :obj:`bool`, optional): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If True, will get token from `"~/.huggingface"`.
task (``str``): The task to prepare the dataset for during training and evaluation. Casts the dataset's :class:`Features` to standardized column names and types as detailed in :py:mod:`datasets.tasks`.
streaming (:obj:`bool`, default ``False``): If set to True, don't download the data files. Instead, it streams the data progressively while
iterating on the dataset. An IterableDataset or IterableDatasetDict is returned instead in this case.
Note that streaming works for datasets that use data formats that support being iterated over like txt, csv, jsonl for example.
Json files may be downloaded completely. Also streaming from remote zip or gzip files is supported but other compressed formats
like rar and xz are not yet supported. The tgz format doesn't allow streaming.
**config_kwargs (additional keyword arguments): Keyword arguments to be passed to the :class:`BuilderConfig`
and used in the :class:`DatasetBuilder`.
Returns:
:class:`Dataset` or :class:`DatasetDict`:
- if `split` is not None: the dataset requested,
- if `split` is None, a ``datasets.DatasetDict`` with each split.
or :class:`IterableDataset` or :class:`IterableDatasetDict`: if streaming=True
- if `split` is not None: the dataset requested,
- if `split` is None, a ``datasets.streaming.IterableDatasetDict`` with each split.
Passing `use_auth_token=True` is required when you want to access a private dataset.
Example:
Load a dataset from the Hugging Face Hub:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset('rotten_tomatoes', split='train')
# Map data files to splits
>>> data_files = {'train': 'train.csv', 'test': 'test.csv'}
>>> ds = load_dataset('namespace/your_dataset_name', data_files=data_files)
```
Load a local dataset:
```py
# Load a CSV file
>>> from datasets import load_dataset
>>> ds = load_dataset('csv', data_files='path/to/local/my_dataset.csv')
# Load a JSON file
>>> from datasets import load_dataset
>>> ds = load_dataset('json', data_files='path/to/local/my_dataset.json')
# Load from a local loading script
>>> from datasets import load_dataset
>>> ds = load_dataset('path/to/local/loading_script/loading_script.py', split='train')
```
Load an [`~datasets.IterableDataset`]:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset('rotten_tomatoes', split='train', streaming=True)
```
Load an image dataset with the `ImageFolder` dataset builder:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset('imagefolder', data_dir='/path/to/images', split='train')
```
zjYou are trying to load a dataset that was saved using `save_to_disk`. Please use `load_from_disk` instead.)
rD r? r r r( r- ro r r r )r )ro r r4 try_from_hf_gcsr N)r r4 Z in_memory)r r ZDATASET_STATE_JSON_FILENAMErG rw r r r3 Zas_streaming_datasetr+ r+ r: r Z dataset_sizeZ
as_datasetZprepare_for_taskZ _save_infos)rD r? r r r r( r- ro r r4 r* r5 r r r6 r7 r0 r2 r8 ZdsrJ rJ rK load_dataset sD -
r9 ) dataset_pathr* rM c C s t |rt| }ntjd}| }|j|s:td| d|jt|tj j
r`t j | ||dS |jt|tj
j
rtj | ||dS td| ddS )a
Loads a dataset that was previously saved using :meth:`Dataset.save_to_disk` from a dataset directory, or
from a filesystem using either :class:`datasets.filesystems.S3FileSystem` or any implementation of
``fsspec.spec.AbstractFileSystem``.
Args:
dataset_path (:obj:`str`): Path (e.g. `"dataset/train"`) or remote URI (e.g.
`"s3://my-bucket/dataset/train"`) of the Dataset or DatasetDict directory where the dataset will be
loaded from.
fs (:class:`~filesystems.S3FileSystem` or ``fsspec.spec.AbstractFileSystem``, optional, default ``None``):
Instance of of the remote filesystem used to download the files from.
keep_in_memory (:obj:`bool`, default ``None``): Whether to copy the dataset in-memory. If `None`, the dataset
will not be copied in-memory unless explicitly enabled by setting `datasets.config.IN_MEMORY_MAX_SIZE` to
nonzero. See more details in the :ref:`load_dataset_enhancing_performance` section.
Returns:
:class:`Dataset` or :class:`DatasetDict`:
- If `dataset_path` is a path of a dataset directory: the dataset requested.
- If `dataset_path` is a path of a dataset dict directory: a ``datasets.DatasetDict`` with each split.
Example:
```py
>>> from datasets import load_from_disk
>>> ds = load_from_disk('path/to/dataset/directory')
```
filez
Directory z
not found)r* z= is neither a dataset directory nor a dataset dict directory.N)r$ r# fsspec
filesystemrG r r r r ZDATASET_INFO_FILENAMEas_posixr load_from_diskZDATASETDICT_JSON_FILENAMEr )r: fsr* Zdest_dataset_pathrJ rJ rK r? s
r? )T)N)NNNNNN)NNNN) Nr r NNFNNN) NNNNNNNNN)NNNNNNNNFNFNNNF)NN)r r rN rR r rC r r
collectionsr Z dataclassesr pathlibr typingr r r r r r
r r r< r Zhuggingface_hubr
r r r Z
arrow_datasetr Zbuilderr r r r r r r r r r Z dataset_dictr r Zdownload.download_configr Zdownload.download_managerr Z#download.streaming_download_managerr r r! r- r" Z filesystemsr# r$ r r% r& Ziterable_datasetr' r r( Zpackaged_modulesr) r* r+ r, splitsr- tasksr. Zutils.file_utilsr/ r0 r1 r2 r3 r4 r5 r6 r7 r8 Zutils.filelockr9 Zutils.info_utilsr: Z
utils.loggingr; Zutils.py_utilsr< Z
utils.versionr= r r r` keysr Z MODULE_NAME_FOR_DYNAMIC_MODULESr rL r[ rg rl r r r r r r r r r r r r r r r r r r r r r$ r% intr, r3 r9 r? rJ rJ rJ rK s ( (
0 $$
>O"WC).G4aI9, < > $Q 6; lz R