/usr/local/lib/python3.6/site-packages/datasets/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/datasets/__pycache__/builder.cpython-36.pyc (52779B)
3
<%EgW @ s d Z ddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddl m Z ddl
mZ ddlmZmZmZmZmZ ddlmZmZ ddlmZ dd lmZm Z m Z m Z m Z dd
l m!Z!m"Z" dd l#m$Z$m%Z% dd l&m'Z'm(Z( dd
l)m*Z* ddl+m,Z,m-Z- ddl.m/Z/ ddl0m1Z1 ddl2m3Z3 ddl4m5Z5 ddl6m7Z7m8Z8m9Z9 ddl:m;Z;m
m?Z? ddl@mAZAmBZB ddlCmDZDmEZEmFZF ddlGmHZH ddlmIZI ddlJmKZKmLZL ddlMmNZN dd lOmPZPmQZQmRZR dd lSmTZTmUZUmVZVmWZWmXZXmYZY eIjZe[Z\G d d d e]Z^G d d! d!e_Z`G d"d# d#e`Zae G d$d% d%ZbG d&d' d'ZcG d(d) d)ecZdG d*d+ d+ecZeG d,d- d-e]ZfG d.d/ d/ecZgdS )0zDatasetBuilder base class. N) dataclass)partial)DictMappingOptionalTupleUnion )configutils)Dataset)HF_GCP_BASE_URL ArrowReaderDatasetNotOnHfGcsErrorMissingFilesOnHfGcsErrorReadInstruction) ArrowWriter
BeamWriter)
DataFilesDictsanitize_patterns) DatasetDictIterableDatasetDict)DownloadConfig)DownloadManager DownloadMode)MockDownloadManager)StreamingDownloadManager)Features)Hasher) DatasetInfoDatasetInfosDictPostProcessedInfo)ExamplesIterableIterableDataset&_generate_examples_from_tables_wrapper)DuplicatedKeysError)"INVALID_WINDOWS_CHARACTERS_IN_PATHcamelcase_to_snakecase)Split SplitDictSplitGenerator)$extend_dataset_builder_for_streaming)logging) cached_path
is_remote_url)FileLock)get_size_checksum_dictverify_checksums
verify_splits)
classpropertyhas_sufficient_disk_space
map_nestedmemoizesize_strtemporary_assignmentc @ s e Zd ZdS )InvalidConfigNameN)__name__
__module__ __qualname__ r= r= :/usr/local/lib/python3.6/site-packages/datasets/builder.pyr9 I s r9 c @ s e Zd ZdS )DatasetBuildErrorN)r: r; r< r= r= r= r> r? M s r? c @ s e Zd ZdS )ManualDownloadErrorN)r: r; r< r= r= r= r> r@ Q s r@ c @ s e Zd ZU dZdZeejdZe e
ejef dZ e e dZ e e
dZe e dd Zdd Zd ee e ed d
d ZdS )
BuilderConfiga Base class for :class:`DatasetBuilder` data configuration.
DatasetBuilder subclasses with data configuration options should subclass
:class:`BuilderConfig` and add their own properties.
Attributes:
name (:obj:`str`, default ``"default"``):
version (:class:`Version` or :obj:`str`, optional):
data_dir (:obj:`str`, optional):
data_files (:obj:`str` or :obj:`Sequence` or :obj:`Mapping`, optional): Path(s) to source data file(s).
description (:obj:`str`, optional):
defaultz0.0.0Nc C s\ x.t D ]&}|| jkrtdt d| j dqW | jd k rXt| jt rXtd| j d S )Nz Bad characters from black list 'z ' found in 'z\'. They could create issues when creating a directory for this config on Windows filesystem.z/Expected a DataFilesDict in data_files but got )r& namer9
data_files
isinstancer
ValueError)selfZ invalid_charr= r= r>
__post_init__j s
zBuilderConfig.__post_init__c s> t jj t jj kr dS t fddjj D S )NFc 3 s* | ]"}|t |f|t |fkV qd S )N)getattr).0k)orG r= r> z s z'BuilderConfig.__eq__..)set__dict__keysall)rG rL r= )rL rG r> __eq__u s zBuilderConfig.__eq__)
config_kwargscustom_featuresreturnc s" d}|j jdd jdd d krD d dkrD jdd r fddt D tdd j D rd jd
d j D }t|d krtj }n
tj }|dk rt }|r|j
| |j
| |j }|r| j d | }t|t
jkr| j d tj | }|S | j S dS )
a/
The config id is used to build the cache directory.
By default it is equal to the config name.
However the name of a config is not sufficient to have a unique identifier for the dataset being generated
since it doesn't take into account:
- the config kwargs that can be used to overwrite attributes
- the custom features used to write the dataset
- the data_files for json/text/csv/pandas datasets
Therefore the config id is just the config name with an optional suffix based on these.
NrC versiondata_dirc s i | ]} | |qS r= r= )rJ rK ) config_kwargs_to_add_to_suffixr= r>
s z2BuilderConfig.create_config_id..c s s | ]}t |ttttfV qd S )N)rE strboolintfloat)rJ vr= r= r> rM s z1BuilderConfig.create_config_id..,c s s. | ]&\}}t |d tjjt | V qdS )=N)rZ urllibparse
quote_plus)rJ rK r^ r= r= r> rM s -)copypopsortedrQ valuesjoinitemslenr hashupdate hexdigestrC r
Z%MAX_DATASET_CONFIG_ID_READABLE_LENGTH)rG rS rT suffixm config_idr= )rX r> create_config_id| s6
z BuilderConfig.create_config_id)N)r: r; r< __doc__rC rZ r VersionrV r r rW rD r descriptionrH rR dictr rs r= r= r= r> rA U s
rA c @ s e Zd ZdZdZeZg ZdZdZ dWe
e e
e e
e e
e e
e e
e
e
eee f e
e e
ee eeef e
e d
ddZdd Zd
d Zee
e d d
dZeed ddZe d ddZdXeee f d ddZ e ee dd Z edd Z dYe d ddZ!d d Z"e#j$e d d d Z%ed d! Z&dZe
e' e
e( eee
e) e
e e
eee f d"d#d$Z*d%d& Z+e'd'd(d)Z,d*d+ Z-d,d- Z.e d d.d/Z/d0d1 Z0d2d3 Z1d4d5 Z2d[e
e3 ee4e5f d6d7d8Z6d\ee e7e3f eeed9d:d;Z8e3j9dfee7e3f ee4d<d=d>Z:ee7e3f e d6d?d@Z;d]e
e e
e eee=d dDdEZ?e4e@e e f e
e4 dFdGdHZAe e
Parameter `name` was renamed to `config_name`.
hash (`str`, *optional*): Hash specific to the dataset code. Used to update the caching directory when the
dataset loading script code is updated (to avoid reusing old data).
The typical caching directory (defined in ``self._relative_data_dir``) is: ``name/version/hash/``.
base_path (`str`, *optional*): Base path for relative paths that are used to download files.
This can be a remote URL.
features ([`Features`], *optional*): Features types to use with this dataset.
It can be used to change the Features types of a dataset, for example.
use_auth_token (`str` or `bool`, *optional*): String or boolean to use as Bearer token for remote files on the
Datasets Hub. If `True`, will get token from ``"~/.huggingface"``.
repo_id (`str`, *optional*): ID of the dataset repository.
Used to distinguish builders with the same name but not coming from the same namespace, for example "squad"
and "lhoestq/squad" repo IDs. In the latter, the builder name would be "lhoestq___squad".
data_files (`str` or `Sequence` or `Mapping`, *optional*): Path(s) to source data file(s).
For builders like "csv" or "json" that need the user to specify data files. They can be either
local or remote files. For convenience, you can use a DataFilesDict.
data_dir (`str`, *optional*): Path to directory containing source data file(s).
Use only if `data_files` is not passed, in which case it is equivalent to passing
``os.path.join(data_dir, "**")`` as `data_files`.
For builders that require manual download, it must be the path to the local directory containing the
manually downloaded data.
name (`str`): Configuration name for the dataset.
Use `config_name` instead.
**config_kwargs (additional keyword arguments): Keyword arguments to be passed to the corresponding builder
configuration class, set on the class attribute [`DatasetBuilder.BUILDER_CONFIG_CLASS`]. The builder
configuration class is [`BuilderConfig`] or a subclass of it.
NF
deprecated)
cache_dir config_namerm base_pathinfofeaturesuse_auth_tokenrepo_idrD rW c
K s | dkrt jdtd | }t| jjdd | _|| _|| _|| _ || _
| d k rpt | t rpt j
t| ||d} dtj| jjjkr|d k r|| d< | d k r| | d< |
d k r|
| d < | j|fd
|i| \| _| _|d kr| j }|j| j | j|_| jj|_| jj |_ || _ |d k r&|| j _ t |p2tj | _!t"| j!rJ| j!n t#j$j%| j!| _!t"| j!rlt&j'nt#j$j'}
|r|
| j!tj(nt tj)| _*t"| j*r| j*n t#j$j%| j*| _*| j+ | _,t"| j!sxt#j-| j!d d t#j$j'| j!| j,j.t#j/d
d }t0|n t#j$j1| j,rnt2t#j3| j,dkrFt4j d t5j6| j,| _ n(t4j7d| j, d| j d t#j8| j, W d Q R X d | _9d| _:t;| d S )Nry z\Parameter 'name' was renamed to 'config_name' in version 2.3.0 and will be removed in 3.0.0.)category.r )r| r r~ rD rW rT T)exist_ok_z.lockr z2Overwrite dataset info from restored data version.zOld caching folder z
for dataset z. exists but not data were found. Removing it. F)<warningswarn
FutureWarningr' r; splitrC rm r| r r rE r Zfrom_local_or_remoter inspect signatureBUILDER_CONFIG_CLASS__init__
parameters_create_builder_configr
rr get_exported_dataset_inforn _info builder_namer{ rV r} r~ rZ ZHF_DATASETS_CACHE_cache_dir_rootr. ospath
expanduser posixpathrj ZDOWNLOADED_DATASETS_DIRZDOWNLOADED_DATASETS_PATH_cache_downloaded_dir_build_cache_dir
_cache_dirmakedirsreplacesepr/ existsrl listdirloggerr from_directorywarningrmdir
dl_manager
_record_infosr+ )rG rz r{ rm r| r} r~ r r rD rW rC rS path_join lock_pathr= r= r> r sl
"
zDatasetBuilder.__init__c C s | j S )N)rO )rG r= r= r> __getstate__e s zDatasetBuilder.__getstate__c C s || _ t| d S )N)rO r+ )rG dr= r= r> __setstate__h s zDatasetBuilder.__setstate__)rU c C s d S )Nr= )rG r= r= r> manual_download_instructionss s z+DatasetBuilder.manual_download_instructionsc C s2 t jj| j tj}t jj|r.tj| j S i S )a" Empty dict if doesn't exist
Example:
```py
>>> from datasets import load_dataset_builder
>>> ds_builder = load_dataset_builder('rotten_tomatoes')
>>> ds_builder.get_all_exported_dataset_infos()
{'default': DatasetInfo(description="Movie Review Dataset.
This is a dataset of containing 5,331 positive and 5,331 negative processed
sentences from Rotten Tomatoes movie reviews. This data was first used in Bo
Pang and Lillian Lee, ``Seeing stars: Exploiting class relationships for
sentiment categorization with respect to rating scales.'', Proceedings of the
ACL, 2005.
", citation='@InProceedings{Pang+Lee:05a,
author = {Bo Pang and Lillian Lee},
title = {Seeing stars: Exploiting class relationships for sentiment
categorization with respect to rating scales},
booktitle = {Proceedings of the ACL},
year = 2005
}
', homepage='http://www.cs.cornell.edu/people/pabo/movie-review-data/', license='', features={'text': Value(dtype='string', id=None), 'label': ClassLabel(num_classes=2, names=['neg', 'pos'], id=None)}, post_processed=None, supervised_keys=SupervisedKeysData(input='', output=''), task_templates=[TextClassification(task='text-classification', text_column='text', label_column='label')], builder_name='rotten_tomatoes_movie_review', config_name='default', version=1.0.0, splits={'train': SplitInfo(name='train', num_bytes=1074810, num_examples=8530, dataset_name='rotten_tomatoes_movie_review'), 'validation': SplitInfo(name='validation', num_bytes=134679, num_examples=1066, dataset_name='rotten_tomatoes_movie_review'), 'test': SplitInfo(name='test', num_bytes=135972, num_examples=1066, dataset_name='rotten_tomatoes_movie_review')}, download_checksums={'https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz': {'num_bytes': 487770, 'checksum': 'a05befe52aafda71d458d188a1c54506a998b1308613ba76bbda2e5029409ce9'}}, download_size=487770, post_processing_size=None, dataset_size=1345461, size_in_bytes=1833231)}
```
) r r rj get_imported_module_dirr
ZDATASETDICT_INFOS_FILENAMEr r r )clsZdset_infos_file_pathr= r= r> get_all_exported_dataset_infosw s
z-DatasetBuilder.get_all_exported_dataset_infosc C s | j j| jjt S )a Empty DatasetInfo if doesn't exist
Example:
```py
>>> from datasets import load_dataset_builder
>>> ds_builder = load_dataset_builder('rotten_tomatoes')
>>> ds_builder.get_exported_dataset_info()
DatasetInfo(description="Movie Review Dataset.
This is a dataset of containing 5,331 positive and 5,331 negative processed
sentences from Rotten Tomatoes movie reviews. This data was first used in Bo
Pang and Lillian Lee, ``Seeing stars: Exploiting class relationships for
sentiment categorization with respect to rating scales.'', Proceedings of the
ACL, 2005.
", citation='@InProceedings{Pang+Lee:05a,
author = {Bo Pang and Lillian Lee},
title = {Seeing stars: Exploiting class relationships for sentiment
categorization with respect to rating scales},
booktitle = {Proceedings of the ACL},
year = 2005
}
', homepage='http://www.cs.cornell.edu/people/pabo/movie-review-data/', license='', features={'text': Value(dtype='string', id=None), 'label': ClassLabel(num_classes=2, names=['neg', 'pos'], id=None)}, post_processed=None, supervised_keys=SupervisedKeysData(input='', output=''), task_templates=[TextClassification(task='text-classification', text_column='text', label_column='label')], builder_name='rotten_tomatoes_movie_review', config_name='default', version=1.0.0, splits={'train': SplitInfo(name='train', num_bytes=1074810, num_examples=8530, dataset_name='rotten_tomatoes_movie_review'), 'validation': SplitInfo(name='validation', num_bytes=134679, num_examples=1066, dataset_name='rotten_tomatoes_movie_review'), 'test': SplitInfo(name='test', num_bytes=135972, num_examples=1066, dataset_name='rotten_tomatoes_movie_review')}, download_checksums={'https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz': {'num_bytes': 487770, 'checksum': 'a05befe52aafda71d458d188a1c54506a998b1308613ba76bbda2e5029409ce9'}}, download_size=487770, post_processing_size=None, dataset_size=1345461, size_in_bytes=1833231)
```
)r getr
rC r )rG r= r= r> r s z(DatasetBuilder.get_exported_dataset_infoc
K s@ d}|dkr| j r| r| jdk rL| jj| j}tjd| j d|j nrt| j dkrd| j d| j d j d}td t | jj
d
| d | j d }tj d | j d|j t |t
r| jj|}|dko| j rtd
| dt | jj
|sR|dk r ||d< d|krDt| drD| jrD| j|d< | jf |}nZtj|}xN|j D ]B\}}|dk rft||std
| d| dt||| qfW |jstd|j |j||d}|| jk} | rtjd| nD|| j|j kr tdt | jj
|js8td
|j d||fS )a Create and validate BuilderConfig object as well as a unique config id for this config.
Raises ValueError if there are multiple builder configs and name and DEFAULT_CONFIG_NAME are None.
config_kwargs override the defaults kwargs in config
Nz$No config specified, defaulting to: /r zload_dataset('z', 'r z')zEConfig name is missing.
Please pick one among the available configs: z
Example of usage:
``z6No config specified, defaulting to the single config: zBuilderConfig z not found. Available: rC rV VERSIONz doesn't have a 'z' key.z$BuilderConfig must have a name, got )rT z Using custom data configuration zvCannot name a custom BuilderConfig the same as an available BuilderConfig. Change the name. Available BuilderConfigs: z must have a version)BUILDER_CONFIGSDEFAULT_CONFIG_NAMEbuilder_configsr r r rC rl rF listrP r} rE rZ hasattrr r rf deepcopyrk setattrrs rV )
rG rC rT rS builder_configZexample_of_usagekeyvaluerr Z is_customr= r= r> r sT
z%DatasetBuilder._create_builder_configc C sD dd | j D }t|t| j kr@dd | j D }td| |S )z:Pre-defined list of configurations for this builder class.c S s i | ] }||j qS r= )rC )rJ r
r= r= r> rY s z2DatasetBuilder.builder_configs..c S s g | ]
}|j qS r= )rC )rJ r
r= r= r>
s z2DatasetBuilder.builder_configs..z5Names in BUILDER_CONFIGS must not be duplicated. Got )r rl rF )r Zconfigsnamesr= r= r> r s
z DatasetBuilder.builder_configsc C s | j S )N)r )rG r= r= r> rz s zDatasetBuilder.cache_dirTc C s | j r&| j jddkr&| j jdd nd}|dkr8| jn| d| j }| j}| j}|r`tjjnt j}|rv||| j
}|r||t | jj }|r|rt
|t r|||}|S )ao Relative path of this dataset in cache_dir:
Will be:
self.name/self.config.version/self.hash/
or if a repo_id with a namespace has been specified:
self.namespace___self.name/self.config.version/self.hash/
If any of these element is missing or if ``with_version=False`` the corresponding subfolders are dropped.
r r NZ___)r countr rC r
rm r r rj r rr rZ rV rE ) rG with_version with_hashis_local namespacebuilder_data_dirr rm r r= r= r> _relative_data_dir s *
z!DatasetBuilder._relative_data_dirc s t | j }|rtjjntj}|| j| jd|d || j| jd|d} fdd}t s| }|r|d d }|| jjkrdt | d| j
d | j d
t | jj d }t j | |S ) z2Return the data directory for the current version.F)r r Tc sf t jj sg S g } x@t j D ]2}y| jtj||f W q tk
rP Y q X q W | jdd | S )z"Returns previous versions on disk.T)reverse) r r r r appendr ru rF sort)Zversion_dirnamesdir_name)r r= r> _other_versions_on_disk
s
z@DatasetBuilder._build_cache_dir.._other_versions_on_diskr zFound a different version z of dataset z in cache_dir z". Using currently defined version r )
r. r r r rj r r r
rV rZ rC r r )rG r r Zversion_data_dirr Z version_dirsZ
other_versionZwarn_msgr= )r r> r s .
z DatasetBuilder._build_cache_dirc C s t dS )a Construct the DatasetInfo object. See `DatasetInfo` for details.
Warning: This function is only called once and the result is cached for all
following .info() calls.
Returns:
info: (DatasetInfo) The dataset information
N)NotImplementedError)rG r= r= r> r * s
zDatasetBuilder._infoc C s t jjtjtj| S )z8Return the path of the module of this class or subclass.)r r dirnamer getfile getmodule)r r= r= r> r 6 s z&DatasetBuilder.get_imported_module_dir)download_config
download_modeignore_verificationstry_from_hf_gcsr r| r c # K s t |p
t j}| } |dk r |n| j}|dkr|dkr^t| jt|t jkt|t jkd|d}t| j|| j j
|| j s~| j p| ndd}nt
|trd}|| _t| j }
|
rtjj| j| jjtjdd } |
rt| ntj |
r>tjj| j} | r>|t jkr>t j d| j d| j d | j | _ | j | dS t j d
| j d| j d |
rt!| j j"pnd | jd st#d
t$| j j"pd dt$| j j%pd dt$| j j&pd dt$| j j'pd d tj(dd }
| j j"rLt)d| j j* d| j j+ dt$| j j% dt$| j j& dt$| j j' dt$| j j" d| j d n&t)d| j j* d| j j+ d| j d | j,| |
| j}t-| d| d}|ry| j.|j/ d}W nB t0t1fk
r t j d Y n t2k
r t j d Y nX |s| j3f || d | t4d d | j j5j6 D | j _&|j7 | j _8| j j&| j j% | j _"| j9 W dQ R X W dQ R X | j | t)d | j d!| j d" W dQ R X dS )#a Downloads and prepares dataset for reading.
Args:
download_config (:class:`DownloadConfig`, optional): specific download configuration parameters.
download_mode (:class:`DownloadMode`, optional): select the download/generate mode - Default to ``REUSE_DATASET_IF_EXISTS``
ignore_verifications (:obj:`bool`): Ignore the verifications of the downloaded/processed dataset information (checksums/size/splits/...)
try_from_hf_gcs (:obj:`bool`): If True, it will try to download the already prepared dataset from the Hf google cloud storage
dl_manager (:class:`DownloadManager`, optional): specific Download Manger to use
base_path (:obj:`str`, optional): base path for relative paths that are used to download files. This can be a remote url.
If not specified, the value of the `base_path` attribute (`self.base_path`) will be used instead.
use_auth_token (:obj:`Union[str, bool]`, optional): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If True, will get token from ~/.huggingface.
**download_and_prepare_kwargs (additional keyword arguments): Keyword arguments.
Example:
```py
>>> from datasets import load_dataset_builder
>>> builder = load_dataset_builder('rotten_tomatoes')
>>> ds = builder.download_and_prepare()
```
NF)rz Zforce_downloadZ
force_extractZuse_etagr ) dataset_namer rW r| record_checksumsr z.lockzReusing dataset z ()zGenerating dataset r ) directoryz Not enough disk space. Needed: z (download: z
, generated: z, post-processed: c s sp t | r| V n\| d }tj|dd z,|V tjj| rDtj| tj|| W dtjj|rjtj| X dS )z4Create temporary dir for dirname and rename on exit.z .incompleteT)r N) r. r r r isdirshutilrmtreerenamer )r tmp_dirr= r= r> incomplete_dir s
z;DatasetBuilder.download_and_prepare..incomplete_dirz"Downloading and preparing dataset r z , total: z) to z...z to r TzJDataset not on Hf google storage. Downloading and preparing it from sourcezGHF google storage unreachable. Downloading and preparing it from source)r verify_infosc s s | ] }|j V qd S )N) num_bytes)rJ r r= r= r> rM s z6DatasetBuilder.download_and_prepare..zDataset z downloaded and prepared to z(. Subsequent calls will reuse this data.):r ZREUSE_DATASET_IF_EXISTSr| r r r[ ZFORCE_REDOWNLOADr rC r
rW $SKIP_CHECKSUM_COMPUTATION_BY_DEFAULTr rE r r r. r r r rj r r r r/
contextlibZ nullcontextr r r
_load_infor} "download_post_processing_resourcesr4
size_in_bytesOSErrorr7
download_size dataset_sizepost_processing_sizecontextmanagerprintr r{ _check_manual_downloadr8 _download_prepared_from_hf_gcsr r r ConnectionError_download_and_preparesumsplitsri get_recorded_sizes_checksumsdownload_checksums
_save_info)rG r r r r r r| r Zdownload_and_prepare_kwargsr r r Z data_existsr Z tmp_data_dirZdownloaded_from_gcsr= r= r> download_and_prepare; s !
X
^$
z#DatasetBuilder.download_and_preparec C sJ | j d k rF|jd krFttjd| j d| jj d| j d| j d d S )Nz The dataset z
with config zp requires manual data.
Please follow the manual download instructions:
za
Manual data can be loaded with:
datasets.load_dataset("z$", data_dir=""))r Z
manual_dirr@ textwrapdedentrC r
)rG r r= r= r> r s z%DatasetBuilder._check_manual_download)r c
C s | j ddd}t| j| j}|j|| tj| j}| jj| td |j t
j d }x| jj D ]}x| j
|j D ]p}t
j |krtd| y,t|d | }tj|t
jj| j| W qt tk
r tjd| d Y qtX qtW q`W tjd d S ) NTF)r r r z+Resources shouldn't be in a sub-directory: z Couldn't download resourse file z from Hf google storage.z*Dataset downloaded from Hf google storage.)r r r r} Zdownload_from_hf_gcsr r rn r
r r r r _post_processing_resourcesri rF r- r mover rj r r ) rG r Zrelative_data_dirreaderZdownloaded_infoZremote_cache_dirr resource_file_name
resource_pathr= r= r> r s
z-DatasetBuilder._download_prepared_from_hf_gcsc K sV t | jd}| j|}| j|f|}|rB|jrBt| jj|j d x|D ]}t |j
jj dkrht dt
jd|j
j d |j|j
y| j|f| W n tk
r } z&td| jpd d t | d
W Y d
d
}~X nB tk
r } z$t|j|jd | j d d
d
W Y d
d
}~X nX |j qHW |r@t| jj| || j_|j| j_d
S )a Downloads and prepares dataset for reading.
This is the internal implementation to overwrite called when user calls
`download_and_prepare`. It should download all required data and generate
the pre-processed datasets files.
Args:
dl_manager: (DownloadManager) `DownloadManager` used to download and cache
data.
verify_infos: bool, if False, do not perform checksums and size tests.
prepare_split_kwargs: Additional options.
)r zdataset source filesrQ z{`all` is a special split keyword corresponding to the union of all splits, so cannot be used as key in ._split_generator().z Generating z splitzCannot find data file. z
Original error:
Nz7To avoid duplicate keys, please fix the dataset script z.py)Zfix_msg)r) rC _make_split_generators_kwargs_split_generatorsr r1 r} r r rZ
split_infolowerrF r add_prepare_splitr r r% r Zduplicate_key_indicesZmanage_extracted_filesr2 r Zdownloaded_sizer ) rG r r prepare_split_kwargs
split_dictsplit_generators_kwargsZsplit_generatorssplit_generatorer= r= r> r s:
z$DatasetBuilder._download_and_preparec C s x| j jD ]}x| j|j D ]p\}}tj|kr>td| tjj| j |}tjj
|s | j |||}|r t j d| d| t
j|| q W q
W d S )Nz+Resources shouldn't be in a sub-directory: z$Downloaded post-processing resource z as )r} r r rk r r rF r rj r r #_download_post_processing_resourcesr r r )rG r r
resource_namer r Zdownloaded_resource_pathr= r= r> r 1 s
z1DatasetBuilder.download_post_processing_resourcesc C s t j| jS )N)r r r )rG r= r= r> r ? s zDatasetBuilder._load_infoc C sF t jj| j| jjt jdd }t| | jj | j W d Q R X d S )Nr z.lock)
r r rj r r r r r/ r} write_to_directory)rG r r= r= r> r B s
zDatasetBuilder._save_infoc C sV t jj| j| jjt jdd }t|$ tf | j j
| j ij | j
W d Q R X d S )Nr z.lock)r r rj r r r r r/ r r
rC r} r r )rG r r= r= r> _save_infosG s
zDatasetBuilder._save_infosc C s ~i S )zFGet kwargs for `self._split_generators()` from `prepare_split_kwargs`.r= )rG r r= r= r> r L s z,DatasetBuilder._make_split_generators_kwargs)r rU c C s t jj| js(td| j d| j dtjd|p>dj | j
j d| j |dkrhdd | j
j D }t t
| j|||d
|d tj d }t|trt|}|S )
a Return a Dataset for the specified split.
Args:
split (`datasets.Split`): Which subset of the data to return.
run_post_process (bool, default=True): Whether to run post-processing dataset transforms and/or add
indexes.
ignore_verifications (bool, default=False): Whether to ignore the verifications of the
downloaded/processed dataset information (checksums/size/splits/...).
in_memory (bool, default=False): Whether to copy the data in-memory.
Returns:
datasets.Dataset
Example:
```py
>>> from datasets import load_dataset_builder
>>> builder = load_dataset_builder('rotten_tomatoes')
>>> ds = builder.download_and_prepare()
>>> ds = builder.as_dataset(split='train')
>>> ds
Dataset({
features: ['text', 'label'],
num_rows: 8530
})
```
zDataset z: could not find data in z. Please make sure to call builder.download_and_prepare(), or pass download=True to datasets.load_dataset() before trying to access the Dataset object.z Constructing Dataset for split z, z, from Nc S s i | ]
}||qS r= r= )rJ sr= r= r> rY z s z-DatasetBuilder.as_dataset..)run_post_processr in_memoryT) map_tupleZ disable_tqdm)r r r r AssertionErrorrC r r debugrj r} r r5 r _build_single_datasetr, is_progress_bar_enabledrE rw r )rG r r r r
datasetsr= r= r>
as_datasetQ s$ &
zDatasetBuilder.as_dataset)r r r r
c sJ | }t |ts:t|}|dkr2dj jjj }t|} j||d}|rFx. j |j
D ] }t j |kr^t
d| q^W fdd j |j D } j||} | dk rF| }i }
d} x$|j D ]\} }
t|
}||
| < qW |o| r, jjdks
jjjdkrd}n jjjj|}t||
d jjdkrDt j_ jjjdkr^i jj_|
jjjt|< td
d jjjj
D j_ jjdk rȈ jjdk rȈ jj jj jj j_ j jj|j _ jj|j _ jj|j _ jjj dk rF jjj j |j j kr8t
d jjj d
|j n jjj |j_ |S )z as_dataset for a single split.rQ +)r r
z+Resources shouldn't be in a sub-directory: c s" i | ]\}}t jj j||qS r= )r r rj r )rJ r r )rG r= r> rY s z8DatasetBuilder._build_single_dataset..NFzpost processing resourcesc s s$ | ] }|j D ]}|d V qqdS )r N)ri )rJ Zsplit_checksums_dictsZchecksums_dictr= r= r> rM s z7DatasetBuilder._build_single_dataset..z:Post-processed features info don't match the dataset:
Got
z
but expected something like
) rE r rZ rj r} r rP r( _as_datasetr ri r r rF rk
_post_processr0 post_processedZresources_checksumsr r1 r! r r r r r r r r~ type)rG r r r r
r Zdsr resources_pathsr Zrecorded_checksumsr r r Z
size_checksumZexpected_checksumsr= )rG r> r s`
z$DatasetBuilder._build_single_dataset)r r
rU c C sB t | j| jj| j|| jjj |d}| j|}tf d|i|S )a Constructs a `Dataset`.
This is the internal implementation to overwrite called when user calls
`as_dataset`. It should read the pre-processed datasets files and generate
the `Dataset` object.
Args:
split: `datasets.Split` which subset of the data to read.
in_memory (bool, default False): Whether to copy the data in-memory.
Returns:
`Dataset`
)rC Z instructionsZ split_infosr
fingerprint) r r r} readrC r ri _get_dataset_fingerprintr )rG r r
Zdataset_kwargsr r= r= r> r s
zDatasetBuilder._as_datasetc C s8 t }|j| j jtjd |jt| |j }|S )zThe dataset fingerprint is the hash of the relative directory dataset_name/config_name/version/hash, as well as the split specs.r )r rn r r r r rZ ro )rG r Zhasherr r= r= r> r s
z'DatasetBuilder._get_dataset_fingerprint)r r| rU c C s t | ttfs td| j dt|p*| jt| jd| j| j j
d}| j | dd | j |D }|d krp|}n*||kr|| }ntd| dt
| t| j|d d
}t |trt|}|S ) NzBuilder z is not streamable.)r )r| r r rW c S s i | ] }||j qS r= )rC )rJ Zsgr= r= r> rY s z7DatasetBuilder.as_streaming_dataset..z Bad split: z. Available splits: T)r )rE GeneratorBasedBuilderArrowBasedBuilderrF rC r r| r r r
rW r r r r5 _as_streaming_dataset_singlerw r )rG r r| r Zsplits_generatorssplits_generatorr r= r= r> as_streaming_dataset s*
z#DatasetBuilder.as_streaming_datasetc C s4 | j |}| jr | j| jini }t|| j|j|dS )N)r} r token_per_repo_id) _get_examples_iterable_for_splitr r r# r} rC )rG r! Z ex_iterabler# r= r= r> r s
z+DatasetBuilder._as_streaming_dataset_single)datasetr rU c C s dS )z%Run dataset transforms or add indexesNr= )rG r% r r= r= r> r s z DatasetBuilder._post_processc C s i S )z+Mapping resource_name -> resource_file_namer= )rG r r= r= r> r # s z)DatasetBuilder._post_processing_resources)r r r rU c C s dS )zPDownload the resource using the download manager and return the downloaded path.Nr= )rG r r r r= r= r> r ' s z2DatasetBuilder._download_post_processing_resources)r c C s
t dS )a Specify feature dictionary generators and dataset splits.
This function returns a list of `SplitGenerator`s defining how to generate
data and what splits to use.
Example::
return [
datasets.SplitGenerator(
name=datasets.Split.TRAIN,
gen_kwargs={'file': 'train_data.zip'},
),
datasets.SplitGenerator(
name=datasets.Split.TEST,
gen_kwargs={'file': 'test_data.zip'},
),
]
The above code will first call `_generate_examples(file='train_data.zip')`
to write the train data, then `_generate_examples(file='test_data.zip')` to
write the test data.
Datasets are typically split into different subsets to be used at various
stages of training and evaluation.
Note that for datasets without a `VALIDATION` split, you can use a
fraction of the `TRAIN` data for evaluation as you iterate on your model
so as not to overfit to the `TEST` data.
For downloads and extractions, use the given `download_manager`.
Note that the `DownloadManager` caches downloads, so it is fine to have each
generator attempt to download the source data.
A good practice is to download all data in this function, and then
distribute the relevant parts to each split with the `gen_kwargs` argument
Args:
dl_manager: (DownloadManager) Download manager to download the data
Returns:
`list`.
N)r )rG r r= r= r> r - s ,z DatasetBuilder._split_generators)r c K s
t dS )a Generate the examples and record them on disk.
Args:
split_generator: `SplitGenerator`, Split generator to process
**kwargs: Additional kwargs forwarded from _download_and_prepare (ex:
beam pipeline)
N)r )rG r kwargsr= r= r> r [ s z DatasetBuilder._prepare_split)r rU c C s
t dS )zGenerate the examples on the fly.
Args:
split_generator: `SplitGenerator`, Split generator to process
N)r )rG r r= r= r> r$ f s z/DatasetBuilder._get_examples_iterable_for_split) NNNNNNNNNNry )NN)TTT)NNFTNNN)NTFF)F)NN)Ir: r; r< rt r rA r r r r r rZ r r r r[ r rw r r r r propertyr classmethodr r r r r3 r6 r rz r r abcabstractmethodr r r r r r r r r r r r r
r r( r r r r r ZTRAINr r r r# r" r r r r r r r* r r" r$ r= r= r= r> rx s ; ^X F ( 8
>> A"
! -
rx c s` e Zd ZdZdZdZdd fdd
Zejdd Z d d
Z
fd d Z e e
d
ddZ ZS )r aw Base class for datasets with data generation based on dict generators.
`GeneratorBasedBuilder` is a convenience class that abstracts away much
of the data writing and reading of `DatasetBuilder`. It expects subclasses to
implement generators of feature dictionaries across the dataset splits
(`_split_generators`). See the method docstrings for details.
TN)writer_batch_sizec s t j|| |p| j| _d S )N)superr DEFAULT_WRITER_BATCH_SIZE_writer_batch_size)rG r+ argsr& ) __class__r= r> r s z GeneratorBasedBuilder.__init__c K s
t dS )aW Default function generating examples for each `SplitGenerator`.
This function preprocess the examples from the raw data to the preprocessed
dataset files.
This function is called once for each `SplitGenerator` defined in
`_split_generators`. The examples yielded here will be written on
disk.
Args:
**kwargs (additional keyword arguments): Arguments forwarded from the SplitGenerator.gen_kwargs
Yields:
key: `str` or `int`, a unique deterministic example identification key.
* Unique: An error will be raised if two examples are yield with the
same key.
* Deterministic: When generating the dataset twice, the same example
should have the same key.
Good keys can be the image id, or line number if examples are extracted
from a text file.
The key will be hashed and sorted to shuffle examples deterministically,
such as generating the dataset multiple times keep examples in the
same order.
example: `dict`, a feature dictionary
ready to be encoded and written to disk. The example will be
encoded with `self.info.features.encode_example({...})`.
N)r )rG r& r= r= r> _generate_examples s z(GeneratorBasedBuilder._generate_examplesc
C s | j jd k r | j j|j }n|j}| j d|j d}tjj| j|}| jf |j }t
| j j || j |j|dj}zTxNt
j|d|jdt
j d|j ddD ]"\}} | j j j| }
|j|
| qW W d |j \} } X W d Q R X | |j_| |j_d S ) Nre z.arrow)r~ r r+ Z hash_saltZcheck_duplicatesz examplesFz Generating z split)unittotalleavedisabledesc)r} r rC r r r rj r r1
gen_kwargsr r~ r. r, tqdm num_examplesr encode_examplewritefinalizer )
rG r check_duplicate_keysr fnamefpath generatorwriterr recordZexampler9 r r= r= r> r s4 z$GeneratorBasedBuilder._prepare_splitc s t j|||d d S )N)r= )r, r )rG r r )r0 r= r> r s z+GeneratorBasedBuilder._download_and_prepare)r rU c C s t | j|jS )N)r" r1 r7 )rG r r= r= r> r$ s z6GeneratorBasedBuilder._get_examples_iterable_for_split)r: r; r< rt test_dummy_datar- r r) r* r1 r r r* r" r$
__classcell__r= r= )r0 r> r o s # r c @ s: e Zd ZdZdZejdd Zdd Ze e
ddd Z d
S ) r zaBase class for datasets with data generation based on Arrow loading functions (CSV/JSON/Parquet).Tc K s
t dS )a Default function generating examples for each `SplitGenerator`.
This function preprocess the examples from the raw data to the preprocessed
dataset files.
This function is called once for each `SplitGenerator` defined in
`_split_generators`. The examples yielded here will be written on
disk.
Args:
**kwargs (additional keyword arguments): Arguments forwarded from the SplitGenerator.gen_kwargs
Yields:
key: `str` or `int`, a unique deterministic example identification key.
* Unique: An error will be raised if two examples are yield with the
same key.
* Deterministic: When generating the dataset twice, the same example
should have the same key.
Good keys can be the image id, or line number if examples are extracted
from a text file.
The key will be hashed and sorted to shuffle examples deterministically,
such as generating the dataset multiple times keep examples in the
same order.
example: `pyarrow.Table`, a feature table
ready to be encoded and written to disk.
N)r )rG r& r= r= r> _generate_tables s z"ArrowBasedBuilder._generate_tablesc
C s | j d|j d}tjj| j|}| jf |j}t| jj |dB}x.t
j |ddt
j dD ]\}}|j
| q^W |j \}} W d Q R X ||j_| |j_| jj d kr|j| j_ d S )Nre z.arrow)r~ r z tablesF)r2 r4 r5 )rC r r rj r rE r7 r r} r~ r, r8 r Z write_tabler< r r9 r Z _features)
rG r r> r? r@ rA r tabler9 r r= r= r> r s z ArrowBasedBuilder._prepare_split)r rU c C s t t| j|jdS )N)r& )r" r$ rE r7 )rG r r= r= r> r$ s z2ArrowBasedBuilder._get_examples_iterable_for_splitN) r: r; r< rt rC r) r* rE r r* r" r$ r= r= r= r> r s
r c @ s e Zd ZdS )MissingBeamOptionsN)r: r; r< r= r= r= r> rG s rG c sb e Zd ZdZdZddd fdd
Zdd Zejd d
Z fd d Z
fd
dZ dd Z Z
S )BeamBasedBuilderzBeam based Builder.FN) beam_runner beam_optionsc s$ || _ || _i | _t j|| d S )N) _beam_runner
_beam_options
_beam_writersr, r )rG rI rJ r/ r& )r0 r= r> r s zBeamBasedBuilder.__init__c C s. i }t j| jjj }d|kr*|d |d< |S )Npipeline)r r r r rP )rG r r Zsplit_generators_arg_namesr= r= r> r s
z.BeamBasedBuilder._make_split_generators_kwargsc K s
t dS )a" Build the beam pipeline examples for each `SplitGenerator`.
This function extracts examples from the raw data with parallel transforms
in a Beam pipeline. It is called once for each `SplitGenerator` defined in
`_split_generators`. The examples from the PCollection will be
encoded and written to disk.
Warning: When running in a distributed setup, make sure that the data
which will be read (download_dir, manual_dir,...) and written (cache_dir)
can be accessed by the workers jobs. The data should be located in a
shared filesystem, like GCS.
Args:
pipeline ([`utils.beam_utils.BeamPipeline`]): Apache Beam pipeline.
**kwargs (additional keyword arguments): Arguments forwarded from the SplitGenerator.gen_kwargs.
Returns:
`beam.PCollection`: Apache Beam PCollection containing the
example to send to `self.info.features.encode_example(...)`.
Example:
```
def _build_pcollection(pipeline, extracted_dir=None):
return (
pipeline
| beam.Create(gfile.io.listdir(extracted_dir))
| beam.Map(_process_file)
)
```
N)r )rG rN r& r= r= r> _build_pcollection% s #z#BeamBasedBuilder._build_pcollectionc s dd l }dd ljj}| j}| j}| rT| rTd| j d| jj d}td| d|pb|j j
j }d|j |j j
j
_|j||d}t j|d|d |j } | j | j }
| jj} xP| jj D ]B\} }
|jj j| d
}|
j|
j |\}}| | }||_ ||_ qW d S ) Nr zload_dataset('z', 'z ', beam_runner='DirectRunner')a* Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
`r F)runneroptions)r rN )r ) apache_beamZdatasets.utils.beam_utilsr
beam_utilsrK rL rC r
rG rQ Zpipeline_optionsZPipelineOptionsZview_asZ TypeOptionsZpipeline_type_checkZ BeamPipeliner, r runZwait_until_finishmetricsr} r rM rk Z
MetricsFilterZwith_namespacer< queryr9 r )rG r r beamrS rI rJ Z
usage_examplerN Zpipeline_resultsrU r
split_name beam_writerZm_filterr9 r r )r0 r= r> r J s6 z&BeamBasedBuilder._download_and_preparec s t jj| jrt j nzdd l}|jjj }|j
t jj | jt j
}| jj| W d Q R X | jjr|j
t jj | jt j}| jj| W d Q R X d S )Nr )r r r r r, r rR ioZ filesystemsZ FileSystemscreaterj r
ZDATASET_INFO_FILENAMEr} Z
_dump_infolicenseZLICENSE_FILENAMEZ
_dump_license)rG rW fsf)r0 r= r> r ~ s
zBeamBasedBuilder._save_infoc s dd l jj}j d| d}tjjj|}tjj ||jdj
|< jj j j fdd}||| ? B }d S )Nr re z.arrow)r~ r r rz c s4 j | fj}|d jfdd? O }j|S )z+PTransformation which build a single split.ZEncodec s | d | d fS )Nr r r= )Zkey_ex)r: r= r> s zMBeamBasedBuilder._prepare_split.._build_pcollection..)rO r7 ZMapZwrite_from_pcollection)rN Zpcoll_examples)rW rY r: rG r r= r> rO s z;BeamBasedBuilder._prepare_split.._build_pcollection)
rR r rC r r rj r r r} r~ rM r: Z
ptransform_fn)rG r rN rX r> r? rO r r= )rW rY r: rG r r> r s
z BeamBasedBuilder._prepare_split)r: r; r< rt rC r r r) r* rO r r r rD r= r= )r0 r> rH s
% 4
rH )hrt r) r rf r r r r r ra r Z dataclassesr functoolsr typingr r r r r r r
r Z
arrow_datasetr Z arrow_readerr
r r r r Z arrow_writerr r rD r r Z dataset_dictr r Zdownload.download_configr Zdownload.download_managerr r Z download.mock_download_managerr Z#download.streaming_download_managerr r~ r r r r} r r r! Ziterable_datasetr" r# r$ Zkeyhashr% Znamingr& r' r r( r) r* Z streamingr+ r, Zutils.file_utilsr- r. Zutils.filelockr/ Zutils.info_utilsr0 r1 r2 Zutils.py_utilsr3 r4 r5 r6 r7 r8
get_loggerr: r rF r9 Exceptionr? r@ rA rx r r rG rH r= r= r= r> sj
` Ab: