/usr/local/lib/python3.6/site-packages/datasets/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/datasets/__pycache__/inspect.cpython-36.pyc (18583B)
3
<%EgP
@ sr d Z ddlZddlZddlZddlmZ ddlmZmZm Z m
Z
m Z m Z ddl
Z
ddlmZ ddlmZ ddlmZ dd lmZ dd
lmZmZmZmZ dd lm Z dd l m Z dd
l m Z e e!Z"G dd de#Z$d'ddZ%d(ddZ&d)e'e'e
e dddZ(d*e'e'e
e dddZ)d+e'e
e eee'f e
e e
e e
e e'e f e
e e*e'f dd d Z+d,e'e
e e'e f e
e e
e e
e' e
e eee'f d d d Z,d-e'e
e' e
e e'e e' e e'e e'e e' f f f e
e e
e e
e e'e f e
e e*e'f ed!d"d#Z-d.e'e
e' e
e e'e e' e e'e e'e e' f f f e
e e
e e
e e'e f e
e e*e'f d$d%d&Z.dS )/z' List and inspect datasets and metrics. N)PurePath)DictListMappingOptionalSequenceUnion )DownloadConfig) DownloadMode)StreamingDownloadManager) DatasetInfo)dataset_module_factoryimport_main_classload_dataset_buildermetric_module_factory)relative_to_absolute_path)
get_logger)Versionc @ s e Zd ZdS )SplitsNotFoundErrorN)__name__
__module__ __qualname__ r r :/usr/local/lib/python3.6/site-packages/datasets/inspect.pyr ' s r TFc C s4 t j|d}| s dd |D }|s0dd |D }|S )aG List all the datasets scripts available on the Hugging Face Hub.
Args:
with_community_datasets (:obj:`bool`, optional, default ``True``): Include the community provided datasets.
with_details (:obj:`bool`, optional, default ``False``): Return the full details on the datasets instead of only the short name.
Example:
```py
>>> from datasets import list_datasets
>>> list_datasets()
['acronym_identification',
'ade_corpus_v2',
'adversarial_qa',
'aeslc',
'afrikaans_ner_corpus',
'ag_news',
...
]
```
)fullc S s g | ]}d |j kr|qS )/)id).0datasetr r r
C s z!list_datasets..c S s g | ]
}|j qS r )r )r r r r r r E s )huggingface_hub
list_datasets)Zwith_community_datasets with_detailsZdatasetsr r r r" + s r" c C s0 t j }| sdd |D }|s,dd |D }|S )a List all the metrics script available on the Hugging Face Hub.
Args:
with_community_metrics (:obj:`bool`, optional, default ``True``): Include the community provided metrics.
with_details (:obj:`bool`, optional, default ``False``): Return the full details on the metrics instead of only the short name.
Example:
```py
>>> from datasets import list_metrics
>>> list_metrics()
['accuracy',
'bertscore',
'bleu',
'bleurt',
'cer',
'chrf',
...
]
```
c S s g | ]}d |j kr|qS )r )r )r metricr r r r a s z list_metrics..c S s g | ]
}|j qS r )r )r r$ r r r r c s )r! list_metrics)Zwith_community_metricsr# Zmetricsr r r r% I s r% )path
local_pathdownload_configc
K s t | fd|i|}t|jdd}tj|}tjj|}xtj|D ]\}} }
tjj |tjj
||} tj | dd dd | D | dd< x,|
D ]$} t j
tjj || tjj | | qW t j|| qDW t|}td| d | d
| d t|j d dS )
a
Allow inspection/modification of a dataset script by copying on local drive at local_path.
Args:
path (`str`): Path to the dataset processing script with the dataset builder. Can be either:
- a local path to processing script or the directory containing the script (if the script has the same name
as the directory),
e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'``.
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`list_datasets`])
e.g. ``'squad'``, ``'glue'`` or ``'openai/webtext'``.
local_path (`str`): Path to the local folder to copy the dataset script to.
download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters.
**download_kwargs (additional keyword arguments): Optional arguments for [`DownloadConfig`] which will override
the attributes of `download_config` if supplied.
r( T)r )exist_okc S s g | ]}|j ds|qS ).__)r* r+ )
startswith)r dirnamer r r r s z#inspect_dataset..Nz"The processing script for dataset z can be inspected at z. The main class is in zP. You can modify this processing script and use it with `datasets.load_dataset("z")`.)r r module_pathinspect
getsourcefileosr& r- walkjoinrelpathmakedirsshutilcopy2copystatr printr as_posix)
r& r' r( download_kwargsdataset_module builder_clsmodule_source_pathmodule_source_dirpathdirpathdirnames filenames dst_dirpathfilenamer r r inspect_datasetg s
$rE c
K s t | fd|i|}t|jdd}tj|}tjj|}xtj|D ]\}} }
tjj |tjj
||} tj | dd dd | D | dd< x,|
D ]$} t j
tjj || tjj | | qW t j|| qDW t|}td | d
| d | d t|j d
dS )a
Allow inspection/modification of a metric script by copying it on local drive at local_path.
Args:
path (``str``): path to the dataset processing script with the dataset builder. Can be either:
- a local path to processing script or the directory containing the script (if the script has the same name as the directory),
e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'``
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``)
e.g. ``'squad'``, ``'glue'`` or ``'openai/webtext'``
local_path (``str``): path to the local folder to copy the datset script to.
download_config (Optional ``datasets.DownloadConfig``): specific download configuration parameters.
**download_kwargs (additional keyword arguments): optional attributes for DownloadConfig() which will override the attributes in download_config if supplied.
r( F)r T)r) c S s g | ]}|j ds|qS )r* r+ )r* r+ )r, )r r- r r r r s z"inspect_metric..Nz"The processing scripts for metric z can be inspected at z. The main class is in zP. You can modify this processing scripts and use it with `datasets.load_metric("z")`.)r r r. r/ r0 r1 r& r- r2 r3 r4 r5 r6 r7 r8 r r9 r r: )
r& r' r( r; Z
metric_moduler= r> r? r@ rA rB rC rD r r r inspect_metric s
$rF )r&
data_filesr(
download_moderevisionuse_auth_tokenc s0 t d} fdd|D S )aI Get the meta information about a dataset, returned as a dict mapping config name to DatasetInfoDict.
Args:
path (``str``): path to the dataset processing script with the dataset builder. Can be either:
- a local path to processing script or the directory containing the script (if the script has the same name as the directory),
e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'``
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``)
e.g. ``'squad'``, ``'glue'`` or ``'openai/webtext'``
revision (Optional ``Union[str, datasets.Version]``):
If specified, the dataset module will be loaded from the datasets repository at this version.
By default:
- it is set to the local version of the lib.
- it will also try to load it from the main branch if it's not available at the local version of the lib.
Specifying a version that is different from your local version of the lib might cause compatibility issues.
download_config (:class:`DownloadConfig`, optional): Specific download configuration parameters.
download_mode (:class:`DownloadMode`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode.
data_files (:obj:`Union[Dict, List, str]`, optional): Defining the data_files of the dataset configuration.
use_auth_token (``str`` or :obj:`bool`, optional): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If True, will get token from `"~/.huggingface"`.
**config_kwargs (additional keyword arguments): optional attributes for builder class which will override the attributes if supplied.
Example:
```py
>>> from datasets import get_dataset_infos
>>> get_dataset_infos('rotten_tomatoes')
{'default': DatasetInfo(description="Movie Review Dataset.
This is a dataset of containing 5,331 positive and 5,331 negative processed
sentences from Rotten Tomatoes movie reviews...), ...}
```
)r& rI r( rH rG c s, i | ]$}t f |d |qS ))r& config_namerG r( rH rI rJ )get_dataset_config_info)r rK )
config_kwargsrG r( rH r& rI rJ r r
s z%get_dataset_infos..)get_dataset_config_names)r& rG r( rH rI rJ rM Z config_namesr )rM rG r( rH r& rI rJ r get_dataset_infos s ' rP )r& rI r( rH dynamic_modules_pathrG c K sD t | f|||||d|}t|j}t|jj pB|jjddgS )a Get the list of available config names for a particular dataset.
Args:
path (``str``): path to the dataset processing script with the dataset builder. Can be either:
- a local path to processing script or the directory containing the script (if the script has the same name as the directory),
e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'``
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``)
e.g. ``'squad'``, ``'glue'`` or ``'openai/webtext'``
revision (Optional ``Union[str, datasets.Version]``):
If specified, the dataset module will be loaded from the datasets repository at this version.
By default:
- it is set to the local version of the lib.
- it will also try to load it from the main branch if it's not available at the local version of the lib.
Specifying a version that is different from your local version of the lib might cause compatibility issues.
download_config (:class:`DownloadConfig`, optional): Specific download configuration parameters.
download_mode (:class:`DownloadMode`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode.
dynamic_modules_path (Optional str, defaults to HF_MODULES_CACHE / "datasets_modules", i.e. ~/.cache/huggingface/modules/datasets_modules):
Optional path to the directory in which the dynamic modules are saved. It must have been initialized with :obj:`init_dynamic_modules`.
By default the datasets and metrics are stored inside the `datasets_modules` module.
data_files (:obj:`Union[Dict, List, str]`, optional): Defining the data_files of the dataset configuration.
**download_kwargs (additional keyword arguments): optional attributes for DownloadConfig() which will override the attributes in download_config if supplied,
for example ``use_auth_token``
Example:
```py
>>> from datasets import get_dataset_config_names
>>> get_dataset_config_names("glue")
['cola',
'sst2',
'mrpc',
'qqp',
'stsb',
'mnli',
'mnli_mismatched',
'mnli_matched',
'qnli',
'rte',
'wnli',
'ax']
```
)rI r( rH rQ rG rK default)r r r. listZbuilder_configskeysZbuilder_kwargsget) r& rI r( rH rQ rG r; r< r= r r r rO s 4
rO )r& rK rG r( rH rI rJ returnc s t f||||||d|}|j} | jdkryH|r<|j nt }|dk rP||_ fdd|jt|j|dD | _W n, t k
r }
zt
d|
W Y dd}
~
X nX | S )aF Get the meta information (DatasetInfo) about a dataset for a particular config
Args:
path (``str``): path to the dataset processing script with the dataset builder. Can be either:
- a local path to processing script or the directory containing the script (if the script has the same name as the directory),
e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'``
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``)
e.g. ``'squad'``, ``'glue'`` or ``'openai/webtext'``
config_name (:obj:`str`, optional): Defining the name of the dataset configuration.
data_files (:obj:`str` or :obj:`Sequence` or :obj:`Mapping`, optional): Path(s) to source data file(s).
download_config (:class:`~download.DownloadConfig`, optional): Specific download configuration parameters.
download_mode (:class:`DownloadMode`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode.
revision (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset script to load:
- For datasets in the `huggingface/datasets` library on GitHub like "squad", the default version of the module is the local version of the lib.
You can specify a different version from your local version of the lib (e.g. "main" or "1.2.0") but it might cause compatibility issues.
- For community datasets like "lhoestq/squad" that have their own git repository on the Datasets Hub, the default version "main" corresponds to the "main" branch.
You can specify a different version that the default "main" by using a commit sha or a git tag of the dataset repository.
use_auth_token (``str`` or :obj:`bool`, optional): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If True, will get token from `"~/.huggingface"`.
**config_kwargs (additional keyword arguments): optional attributes for builder class which will override the attributes if supplied.
)namerG r( rH rI rJ Nc s i | ]}|j d |j qS ))rW Z dataset_name)rW )r Zsplit_generator)r& r r rN ` s z+get_dataset_config_info..) base_pathr( z>> from datasets import get_dataset_split_names
>>> get_dataset_split_names('rotten_tomatoes')
['train', 'validation', 'test']
```
)rK rG r( rH rI rJ )rL rS rZ rT ) r& rK rG r( rH rI rJ rM rY r r r get_dataset_split_namesk s )r^ )TF)TF)N)N)NNNNN)NNNNN)NNNNNN)NNNNNN)/__doc__r/ r1 r6 pathlibr typingr r r r r r r! Zdownload.download_configr
Zdownload.download_managerr Z#download.streaming_download_managerr rY r
loadr r r r Zutils.file_utilsr Z
utils.loggingr Z
utils.versionr r logger
ValueErrorr r" r% strrE rF boolrP rO rL r^ r r r r sV
&% H9 @= Z6