/usr/local/lib/python3.6/site-packages/datasets/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/datasets/__pycache__/info.cpython-36.pyc (17359B)
3
<%Eg$E @ sJ d Z ddlZddlZddlZddlZddlmZmZ ddlmZm Z m
Z
m Z ddl m
Z
ddlmZmZ ddlmZ dd lmZmZ dd
lmZ dd lmZ dd lmZm Z ee Z eG d
d dZ eG dd dZ G dd de!Z"G dd de!Z#eG dd dZ$eG dd dZ%G dd dee&e%f Z'eG dd d Z(dS ) a DatasetInfo and MetricInfo record information we know about a dataset and a metric.
This includes things that we know about the dataset statically, i.e.:
- description
- canonical location
- does it have validation and tests splits
- size
- etc.
This also includes the things that can and should be computed once we've
processed the dataset as well:
- number of examples (in each split)
- etc.
N) dataclassfield)DictListOptionalUnion )config)FeaturesValue) SplitDict) TaskTemplatetask_template_from_dict)Version)
get_logger)asdict
unique_valuesc @ s e Zd ZU dZedZedS )SupervisedKeysData N)__name__
__module__ __qualname__inputstroutput r r 7/usr/local/lib/python3.6/site-packages/datasets/info.pyr 2 s
r c @ s e Zd ZU dZedZedS )DownloadChecksumsEntryDatar N)r r r keyr valuer r r r r 8 s
r c @ s e Zd ZdZdS ) MissingCachedSizesConfigErrorz;The expected cached sizes of the download file are missing.N)r r r __doc__r r r r r > s r c @ s e Zd ZdZdS )NonMatchingCachedSizesErrorz/The prepared split doesn't have expected sizes.N)r r r r! r r r r r" B s r" c @ sB e Zd ZU dZee dZee dd Ze ed dddZ
dS )PostProcessedInfoNc C s* | j d k r&t| j t r&tj| j | _ d S )N)features
isinstancer
from_dict)selfr r r
__post_init__K s z PostProcessedInfo.__post_init__)post_processed_info_dictreturnc s0 dd t j| D | f fdd|j D S )Nc S s h | ]
}|j qS r )name).0fr r r
R s z.PostProcessedInfo.from_dict..c s i | ]\}}| kr||qS r r )r, kv) field_namesr r
S s z/PostProcessedInfo.from_dict..) dataclassesfieldsitems)clsr) r )r1 r r& P s zPostProcessedInfo.from_dict) r r r r$ r r
Zresources_checksumsdictr( classmethodr& r r r r r# F s
r# c @ sz e Zd ZU dZeedZeeedZeeedZeeedZ e dZ
e e
dZ
e e
dZe e dZe ee dZe e dZe e dZe eeef dZe e dZe e dZ e e dZ e e dZ e e dZ e e dd Z!dd Z"d d d
Z#d d d Z$d
d Z%e&ed dddZ'e&ed dddZ(e&ed dddZ)d!d dddZ*d d d d Z+dS )" DatasetInfoa
Information about a dataset.
`DatasetInfo` documents datasets, including its name, version, and features.
See the constructor arguments and properties for a full list.
Note: Not all fields are known on construction and may be updated later.
Attributes:
description (str): A description of the dataset.
citation (str): A BibTeX citation of the dataset.
homepage (str): A URL to the official homepage for the dataset.
license (str): The dataset's license. It can be the name of the license or a paragraph containing the terms of the license.
features (Features, optional): The features used to specify the dataset's column types.
post_processed (PostProcessedInfo, optional): Information regarding the resources of a possible post-processing of a dataset. For example, it can contain the information of an index.
supervised_keys (SupervisedKeysData, optional): Specifies the input feature and the label for supervised learning if applicable for the dataset (legacy from TFDS).
builder_name (str, optional): The name of the :class:`GeneratorBasedBuilder` subclass used to create the dataset. Usually matched to the corresponding script name. It is also the snake_case version of the dataset builder class name.
config_name (str, optional): The name of the configuration derived from :class:`BuilderConfig`
version (str or Version, optional): The version of the dataset.
splits (dict, optional): The mapping between split name and metadata.
download_checksums (dict, optional): The mapping between the URL to download the dataset's checksums and corresponding metadata.
download_size (int, optional): The size of the files to download to generate the dataset, in bytes.
post_processing_size (int, optional): Size of the dataset in bytes after post-processing, if any.
dataset_size (int, optional): The combined size in bytes of the Arrow tables for all splits.
size_in_bytes (int, optional): The combined size in bytes of all files associated with the dataset (downloaded files + Arrow files).
task_templates (List[TaskTemplate], optional): The task templates to prepare the dataset for during training and evaluation. Each template casts the dataset's :class:`Features` to standardized column names and types as detailed in :py:mod:`datasets.tasks`.
**config_kwargs (additional keyword arguments): Keyword arguments to be passed to the :class:`BuilderConfig` and used in the :class:`DatasetBuilder`.
)default_factoryNc s j d k r&t j t r&tj j _ jd k rLt jt rLtj j _ jd k rt jt rt jtr~t j _ntj j _ j d k rt j t
rt
j j _ j d k rt j t
rt j ttfrt
j _ nt
f j _ jd k rpt jttfr6dd jD }dd |D _n:t jtrP jg _n t j}|d k rj|gng _ jd k rt j _ j d k r fdd jD _d S )Nc S s" g | ]}t |tr|nt|qS r )r% r
r )r, templater r r
s z-DatasetInfo.__post_init__..c S s g | ]}|d k r|qS )Nr )r, r; r r r r< s c s g | ]}|j jqS r )Zalign_with_featuresr$ )r, r; )r' r r r< s )r$ r% r
r& post_processedr# versionr r splitsr Zfrom_split_dictsupervised_keysr tuplelisttask_templatesr
r )r' Z templatesr; r )r' r r( s6
zDatasetInfo.__post_init__c C s t jj|tjS )N)ospathjoinr LICENSE_FILENAME)r' dataset_info_dirr r r
_license_path s zDatasetInfo._license_pathFc C sf t tjj|tjd}| j||d W dQ R X | jrbt tjj|tjd}| j | W dQ R X dS )a Write `DatasetInfo` and license (if present) as JSON files to `dataset_info_dir`.
Args:
dataset_info_dir (str): Destination directory.
pretty_print (bool, default ``False``): If True, the JSON will be pretty-printed with the indent level of 4.
Example:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset("rotten_tomatoes", split="validation")
>>> ds.info.write_to_directory("/path/to/directory/")
```
wb) pretty_printN)
openrD rE rF r DATASET_INFO_FILENAME
_dump_infolicenserG
_dump_license)r' rH rK r- r r r write_to_directory s
z DatasetInfo.write_to_directoryc C s* |j tjt| |rdnddjd dS )zQDump info in `file` file-like object open in bytes mode (to support remote files) N)indentzutf-8)writejsondumpsr encode)r' filerK r r r rN s zDatasetInfo._dump_infoc C s |j | jjd dS )zTDump license in `file` file-like object open in bytes mode (to support remote files)zutf-8N)rT rO rW )r' rX r r r rP s zDatasetInfo._dump_license)
dataset_infosc
C s dd |D }dj tdd |D j }dj tdd |D j }dj tdd |D j }dj tdd |D j }d }d }d }d d |D } t| d
krtt| d j| d
d }nt| rtt| d }|r|nd }| |||||||d S )
Nc S s g | ]}|d k r|j qS )N)copy)r, dset_infor r r r< s z*DatasetInfo.from_merge..z
c s s | ] }|j V qd S )N) description)r, infor r r s z)DatasetInfo.from_merge..c s s | ] }|j V qd S )N)citation)r, r] r r r r^ s c s s | ] }|j V qd S )N)homepage)r, r] r r r r^ s c s s | ] }|j V qd S )N)rO )r, r] r r r r^ s c S s g | ]}|j d k r|j qS )N)rC )r, r] r r r r< s r r )r\ r_ r` rO r$ r@ rC )rF r striplenrB set intersection)
r6 rY r\ r_ r` rO r$ r@ rC Zall_task_templatesr r r
from_merge s, zDatasetInfo.from_merge)rH r* c C sT t jd| |s tdttjj|tjdd}t j
|}W dQ R X | j |S )aX Create DatasetInfo from the JSON file in `dataset_info_dir`.
This function updates all the dynamically generated fields (num_examples,
hash, time of creation,...) of the DatasetInfo.
This will overwrite all previous metadata.
Args:
dataset_info_dir (`str`): The directory containing the metadata file. This
should be the root directory of a specific dataset version.
Example:
```py
>>> from datasets import DatasetInfo
>>> ds_info = DatasetInfo.from_directory("/path/to/directory/")
```
zLoading Dataset info from zECalling DatasetInfo.from_directory() with undefined dataset_info_dir.zutf-8)encodingN) loggerr]
ValueErrorrL rD rE rF r rM rU loadr& )r6 rH r- dataset_info_dictr r r from_directory s zDatasetInfo.from_directory)rj r* c s0 dd t j| D | f fdd|j D S )Nc S s h | ]
}|j qS r )r+ )r, r- r r r r. s z(DatasetInfo.from_dict..c s i | ]\}}| kr||qS r r )r, r/ r0 )r1 r r r2 s z)DatasetInfo.from_dict..)r3 r4 r5 )r6 rj r )r1 r r&
s zDatasetInfo.from_dictT)other_dataset_infoc s* | j }|jf fdd|j j D d S )Nc s* i | ]"\}}|d k s rt j||qS )N)rZ deepcopy)r, r/ r0 ) ignore_noner r r2 s z&DatasetInfo.update..)__dict__updater5 )r' rl rn Z self_dictr )rn r rp s
zDatasetInfo.update)r* c C s | j f dd | jj D S )Nc S s i | ]\}}t j||qS r )rZ rm )r, r/ r0 r r r r2 s z$DatasetInfo.copy..) __class__ro r5 )r' r r r rZ s zDatasetInfo.copy)F)F)T),r r r r! r r r\ r_ r` rO r$ r r
r= r# r@ r rC r r
Z builder_name config_namer> r r r? r7 Zdownload_checksums
download_sizeintZpost_processing_sizeZ dataset_sizeZ
size_in_bytesr( rI rQ rN rP r8 re rk r& rp rZ r r r r r9 V s>
)
r9 c @ s" e Zd ZdddZedd ZdS )DatasetInfosDictFc C s i }t jj|tj}t jj|rD| rDtjd| d | j|}ntjd| |j | t
|ddd,}t j dd |j
D ||rd nd d
W d Q R X d S ) Nz Dataset Infos already exists in z . Completing it with new infos.z Writing new Dataset Infos in wzutf-8)rf c S s i | ]\}}t ||qS r )r )r, rr r[ r r r r2 , s z7DatasetInfosDict.write_to_directory..rR )rS )rD rE rF r DATASETDICT_INFOS_FILENAMEexistsrg r] rk rp rL rU dumpr5 )r' dataset_infos_dir overwriterK Ztotal_dataset_infosZdataset_infos_pathr- r r r rQ ! s
z#DatasetInfosDict.write_to_directoryc
C sV t jd| ttjj|tjdd }dd tj |j
D }W d Q R X | f |S )NzLoading Dataset Infos from zutf-8)rf c S s i | ]\}}t j||qS r )r9 r& )r, rr rj r r r r2 5 s z3DatasetInfosDict.from_directory..) rg r] rL rD rE rF r rw rU ri r5 )r6 rz r- Zdataset_infos_dictr r r rk 1 s
z DatasetInfosDict.from_directoryN)FF)r r r rQ r8 rk r r r r ru s
ru c @ s e Zd ZU dZeeee edZ
e
e edZ e e edZ e e e
dZee e e
dZee dZedZee dZee dZee dZee dd ZdddZed d d
d Zee d d d
dZ dS )
MetricInfoa Information about a metric.
`MetricInfo` documents a metric, including its name, version, and features.
See the constructor arguments and properties for a full list.
Note: Not all fields are known on construction and may be updated later.
)r: FNc C sH | j d k rDx8| jj D ]*\}}t|tstd| d|jj qW d S )NzSWhen using 'numpy' format, all features should be a `datasets.Value` feature. Here z is an instance of )formatr$ r5 r% r rh rq r )r' r r r r r r( W s
zMetricInfo.__post_init__c C s~ t tjj|tjddd"}tjt| ||r0dndd W dQ R X | j rzt tjj|tj
ddd}|j | j W dQ R X dS )a Write `MetricInfo` as JSON to `metric_info_dir`.
Also save the license separately in LICENCE.
If `pretty_print` is True, the JSON will be pretty-printed with the indent level of 4.
Example:
```py
>>> from datasets import load_metric
>>> metric = load_metric("accuracy")
>>> metric.info.write_to_directory("/path/to/directory/")
```
rv zutf-8)rf rR N)rS ) rL rD rE rF r METRIC_INFO_FILENAMErU ry r rO rG rT )r' metric_info_dirrK r- r r r rQ ` s
& z MetricInfo.write_to_directory)r* c C sT t jd| |s tdttjj|tjdd}t j
|}W dQ R X | j |S )a Create MetricInfo from the JSON file in `metric_info_dir`.
Args:
metric_info_dir: `str` The directory containing the metadata file. This
should be the root directory of a specific dataset version.
Example:
```py
>>> from datasets import MetricInfo
>>> metric_info = MetricInfo.from_directory("/path/to/directory/")
```
zLoading Metric info from zCCalling MetricInfo.from_directory() with undefined metric_info_dir.zutf-8)rf N) rg r] rh rL rD rE rF r r~ rU ri r& )r6 r r- metric_info_dictr r r rk t s zMetricInfo.from_directory)r r* c s0 dd t j| D | f fdd|j D S )Nc S s h | ]
}|j qS r )r+ )r, r- r r r r. s z'MetricInfo.from_dict..c s i | ]\}}| kr||qS r r )r, r/ r0 )r1 r r r2 s z(MetricInfo.from_dict..)r3 r4 r5 )r6 r r )r1 r r& s zMetricInfo.from_dict)F) r r r r! r r\ r_ r
r$ r Zinputs_descriptionr` rO rB Z
codebase_urlsr Zreference_urlsZ
streamableboolr} r Z metric_namerr Z
experiment_idr( rQ r8 rk r7 r& r r r r r| < s(
r| ))r! rZ r3 rU rD r r typingr r r r r r r$ r
r r? r tasksr
r utilsr Z
utils.loggingr Zutils.py_utilsr r r rg r r Exceptionr r" r# r9 r ru r| r r r r s. K