Name	Size	Mode	Actions
arrow_dataset.cpython-36.pyc	189850	0644	editdlrm
arrow_reader.cpython-36.pyc	22483	0644	editdlrm
arrow_writer.cpython-36.pyc	22110	0644	editdlrm
builder.cpython-36.pyc	52779	0644	editdlrm
combine.cpython-36.pyc	5508	0644	editdlrm
config.cpython-36.pyc	5181	0644	editdlrm
dataset_dict.cpython-36.pyc	83319	0644	editdlrm
data_files.cpython-36.pyc	30153	0644	editdlrm
fingerprint.cpython-36.pyc	18979	0644	editdlrm
info.cpython-36.pyc	17359	0644	editdlrm
inspect.cpython-36.pyc	18583	0644	editdlrm
iterable_dataset.cpython-36.pyc	62932	0644	editdlrm
keyhash.cpython-36.pyc	3341	0644	editdlrm
load.cpython-36.pyc	62922	0644	editdlrm
metric.cpython-36.pyc	23217	0644	editdlrm
naming.cpython-36.pyc	2571	0644	editdlrm
search.cpython-36.pyc	31339	0644	editdlrm
splits.cpython-36.pyc	22522	0644	editdlrm
streaming.cpython-36.pyc	4119	0644	editdlrm
table.cpython-36.pyc	79612	0644	editdlrm
__init__.cpython-36.pyc	2282	0644	editdlrm

3 <%Eg[V�@s\
dZd
dl
Z
d
dlZd
dlZd
dlmZmZ
d
dlmZm Z m Z mZ
ddlm Z mZ
ddlmZ
ddlmZ
eGd d �d ��
ZeGdd�d��
ZGd d�de
jd�ZGdd�de�ZGdd�ded�ZeZGdd�de�ZGdd�de�ZGdd�de�ZGdd�de�ZGdd�d�Zej ddd g�Z!Gd!d"�d"�Z"Gd#d$�d$e#�Z$eGd%d&�d&��
Z%dS)'zSplits related API.�N)� dataclass�field)�Dict�List�Optional�Union�
)�FileInstructions�make_file_instructions)
� _split_re)
�NonMutableDictc@s>eZ
dZUd
ZedZedZedZe ee dd��
ZdS)� SplitInfo�r
Nc
Cst|j
|g
t|j�
d
�}
|
jS)z/Returns the list of dict(filename, take, skip).)�name�split_infos�instruction)r �dataset_name�strr�file_instructions)�self�instructions�r�9/usr/local/lib/python3.6/site-packages/datasets/splits.pyr%s 


zSplitInfo.file_instructions)�__name__� __module__�__qualname__rr� num_bytes�int�num_examplesrr�propertyrrrrrr s 


r c@s.eZ
dZUd
Zeedd��
Zedd��
ZdS)�SubSplitInfoz�Wrapper around a sub split info. This class expose info on the subsplit: ``` ds, info = datasets.load_dataset(..., split='train[75%:]', with_info=True) info.splits['train[75%:]'].num_examples ``` c


Cs|jj
S)
z.Returns the number of example in the subsplit.)rr)
rrrrr=szSubSplitInfo.num_examplesc


Cs|jj
S)
z/Returns the list of dict(filename, take, skip).)rr)
rrrrrBszSubSplitInfo.file_instructionsN) rrr�__doc__r rrrrrrrrr 1s r c@s@eZ
dZd
Zejdd��
Zdd�Zdd�Zdd �Z d dd�
Z d S)� SplitBasea�Abstract base class for Split compositionality. See the [guide on splits](/docs/datasets/loading#slice-splits) for more information. There are three parts to the composition: 1) The splits are composed (defined, merged, split,...) together before calling the `.as_dataset()` function. This is done with the `__add__`, `__getitem__`, which return a tree of `SplitBase` (whose leaf are the `NamedSplit` objects) ``` split = datasets.Split.TRAIN + datasets.Split.TEST.subsplit(datasets.percent[:50]) ``` 2) The `SplitBase` is forwarded to the `.as_dataset()` function to be resolved into actual read instruction. This is done by the `.get_read_instruction()` method which takes the real dataset splits (name, number of shards,...) and parse the tree to return a `SplitReadInstruction()` object ``` read_instruction = split.get_read_instruction(self.info.splits) ``` 3) The `SplitReadInstruction` is then used in the `tf.data.Dataset` pipeline to define which files to read and how to skip examples within file. cCstd
�
�
dS)z�Parse the descriptor tree and compile all read instructions together. Args: split_dict: `dict`, The `dict[split_name, SplitInfo]` of the dataset Returns: split_read_instruction: `SplitReadInstruction` zAbstract methodN)
�NotImplementedError)r� split_dictrrr�get_read_instructionjs zSplitBase.get_read_instructioncCst|
t
tf�rd
Std�
�
dS)z*Equality: datasets.Split.TRAIN == 'train'.Fz6Equality is not implemented between merged/sub splits.N)� isinstance� NamedSplitrr#)r�otherrrr�__eq__vs

zSplitBase.__eq__cCs|j|
�
S)
z+InEquality: datasets.Split.TRAIN != 'test'.)
r))rr(rrr�__ne__|szSplitBase.__ne__cCs t||
�S)
z4Merging: datasets.Split.TRAIN + datasets.Split.TEST.)
�_SplitMerged)rr(rrr�__add__�szSplitBase.__add__Nc s�
td
d�|
|||fD�
�
dkr&t
d�
�
t|
t�r6|
}nt|
t�rF|
}nt|
t�rT|
}|p^|p^|spt
d|
�d��
�
dd�}|r�d |ko�d k
n
s�t
d|���
�
d |�
�
f
dd �t|�
D�
}t|djd �|d<||�

t�f
dd�|D�
�
S|�
rt �|�S|�
r�t|�
��f
dd �|D�
}d }d }g}x*|D]"} || 7}|j t||��

|}�
q4Wt|djd �|d<||�

t�f
dd�|D�
�
St
d�
�
dS)a6Divides this split into subsplits. There are 3 ways to define subsplits, which correspond to the 3 arguments `k` (get `k` even subsplits), `percent` (get a slice of the dataset with `datasets.percent`), and `weighted` (get subsplits with proportions specified by `weighted`). Example:: ``` # 50% train, 50% test train, test = split.subsplit(k=2) # 50% train, 25% test, 25% validation train, test, validation = split.subsplit(weighted=[2, 1, 1]) # Extract last 20% subsplit = split.subsplit(datasets.percent[-20:]) ``` Warning: k and weighted will be converted into percent which mean that values below the percent will be rounded up or down. The final split may be bigger to deal with remainders. For instance: ``` train, test, valid = split.subsplit(k=3) # 33%, 33%, 34% s1, s2, s3, s4 = split.subsplit(weighted=[2, 2, 1, 1]) # 33%, 33%, 16%, 18% ``` Args: arg: If no kwargs are given, `arg` will be interpreted as one of `k`, `percent`, or `weighted` depending on the type. For example: ``` split.subsplit(10) # Equivalent to split.subsplit(k=10) split.subsplit(datasets.percent[:-20]) # percent=datasets.percent[:-20] split.subsplit([1, 1, 2]) # weighted=[1, 1, 2] ``` k: `int` If set, subdivide the split into `k` equal parts. percent: `datasets.percent slice`, return a single subsplit corresponding to a slice of the original split. For example: `split.subsplit(datasets.percent[-20:]) # Last 20% of the dataset`. weighted: `list[int]`, return a list of subsplits whose proportions match the normalized sum of the list. For example: `split.subsplit(weighted=[1, 1, 2]) # 25%, 25%, 50%`. Returns: A subsplit or list of subsplits extracted from this split object. c
ss|]}
t|
�
V
qdS)
N)
�bool)�.0�
xrrr� �sz%SplitBase.subsplit..rz,Only one argument of subsplit should be set.zInvalid split argument zg. Only list, slice and int supported. One of k, weighted or percent should be set to a non empty value.c

Ss(td
d�|D�
g�t
td�
�
ks$t�
dS)Nc
ss |]}
tt
|
jd�
��
V
qd
S)�dN)�list�range�indices)r.�
srrrr0�szESplitBase.subsplit..assert_slices_coverage..r1)�sumr2r3�AssertionError)
�slicesrrr�assert_slices_coverage�sz2SplitBase.subsplit..assert_slices_coverager
r1z,Subsplit k should be between 0 and 100, got c
s"g|]}
t|
�|
d���qS)
r)
�slice)r.�
i)
�shiftrr� �sz&SplitBase.subsplit..c
3s|]}
t�|
�V
qdS)
N)
� _SubSplit)r.r5)
rrrr0�sc
sg|]}
d|
��qS)
r1r)r.r/)
�totalrrr=�sc
3s|]}
t�|
�V
qdS)
N)
r>)r.r5)
rrrr0�szCould not determine the splitN�����r@r@r@)r6� ValueErrorr&rr:r2r3�start�tupler>�append) r�arg�
k�percentZweightedr9r8rB�stop�
vr)rr<r?r�subsplit�sH2























zSplitBase.subsplit)NNNN)rrrr!�abc�abstractmethodr%r)r*r,rJrrrrr"Hsr")
� metaclassc@seZ
dZd
d�ZdS)�PercentSliceMetacCst|
t
�std
|
���
�
|
S)Nz7datasets.percent should only be called with slice, not )r&r:rA)�cls�slice_valuerrr�__getitem__�s


zPercentSliceMeta.__getitem__N)rrrrQrrrrrN�s
rNc
@seZ
dZd
ZdS)�PercentSlicez�Syntactic sugar for defining slice subsplits: `datasets.percent[75:-5]`. See the [guide on splits](/docs/datasets/loading#slice-splits) for more information. N)rrrr!rrrrrR�srRc@s(eZ
dZd
Zdd�Zdd�Zdd�ZdS) r+z0Represent two split descriptors merged together.cCs|
|_||_
dS)
N)�_split1�_split2)rZsplit1Zsplit2rrr�__init__
s

z_SplitMerged.__init__cCs |jj
|
�
}|jj
|
�
}||S)
N)rSr%rT)rr$Zread_instruction1Zread_instruction2rrrr%
s


z!_SplitMerged.get_read_instructionc

Csd
t|j
�
�dt|j�
�d�S)N�
(z + �
))�reprrSrT)
rrrr�__repr__
s
z_SplitMerged.__repr__N)rrrr!rUr%rYrrrrr+
s
r+c@s(eZ
dZd
Zdd�Zdd�Zdd�ZdS) r>z,Represent a sub split of a split descriptor.cCs|
|_||_
dS)
N)�_split�_slice_value)r�splitrPrrrrU
s

z_SubSplit.__init__cCs|jj
|
�
|jS)
N)rZr%r[)rr$rrrr%
s
z_SubSplit.get_read_instructionc
Csjd
}
|jj
dk r|
d7}
|
j|jjdkr,dn|jj|jjdkrBdn|jj|jj
d�}
t|j�
�d|
�d�S)Nz{start}:{stop}z:{step}r)rBrH�stepz(datasets.percent[z]))r[r]�formatrBrHrXrZ)rZ slice_strrrrrY#
s






z_SubSplit.__repr__N)rrrr!rUr%rYrrrrr>
s
r>c@sHeZ
dZd
Zdd�Zdd�Zdd�Zdd �Zd d�Zdd �Z dd�Z dS)r'agDescriptor corresponding to a named split (train, test, ...). Example:: Each descriptor can be composed with other using addition or slice. Ex:: split = datasets.Split.TRAIN.subsplit(datasets.percent[0:25]) + datasets.Split.TEST The resulting split will correspond to 25% of the train split merged with 100% of the test split. Warning: A split cannot be added twice, so the following will fail:: split = ( datasets.Split.TRAIN.subsplit(datasets.percent[:25]) + datasets.Split.TRAIN.subsplit(datasets.percent[75:]) ) # Error split = datasets.Split.TEST + datasets.Split.ALL # Error Warning: The slices can be applied only one time. So the following are valid:: split = ( datasets.Split.TRAIN.subsplit(datasets.percent[:25]) + datasets.Split.TEST.subsplit(datasets.percent[:50]) ) split = (datasets.Split.TRAIN + datasets.Split.TEST).subsplit(datasets.percent[:50]) But not:: train = datasets.Split.TRAIN test = datasets.Split.TEST split = train.subsplit(datasets.percent[:25]).subsplit(datasets.percent[:25]) split = (train.subsplit(datasets.percent[:25]) + test).subsplit(datasets.percent[:50]) cCsN|
|_d
d�|
j
d�
D�
}x.|D]&}tjt|�s tdt�d|�d��
�
q WdS)Nc
Ssg|]}
|
jd�
d
�qS)�
[r
)
r\)r.�split_instructionrrrr=W
sz'NamedSplit.__init__..�
+zSplit name should match 'z' but got 'z'.)�_namer\�re�matchrrA)rrZsplit_names_from_instructionZ split_namerrrrUU
s




zNamedSplit.__init__c


Cs|jS)
N)
rb)
rrrr�__str__\
s
zNamedSplit.__str__c

Csd
|j�d�S)NzNamedSplit(rW)
rb)
rrrrrY_
s
zNamedSplit.__repr__cCsPt|
t
�r|j|
jkSt|
t�r$d
St|
t�r8|j|
kStd|�d|
���
�
dS)z*Equality: datasets.Split.TRAIN == 'train'.Fz%Equality not supported between split z and N)r&r'rbr"rrA)rr(rrrr)b
s




zNamedSplit.__eq__cCs|j|
jkS)
N)
rb)rr(rrr�__lt__m
s
zNamedSplit.__lt__c

Cs t|j
�
S)
N)�hashrb)
rrrr�__hash__p
s
zNamedSplit.__hash__cCst|
|j
�
S)
N)�SplitReadInstructionrb)rr$rrrr%s
s
zNamedSplit.get_read_instructionN)rrrr!rUrerYr)rfrhr%rrrrr'/
s$r'cs0eZ
dZd
Z�f
dd�Zdd�Zdd�Z�ZS)� NamedSplitAllz?Split corresponding to the union of all defined dataset splits.c

st�j
d
�

dS)N�all)�superrU)
r)
� __class__rrrUz
s
zNamedSplitAll.__init__c


Csd
S)NzNamedSplitAll()r)
rrrrrY}
s
zNamedSplitAll.__repr__cCsd
d�|
j�D�
}t
|t��S)Nc
Ssg|]}
t|
�
�qSr)
ri)r.r5rrrr=�
sz6NamedSplitAll.get_read_instruction..)�valuesr6ri)rr$Zread_instructionsrrrr%�
s
z"NamedSplitAll.get_read_instruction)rrrr!rUrYr%� __classcell__rr)
rmrrjw
s
rjc@s6eZ
dZd
Zed�
Zed�
Zed�
Ze�Z dd�Z dS)�Splita#`Enum` for dataset splits. Datasets are typically split into different subsets to be used at various stages of training and evaluation. - `TRAIN`: the training data. - `VALIDATION`: the validation data. If present, this is typically used as evaluation data while iterating on a model (e.g. changing hyperparameters, model architecture, etc.). - `TEST`: the testing data. This is the data to report metrics on. Typically you do not want to use this during model iteration as you may overfit to it. - `ALL`: the union of all defined dataset splits. Note: All splits, including compositions inherit from `datasets.SplitBase` See the :doc:`guide on splits ` for more information. Example: ```py >>> datasets.SplitGenerator( ... name=datasets.Split.TRAIN, ... gen_kwargs={"split_key": "train", "files": dl_manager.download_and extract(url)}, ... ), ... datasets.SplitGenerator( ... name=datasets.Split.VALIDATION, ... gen_kwargs={"split_key": "validation", "files": dl_manager.download_and extract(url)}, ... ), ... datasets.SplitGenerator( ... name=datasets.Split.TEST, ... gen_kwargs={"split_key": "test", "files": dl_manager.download_and extract(url)}, ... ) ``` Ztrain�testZ validationcCs|
d
krt�St
|
�
S)z9Create a custom split with datasets.Split('custom_name').rk)rjr')rOrrrr�__new__�
sz Split.__new__N)rrrr!r'ZTRAINZTESTZ VALIDATIONrj�ALLrrrrrrrp�
s#


rp�SlicedSplitInfo� split_inforPc@s:eZ
dZd
Zd dd�
Zdd�Zdd�Zd d �Zdd�ZdS)riaObject containing the reading instruction for the dataset. Similarly to `SplitDescriptor` nodes, this object can be composed with itself, but the resolution happens instantaneously, instead of keeping track of the tree, such as all instructions are compiled and flattened in a single SplitReadInstruction object containing the list of files and slice to use. Once resolved, the instructions can be accessed with: ``` read_instructions.get_list_sliced_split_info() # List of splits to use ``` NcCs&td
d�
|_
|
r"|jt|
dd��

dS)Nz?Overlap between splits. Split {key} has been added with itself.)
� error_msg)rurP)r�_splits�addrt)rrurrrrU�
s

zSplitReadInstruction.__init__cCs|
|j|
j
j<d
S)z,Add a SlicedSplitInfo the read instructions.N)rwrur)rZsliced_splitrrrrx�
szSplitReadInstruction.addcCs&t�}|j
j|j
�

|j
j|
j
�

|S)
zMerging split together.)rirw�update)rr(r`rrrr,�
s


zSplitReadInstruction.__add__cCs\t�}xP|j
j�D]B}|jd
k r4td|jj�d��
�
|j�}|
|d<|jt f|�
�

qW|S)zSub-splits.NzTrying to slice Split z which has already been slicedrP) rirwrnrPrArur�_asdictrxrt)rrPr`rIrrrrQ�
s






z SplitReadInstruction.__getitem__c

Cstt
|jj�d
d�d��
S)Nc


Ss|jj
S)
N)rur)
r/rrr��
szASplitReadInstruction.get_list_sliced_split_info..)
�key)r2�sortedrwrn)
rrrr�get_list_sliced_split_info�
s
z/SplitReadInstruction.get_list_sliced_split_info)
N) rrrr!rUrxr,rQr~rrrrri�
s  rics�eZ
dZd
Zdd�
�f
dd� Zeeefd�
�f
dd�Zeeefe d ��f
d d�Z e d�
�f
d d�Zedd��
Z edeeefeed�dd��
Zdd�Zdd�Z�ZS)� SplitDictzSplit info object.N)
rc

st�j
||�

|
|_dS)
N)rlrUr)rr�args�kwargs)
rmrrrU�
s

zSplitDict.__init__)
r|cs<t|
�
|krt
�jt|
�
�
St|j|j�|
d
�}t|�
SdS)N)rrr)rrlrQr rrnr )rr|r)
rmrrrQ�
s



zSplitDict.__getitem__)r|�valuecsL|
|jkr"t
d
|
�d|j�d��
�
|
|kr:t
d|
�d��
�
t�j|
|�
dS)Nz!Cannot add elem. (key mismatch: 'z' != 'z')zSplit z already present)rrArl�__setitem__)rr|r�)
rmrrr� s




zSplitDict.__setitem__)
rucs8|
j|krt
d
|
j�d��
�
|j|
_t�j|
j|
�
dS)zAdd the split info.zSplit z already presentN)rrArrlr�)rru)
rmrrrxs


z SplitDict.addc

Cstd
d�|j
�D�
�
S)z$Return the total number of examples.c
ss|]}
|
jV
qdS)
N)
r)r.r5rrrr0sz/SplitDict.total_num_examples..)r6rn)
rrrr�total_num_examplesszSplitDict.total_num_examples)rrcCslt|
t
�rt|
j��
}
|d
kr2|
r.|
ddnd
}||d�
}x*|
D]"}t|t
�rZtf|�
}|j|�

qBW|S)zIReturns a new SplitDict initialized from a Dict or List of `split_infos`.Nr
r)
r)r&�dictr2rnr rx)rOrrr$rurrr�from_split_dicts

 


zSplitDict.from_split_dictc

Cstd
d�|j
�D�
dd�d�S)z0Returns a list of SplitInfo protos that we have.c
ss|] }
|
V
qdS)
Nr)r.r5rrrr02sz*SplitDict.to_split_dict..c


Ss|jS)
N)
r)
r5rrrr{2sz)SplitDict.to_split_dict..)
r|)r}rn)
rrrr� to_split_dict/szSplitDict.to_split_dictc

Cstj
|j�|j�S)
N)rr�r�r)
rrrr�copy4s
zSplitDict.copy)
N)rrrr!rUrr"rrQr r�rxrr��classmethodrrrr�r�r�rorr)
rmrr�
s
 
rc@s:eZ
dZUd
Zeeed�
Ze edd�
Z e dd�ZdS)�SplitGeneratora�Defines the split information for the generator. This should be used as returned value of :meth:`GeneratorBasedBuilder._split_generators`. See :meth:`GeneratorBasedBuilder._split_generators` for more info and example of usage. Args: name (str): Name of the Split for which the generator will create the examples. **gen_kwargs: Keyword arguments to forward to the :meth:`DatasetBuilder._generate_examples` method of the builder. Example: ```py >>> datasets.SplitGenerator( ... name=datasets.Split.TRAIN, ... gen_kwargs={"split_key": "train", "files": dl_manager.download_and_extract(url)}, ... ) ``` )
�default_factoryF)
�initc

Cs(t|j
�
|_
t|j
�

t|j
d
�
|_dS)N)
r)rrr'r ru)
rrrr� __post_init__Us


zSplitGenerator.__post_init__N) rrrr!rrrr�Z gen_kwargsrrur r�rrrrr�8s 

r�)&r!rK�collectionsrcZdataclassesrr�typingrrrrZarrow_readerr r ZnamingrZutils.py_utilsrr r �ABCMetar"�typerNrRrGr+r>r'rjrp� namedtuplertrir�rr�rrrr�s6





,H0

7B