Name	Size	Mode	Actions
arrow_dataset.cpython-36.pyc	189850	0644	editdlrm
arrow_reader.cpython-36.pyc	22483	0644	editdlrm
arrow_writer.cpython-36.pyc	22110	0644	editdlrm
builder.cpython-36.pyc	52779	0644	editdlrm
combine.cpython-36.pyc	5508	0644	editdlrm
config.cpython-36.pyc	5181	0644	editdlrm
dataset_dict.cpython-36.pyc	83319	0644	editdlrm
data_files.cpython-36.pyc	30153	0644	editdlrm
fingerprint.cpython-36.pyc	18979	0644	editdlrm
info.cpython-36.pyc	17359	0644	editdlrm
inspect.cpython-36.pyc	18583	0644	editdlrm
iterable_dataset.cpython-36.pyc	62932	0644	editdlrm
keyhash.cpython-36.pyc	3341	0644	editdlrm
load.cpython-36.pyc	62922	0644	editdlrm
metric.cpython-36.pyc	23217	0644	editdlrm
naming.cpython-36.pyc	2571	0644	editdlrm
search.cpython-36.pyc	31339	0644	editdlrm
splits.cpython-36.pyc	22522	0644	editdlrm
streaming.cpython-36.pyc	4119	0644	editdlrm
table.cpython-36.pyc	79612	0644	editdlrm
__init__.cpython-36.pyc	2282	0644	editdlrm

3 <%Eg��@s�dd
lm
Z
mZmZ
ddlmZmZmZ
ddlm Z
ddl mZmZm Z
ddlmZ
ddlmZ
eje�
Zedd d �Zde
eee
eeeee eeed�d d�Zde
eee eeed�dd�ZdS)�)�List�Optional�TypeVar�
)�Dataset�_concatenate_map_style_datasets�_interleave_map_style_datasets)
�DatasetInfo)�IterableDataset�_concatenate_iterable_datasets�_interleave_iterable_datasets)
� NamedSplit)
�logging�DatasetTyperr N)�datasets� probabilities�seed�info�split�returnc Cs�d
dlm
}
d
dlm}
|s$td�
�
t|d|�}t|d|�}||As^tdt|d�
���
�
xV|d
d�D]F} |r�t| |�s�|rlt| |�rltdt|d�
�d t| �
�d ��
�
qlW|r�t||
|||d�St||
|||d�SdS)u� Interleave several datasets (sources) into a single dataset. The new dataset is constructed by alternating between the sources to get the examples. You can use this function on a list of :class:`Dataset` objects, or on a list of :class:`IterableDataset` objects. If ``probabilities`` is ``None`` (default) the new dataset is constructed by cycling between each source to get the examples. If ``probabilities`` is not ``None``, the new dataset is constructed by getting examples from a random source at a time according to the provided probabilities. The resulting dataset ends when one of the source datasets runs out of examples. Args: datasets (:obj:`List[Dataset]` or :obj:`List[IterableDataset]`): list of datasets to interleave probabilities (:obj:`List[float]`, optional, default None): If specified, the new dataset is constructued by sampling examples from one source at a time according to these probabilities. seed (:obj:`int`, optional, default None): The random seed used to choose a source for each example. Returns: :class:`Dataset` or :class:`IterableDataset`: Return type depends on the input `datasets` parameter. `Dataset` if the input is a list of `Dataset`, `IterableDataset` if the input is a list of `IterableDataset`. Example:: For regular datasets (map-style): >>> from datasets import Dataset, interleave_datasets >>> d1 = Dataset.from_dict({"a": [0, 1, 2]}) >>> d2 = Dataset.from_dict({"a": [10, 11, 12]}) >>> d3 = Dataset.from_dict({"a": [20, 21, 22]}) >>> dataset = interleave_datasets([d1, d2, d3]) >>> dataset["a"] [0, 10, 20, 1, 11, 21, 2, 12, 22] >>> dataset = interleave_datasets([d1, d2, d3], probabilities=[0.7, 0.2, 0.1], seed=42) >>> dataset["a"] [10, 0, 11, 1, 2, 20, 12] For datasets in streaming mode (iterable): >>> from datasets import load_dataset, interleave_datasets >>> d1 = load_dataset("oscar", "unshuffled_deduplicated_en", split="train", streaming=True) >>> d2 = load_dataset("oscar", "unshuffled_deduplicated_fr", split="train", streaming=True) >>> dataset = interleave_datasets([d1, d2]) >>> iterator = iter(dataset) >>> next(iterator) {'text': 'Mtendere Village was inspired by the vision... >>> next(iterator) {'text': "Média de débat d'idées, de culture... r)
r)
r z/Unable to interleave an empty list of datasets.r
z`Expected a list of Dataset objects or a list of IterableDataset objects, but first element is a NzUnable to interleave a z with a zJ. Expected a list of Dataset objects or a list of IterableDataset objects.)rr) � arrow_datasetr�iterable_datasetr � ValueError� isinstance�typerr) rrrrrrr �iterable� map_style�dataset�r�:/usr/local/lib/python3.6/site-packages/datasets/combine.py�interleave_datasetss 8









$
r )�dsetsrr�axiscCs�|std
�
�
t
|dt�}t
|dt�}||AsFtdt|d�
���
�
xV|dd�D]F}|rht
|t�sx|rTt
|t�rTtdt|d�
�dt|�
�d��
�
qTW|r�t||
||d �St||
||d �SdS) a� Converts a list of :class:`Dataset` with the same schema into a single :class:`Dataset`. Args: dsets (:obj:`List[datasets.Dataset]`): List of Datasets to concatenate. info (:class:`DatasetInfo`, optional): Dataset information, like description, citation, etc. split (:class:`NamedSplit`, optional): Name of the dataset split. axis (``{0, 1}``, default ``0``, meaning over rows): Axis to concatenate over, where ``0`` means over rows (vertically) and ``1`` means over columns (horizontally). *New in version 1.6.0* Example: ```py >>> ds3 = concatenate_datasets([ds1, ds2]) ``` z0Unable to concatenate an empty list of datasets.r
z`Expected a list of Dataset objects or a list of IterableDataset objects, but first element is a rNzUnable to concatenate a z with a zJ. Expected a list of Dataset objects or a list of IterableDataset objects.)rrr")rrr rrrr)r!rrr"rrrrrr�concatenate_datasets^s








$
r#)NNNN)NNr
)�typingrrrrrrrrr rr rr�splitsr �utilsr� get_logger�__name__�loggerr�float�intr r#rrrr�
s



 


$
J