/usr/local/lib64/python3.6/site-packages/torch/utils/data/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/utils/data/__pycache__/dataset.cpython-36.pyc (17051B)
3
Eg: @ sv d dl Z d dlZd dlZd dlmZmZmZmZmZm Z m
Z
m Z m Z m
Z
d dlmZmZ d dlmZ d dlmZ ddlmZmZ e
dd d
Ze
d Zd d
ddgZG dd deee Z G dd dee Z G dd de e edZ G dd de Z G dd de e edf Z G d d d e e Z!G d d d e Z"G d d! d!e e Z#efe e e e$ e
e e e#e d"d#d$Z%dS )% N)
CallableDictGenericIterableIteratorListOptionalSequenceTupleTypeVar)default_generatorrandperm) _accumulate)
_DataPipeMeta ) GeneratorTensorT_coT) covariantTbatchgroupbyZ_dataframes_as_tuplestrace_as_dataframec sD e Zd Z fddZd ddZee d fdd Zd d
Z Z S ) DataChunkc s t j| || _d S )N)super__init__items)selfr ) __class__ D/usr/local/lib64/python3.6/site-packages/torch/utils/data/dataset.pyr " s zDataChunk.__init__ c C s( |d dj dd t| D d }|S )N[z, c s s | ]}t |V qd S )N)str).0ir r r
' s z#DataChunk.as_str..])joiniter)r indentresr r r as_str& s $zDataChunk.as_str)returnc # s xt j D ]
}|V q W d S )N)r __iter__)r r% )r r r r. * s zDataChunk.__iter__c c s x| j D ]
}|V qW d S )N)r )r r% r r r raw_iterator. s zDataChunk.raw_iterator)r! )
__name__
__module__ __qualname__r r, r r r. r/
__classcell__r r )r r r ! s
r c @ sb e Zd ZU dZi Zeeef edddZ ddddd Z
d
d Z e d d
Z
e dddZdS )Dataseta An abstract class representing a :class:`Dataset`.
All datasets that represent a map from keys to data samples should subclass
it. All subclasses should overwrite :meth:`__getitem__`, supporting fetching a
data sample for a given key. Subclasses could also optionally overwrite
:meth:`__len__`, which is expected to return the size of the dataset by many
:class:`~torch.utils.data.Sampler` implementations and the default options
of :class:`~torch.utils.data.DataLoader`.
.. note::
:class:`~torch.utils.data.DataLoader` by default constructs a index
sampler that yields integral indices. To make it work with a map-style
dataset with non-integral indices/keys, a custom sampler must be provided.
)r- c C s t d S )N)NotImplementedError)r indexr r r __getitem__D s zDataset.__getitem__z
Dataset[T_co]zConcatDataset[T_co])otherr- c C s t | |gS )N)
ConcatDataset)r r8 r r r __add__G s zDataset.__add__c C s( |t jkr tjt j| | }|S td S )N)r4 functions functoolspartialAttributeError)r attribute_namefunctionr r r __getattr__N s
zDataset.__getattr__c C s || j |< d S )N)r; )cls
function_namer@ r r r register_functionU s zDataset.register_functionFc s@ | j krtdj fdd}tj|||}|| j < d S )Nz>Unable to add DataPipe function name {} as it is already takenc s< | |f||}t |tr8|s(t |tr8 tkr8|j }|S )N)
isinstancer4 DFIterDataPipeUNTRACABLE_DATAFRAME_PIPESr )rB enable_df_api_tracingZ source_dpargskwargsZ result_pipe)rC r r class_function^ s
z=Dataset.register_datapipe_as_function..class_function)r; Exceptionformatr< r= )rB rC Zcls_to_registerrH rK r@ r )rC r register_datapipe_as_functionY s
z%Dataset.register_datapipe_as_functionN)F)r0 r1 r2 __doc__r; r r# r r r7 r: rA classmethodrD rN r r r r r4 3 s
r4 c sv e Zd ZU dZi Zeeef dZe e e
e dddZ e
e dddZd d
Z fd d Zed
d Z ZS )IterableDataseta An iterable Dataset.
All datasets that represent an iterable of data samples should subclass it.
Such form of datasets is particularly useful when data come from a stream.
All subclasses should overwrite :meth:`__iter__`, which would return an
iterator of samples in this dataset.
When a subclass is used with :class:`~torch.utils.data.DataLoader`, each
item in the dataset will be yielded from the :class:`~torch.utils.data.DataLoader`
iterator. When :attr:`num_workers > 0`, each worker process will have a
different copy of the dataset object, so it is often desired to configure
each copy independently to avoid having duplicate data returned from the
workers. :func:`~torch.utils.data.get_worker_info`, when called in a worker
process, returns information about the worker. It can be used in either the
dataset's :meth:`__iter__` method or the :class:`~torch.utils.data.DataLoader` 's
:attr:`worker_init_fn` option to modify each copy's behavior.
Example 1: splitting workload across all workers in :meth:`__iter__`::
>>> class MyIterableDataset(torch.utils.data.IterableDataset):
... def __init__(self, start, end):
... super(MyIterableDataset).__init__()
... assert end > start, "this example code only works with end >= start"
... self.start = start
... self.end = end
...
... def __iter__(self):
... worker_info = torch.utils.data.get_worker_info()
... if worker_info is None: # single-process data loading, return the full iterator
... iter_start = self.start
... iter_end = self.end
... else: # in a worker process
... # split workload
... per_worker = int(math.ceil((self.end - self.start) / float(worker_info.num_workers)))
... worker_id = worker_info.id
... iter_start = self.start + worker_id * per_worker
... iter_end = min(iter_start + per_worker, self.end)
... return iter(range(iter_start, iter_end))
...
>>> # should give same set of data as range(3, 7), i.e., [3, 4, 5, 6].
>>> ds = MyIterableDataset(start=3, end=7)
>>> # Single-process loading
>>> print(list(torch.utils.data.DataLoader(ds, num_workers=0)))
[3, 4, 5, 6]
>>> # Mult-process loading with two worker processes
>>> # Worker 0 fetched [3, 4]. Worker 1 fetched [5, 6].
>>> print(list(torch.utils.data.DataLoader(ds, num_workers=2)))
[3, 5, 4, 6]
>>> # With even more workers
>>> print(list(torch.utils.data.DataLoader(ds, num_workers=20)))
[3, 4, 5, 6]
Example 2: splitting workload across all workers using :attr:`worker_init_fn`::
>>> class MyIterableDataset(torch.utils.data.IterableDataset):
... def __init__(self, start, end):
... super(MyIterableDataset).__init__()
... assert end > start, "this example code only works with end >= start"
... self.start = start
... self.end = end
...
... def __iter__(self):
... return iter(range(self.start, self.end))
...
>>> # should give same set of data as range(3, 7), i.e., [3, 4, 5, 6].
>>> ds = MyIterableDataset(start=3, end=7)
>>> # Single-process loading
>>> print(list(torch.utils.data.DataLoader(ds, num_workers=0)))
[3, 4, 5, 6]
>>>
>>> # Directly doing multi-process loading yields duplicate data
>>> print(list(torch.utils.data.DataLoader(ds, num_workers=2)))
[3, 3, 4, 4, 5, 5, 6, 6]
>>> # Define a `worker_init_fn` that configures each dataset copy differently
>>> def worker_init_fn(worker_id):
... worker_info = torch.utils.data.get_worker_info()
... dataset = worker_info.dataset # the dataset copy in this worker process
... overall_start = dataset.start
... overall_end = dataset.end
... # configure the dataset to only process the split workload
... per_worker = int(math.ceil((overall_end - overall_start) / float(worker_info.num_workers)))
... worker_id = worker_info.id
... dataset.start = overall_start + worker_id * per_worker
... dataset.end = min(dataset.start + per_worker, overall_end)
...
>>> # Mult-process loading with the custom `worker_init_fn`
>>> # Worker 0 fetched [3, 4]. Worker 1 fetched [5, 6].
>>> print(list(torch.utils.data.DataLoader(ds, num_workers=2, worker_init_fn=worker_init_fn)))
[3, 5, 4, 6]
>>> # With even more workers
>>> print(list(torch.utils.data.DataLoader(ds, num_workers=20, worker_init_fn=worker_init_fn)))
[3, 4, 5, 6]
N)r- c C s t d S )N)r5 )r r r r r. s zIterableDataset.__iter__)r8 c C s t | |gS )N) ChainDataset)r r8 r r r r: s zIterableDataset.__add__c C s( |t jkr tjt j| | }|S td S )N)rQ r; r< r= r> )r r? r@ r r r rA s
zIterableDataset.__getattr__c s8 t jd k r*y
t j| S tk
r( Y nX t j||S )N)rQ reduce_ex_hookr5 r
__reduce_ex__)r rI rJ )r r r rT s
z IterableDataset.__reduce_ex__c C s$ t jd k r|d k rtd|t _d S )Nz+Attempt to override existing reduce_ex_hook)rQ rS rL )rB Zhook_fnr r r set_reduce_ex_hook s z"IterableDataset.set_reduce_ex_hook)r0 r1 r2 rO r; r r# r rS r r r r. r4 r: rA rT rP rU r3 r r )r r rQ k s
e rQ ) metaclassc @ s e Zd Zdd ZdS )rF c C s dS )NTr )r r r r
_is_dfpipe s zDFIterDataPipe._is_dfpipeN)r0 r1 r2 rW r r r r rF s rF c @ s> e Zd ZU dZeedf eddddZdd Zd d
Z dS )
TensorDatasetzDataset wrapping tensors.
Each sample will be retrieved by indexing tensors along the first dimension.
Args:
*tensors (Tensor): tensors that have the same size of the first dimension.
.N)tensorsr- c s( t fdd D s td | _d S )Nc 3 s& | ] } d j d |j d kV qdS )r N)size)r$ tensor)rY r r r& s z)TensorDataset.__init__..z Size mismatch between tensors)allAssertionErrorrY )r rY r )rY r r s zTensorDataset.__init__c s t fdd| jD S )Nc 3 s | ]}| V qd S )Nr )r$ r[ )r6 r r r& s z,TensorDataset.__getitem__..)tuplerY )r r6 r )r6 r r7 s zTensorDataset.__getitem__c C s | j d jdS )Nr )rY rZ )r r r r __len__ s zTensorDataset.__len__)
r0 r1 r2 rO r
r rY r r7 r_ r r r r rX s
rX .c sj e Zd ZU dZeee ee e
dd Z e e dd fdd Z
dd Zd
d Zed d
Z ZS )r9 zDataset as a concatenation of multiple datasets.
This class is useful to assemble different existing datasets.
Args:
datasets (sequence): List of datasets to be concatenated
c C s: g d }}x*| D ]"}t |}|j|| ||7 }qW |S )Nr )lenappend)sequencerselr r r cumsum s
zConcatDataset.cumsumN)datasetsr- c sd t t| j t|| _t| jdks.tdx"| jD ]}t|t s6tdq6W | j | j| _
d S )Nr z(datasets should not be an empty iterablez.ConcatDataset does not support IterableDataset) r r9 r listrh r` r] rE rQ rg cumulative_sizes)r rh d)r r r r s
zConcatDataset.__init__c C s
| j d S )N )rj )r r r r r_ ' s zConcatDataset.__len__c C sf |dk r*| t | kr tdt | | }tj| j|}|dkrF|}n|| j|d }| j| | S )Nr z8absolute value of index should not exceed dataset lengthrl )r`
ValueErrorbisect bisect_rightrj rh )r idxZ dataset_idxZ
sample_idxr r r r7 * s zConcatDataset.__getitem__c C s t jdtdd | jS )Nz:cummulative_sizes attribute is renamed to cumulative_sizes )
stacklevel)warningswarnDeprecationWarningrj )r r r r cummulative_sizes6 s
z ConcatDataset.cummulative_sizes)r0 r1 r2 rO r r4 r rh intrj staticmethodrg r r r_ r7 propertyrw r3 r r )r r r9 s
r9 c s< e Zd ZdZee dd fdd Zdd Zdd Z Z S )
rR a_ Dataset for chaining multiple :class:`IterableDataset` s.
This class is useful to assemble different existing dataset streams. The
chaining operation is done on-the-fly, so concatenating large-scale
datasets with this class will be efficient.
Args:
datasets (iterable of IterableDataset): datasets to be chained together
N)rh r- c s t t| j || _d S )N)r rR r rh )r rh )r r r r G s zChainDataset.__init__c c s: x4| j D ]*}t|ts tdx|D ]
}|V q$W qW d S )Nz*ChainDataset only supports IterableDataset)rh rE rQ r] )r rk xr r r r. K s
zChainDataset.__iter__c C s6 d}x,| j D ]"}t|ts"td|t|7 }q W |S )Nr z*ChainDataset only supports IterableDataset)rh rE rQ r] r` )r totalrk r r r r_ Q s
zChainDataset.__len__)
r0 r1 r2 rO r r4 r r. r_ r3 r r )r r rR = s rR c @ sL e Zd ZU dZee ee ee ee ddddZ
dd Z dd Z dS )
Subsetz
Subset of a dataset at specified indices.
Args:
dataset (Dataset): The whole Dataset
indices (sequence): Indices in the whole set selected for subset
N)datasetindicesr- c C s || _ || _d S )N)r~ r )r r~ r r r r r d s zSubset.__init__c s2 t |tr" j fdd|D S j j| S )Nc s g | ]} j | qS r )r )r$ r% )r r r
j s z&Subset.__getitem__..)rE ri r~ r )r rq r )r r r7 h s
zSubset.__getitem__c C s
t | jS )N)r` r )r r r r r_ m s zSubset.__len__)
r0 r1 r2 rO r4 r r~ r rx r r r7 r_ r r r r r} Y s
r} )r~ lengths generatorr- c sJ t |t krtdtt ||dj fddtt||D S )a
Randomly split a dataset into non-overlapping new datasets of given lengths.
Optionally fix the generator for reproducible results, e.g.:
>>> random_split(range(10), [3, 7], generator=torch.Generator().manual_seed(42))
Args:
dataset (Dataset): Dataset to be split
lengths (sequence): lengths of splits to be produced
generator (Generator): Generator used for the random permutation.
zDSum of input lengths does not equal the length of the input dataset!)r c s& g | ] \}}t || | qS r )r} )r$ offsetlength)r~ r r r r s z random_split..)sumr` rn r
tolistzipr )r~ r r r )r~ r r random_splitq s r )&ro r< rt typingr r r r r r r r r
r Ztorchr r
Z torch._utilsr Ztorch.utils.data._typingr r! r r r r rG ri r r4 rQ rF rX r9 rR r} rx r r r r r s. 0 8 2