/usr/local/lib64/python3.6/site-packages/torch/utils/data/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/utils/data/__pycache__/dataloader.cpython-36.pyc (24145B)
3
Eg @ s` U d Z ddlZddlZddlZddlZddlZddlmZmZm Z m
Z
m Z m Z m
Z
ddlZddlZddljZddlmZ ddlmZ ddlmZmZmZmZmZm Z ddlm Z e d d
d Z e d Z ee gdf Z!ee e gef Z"e j#j$Z$e"$e j%j&Z&G d
d de'Z(G dd deZ)G dd de
e Z*G dd de'Z+G dd de+Z,G dd de+Z-dS )a Definition of the DataLoader and associated iterators that subclass _BaseDataLoaderIter
To support these two classes, in `./_utils` we define many utility methods and
functions to be run in multiprocessing. E.g., the data loading worker loop is
in `./_utils/worker.py`.
N)AnyCallableTypeVarGenericSequenceListOptional)ExceptionWrapper)string_classes )IterableDatasetSamplerSequentialSampler
RandomSampler BatchSamplerDataset)_utilsT_coT) covariantTc @ s e Zd ZdZdZedd ZdS ) _DatasetKindr r c C s2 | t jkr tjj||||S tjj||||S d S )N)r Mapr fetchZ_MapDatasetFetcherZ_IterableDatasetFetcher)kinddatasetZauto_collation
collate_fn drop_last r G/usr/local/lib64/python3.6/site-packages/torch/utils/data/dataloader.pycreate_fetcher/ s
z_DatasetKind.create_fetcherN)__name__
__module__ __qualname__r Iterable staticmethodr r r r r r + s r c s( e Zd ZdZ fddZdd Z ZS )_InfiniteConstantSamplerzAnalogous to ``itertools.repeat(None, None)``.
Used as sampler for :class:`~torch.utils.data.IterableDataset`.
Args:
data_source (Dataset): dataset to sample from
c s t t| jd d S )N)superr% __init__)self) __class__r r r' ? s z!_InfiniteConstantSampler.__init__c c s x
d V qW d S )Nr )r( r r r __iter__B s z!_InfiniteConstantSampler.__iter__)r r! r" __doc__r' r*
__classcell__r r )r) r r% 7 s r% c s e Zd ZU dZee ee e
e e
eeeed dZd dddee ee e ee eee eee e e eee ee d
d
d Zdd d
dZedd Zej dd Z fddZ dd ddZ edd Z edd Z ed ddZ!d d Z" Z#S )
DataLoadera
Data loader. Combines a dataset and a sampler, and provides an iterable over
the given dataset.
The :class:`~torch.utils.data.DataLoader` supports both map-style and
iterable-style datasets with single- or multi-process loading, customizing
loading order and optional automatic batching (collation) and memory pinning.
See :py:mod:`torch.utils.data` documentation page for more details.
Args:
dataset (Dataset): dataset from which to load the data.
batch_size (int, optional): how many samples per batch to load
(default: ``1``).
shuffle (bool, optional): set to ``True`` to have the data reshuffled
at every epoch (default: ``False``).
sampler (Sampler or Iterable, optional): defines the strategy to draw
samples from the dataset. Can be any ``Iterable`` with ``__len__``
implemented. If specified, :attr:`shuffle` must not be specified.
batch_sampler (Sampler or Iterable, optional): like :attr:`sampler`, but
returns a batch of indices at a time. Mutually exclusive with
:attr:`batch_size`, :attr:`shuffle`, :attr:`sampler`,
and :attr:`drop_last`.
num_workers (int, optional): how many subprocesses to use for data
loading. ``0`` means that the data will be loaded in the main process.
(default: ``0``)
collate_fn (callable, optional): merges a list of samples to form a
mini-batch of Tensor(s). Used when using batched loading from a
map-style dataset.
pin_memory (bool, optional): If ``True``, the data loader will copy Tensors
into CUDA pinned memory before returning them. If your data elements
are a custom type, or your :attr:`collate_fn` returns a batch that is a custom type,
see the example below.
drop_last (bool, optional): set to ``True`` to drop the last incomplete batch,
if the dataset size is not divisible by the batch size. If ``False`` and
the size of dataset is not divisible by the batch size, then the last batch
will be smaller. (default: ``False``)
timeout (numeric, optional): if positive, the timeout value for collecting a batch
from workers. Should always be non-negative. (default: ``0``)
worker_init_fn (callable, optional): If not ``None``, this will be called on each
worker subprocess with the worker id (an int in ``[0, num_workers - 1]``) as
input, after seeding and before data loading. (default: ``None``)
generator (torch.Generator, optional): If not ``None``, this RNG will be used
by RandomSampler to generate random indexes and multiprocessing to generate
`base_seed` for workers. (default: ``None``)
prefetch_factor (int, optional, keyword-only arg): Number of samples loaded
in advance by each worker. ``2`` means there will be a total of
2 * num_workers samples prefetched across all workers. (default: ``2``)
persistent_workers (bool, optional): If ``True``, the data loader will not shutdown
the worker processes after a dataset has been consumed once. This allows to
maintain the workers `Dataset` instances alive. (default: ``False``)
.. warning:: If the ``spawn`` start method is used, :attr:`worker_init_fn`
cannot be an unpicklable object, e.g., a lambda function. See
:ref:`multiprocessing-best-practices` on more details related
to multiprocessing in PyTorch.
.. warning:: ``len(dataloader)`` heuristic is based on the length of the sampler used.
When :attr:`dataset` is an :class:`~torch.utils.data.IterableDataset`,
it instead returns an estimate based on ``len(dataset) / batch_size``, with proper
rounding depending on :attr:`drop_last`, regardless of multi-process loading
configurations. This represents the best guess PyTorch can make because PyTorch
trusts user :attr:`dataset` code in correctly handling multi-process
loading to avoid duplicate data.
However, if sharding results in multiple workers having incomplete last batches,
this estimate can still be inaccurate, because (1) an otherwise complete batch can
be broken into multiple ones and (2) more than one batch worth of samples can be
dropped when :attr:`drop_last` is set. Unfortunately, PyTorch can not detect such
cases in general.
See `Dataset Types`_ for more details on these two types of datasets and how
:class:`~torch.utils.data.IterableDataset` interacts with
`Multi-process data loading`_.
.. warning:: See :ref:`reproducibility`, and :ref:`dataloader-workers-random-seed`, and
:ref:`data-loading-randomness` notes for random seed related questions.
_BaseDataLoaderIterFr Nr )prefetch_factorpersistent_workers)
r
batch_sizeshufflesampler
batch_sampler num_workersr
pin_memoryr timeoutworker_init_fnr0 r1 c C s0 t jjd |dk r td|
dk r,td|dkrD|dkrDtd|dksPt|rd|dkrdtd|| _|| _|| _|| _|
| _ | | _
| | _ t |t
rtj| _|dk rtd j|q|d k rtd
j|q|d k rtd j|ntj| _|d k o|rtd |d k rB|d
ks0|s0|d k s0| r8tdd }d} n|d krZ| rZtd|d kr| jtjkrzt }n |rt||
d}nt|}|d k r|d krt||| }|| _| | _|| _|| _|
| _|d kr| j rt j j }nt j j }|| _!|| _"d| _#d | _$d | _%| j& t j'ddd d S )Nzpython.data_loaderr zXnum_workers option should be non-negative; use num_workers=0 to disable multiprocessing.z%timeout option should be non-negativer/ zpprefetch_factor option could only be specified in multiprocessing.let num_workers > 0 to enable multiprocessing.z/persistent_workers option needs num_workers > 0FzXDataLoader with IterableDataset: expected unspecified shuffle option, but got shuffle={}zXDataLoader with IterableDataset: expected unspecified sampler option, but got sampler={}zdDataLoader with IterableDataset: expected unspecified batch_sampler option, but got batch_sampler={}z1sampler option is mutually exclusive with shuffler z[batch_sampler option is mutually exclusive with batch_size, shuffle, sampler, and drop_lastzVbatch_size=None option disables auto-batching and is mutually exclusive with drop_last) generatorTZ
DataloaderenabledTrue)(torch_CZ_log_api_usage_once
ValueErrorAssertionErrorr r6 r0 r7 r8 r9 multiprocessing_context
isinstancer r r#
_dataset_kindformatr r% r r r r2 r r4 r5 r: _auto_collationr collatedefault_collateZdefault_convertr r1 _DataLoader__initialized_IterableDataset_len_called _iterator check_worker_number_rationalityZ set_vital)r( r r2 r3 r4 r5 r6 r r7 r r8 r9 rA r: r0 r1 r r r r' s
zDataLoader.__init__)returnc C s& | j dkrt| S | j t| S d S )Nr )r6 _SingleProcessDataLoaderIterrK _MultiProcessingDataLoaderIter)r( r r r
_get_iterator, s
zDataLoader._get_iteratorc C s | j S )N)$_DataLoader__multiprocessing_context)r( r r r rA 3 s z"DataLoader.multiprocessing_contextc C s~ |d k rt| j dkrdt|trFtj }||kr
0), but got num_workers={})
r6 rB r
multiprocessingget_all_start_methodsr? rD get_contextpython_multiprocessingcontext BaseContext TypeErrorrP )r( rA Zvalid_start_methodsr r r rA 7 s
c s8 | j r"|dkr"tdj|| jjtt| j|| d S ) Nr2 r5 r4 r r r1 z6{} attribute should not be set after {} is initialized)r2 r5 r4 r r r1 )rH r? rD r) r r& r- __setattr__)r( attrval)r) r r rX P s
zDataLoader.__setattr__c C sD | j r8| jdkr8| jd kr&| j | _n | jj| | jS | j S d S )Nr )r1 r6 rJ rO _reset)r( r r r r* Z s
zDataLoader.__iter__c C s
| j d k S )N)r5 )r( r r r rE i s zDataLoader._auto_collationc C s | j r | jS | jS d S )N)rE r5 r4 )r( r r r _index_samplerm s zDataLoader._index_samplerc C sd | j tjkrVt| j }| _| jd k rRddlm} | j rD|| j }n||| j }|S t| j
S d S )Nr )ceil) rC r r# lenr rI r2 mathr] r r\ )r( lengthr] r r r __len__y s
zDataLoader.__len__c
C s dd }| j s| j dkr d S d }d}ttdr\yttjd}d}W n tk
rZ Y nX |d krxtj }|d k rx|}|d krtj||| j | d S | j |krtj||| j | d S )Nc S s0 | d k r dj | |rdndnd}dj ||}|S )Nz|Our suggested max number of worker in current system is {}{}, which is smaller than what this DataLoader is going to create. z% (`cpuset` is not taken into account)zUDataLoader is not able to compute a suggested max number of worker in current system.zThis DataLoader will create {} worker processes in total. {} Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.)rD )Znum_worker_suggestZnum_worker_createdcpuset_checkedZsuggested_max_worker_msgwarn_msgr r r _create_warning_msg s zGDataLoader.check_worker_number_rationality.._create_warning_msgr Fsched_getaffinityT) r6 hasattrosr^ rf Exception cpu_countwarningswarn)r( re Zmax_num_worker_suggestrc rj r r r rK s4
z*DataLoader.check_worker_number_rationality) r FNNr NFFr NNN)$r r! r" r+ r r r r intr2 r6 boolr7 r floatr8 r
r4 r0 rJ rH r
_collate_fn_t_worker_init_fn_tr' rO propertyrA setterrX r* rE r\ ra rK r, r r )r) r r- G s<
O H
r- c @ sl e Zd ZeddddZd dddZdd d
Zd d Zd
d Ze dddZ
e
Z e dddZ
dd ZdS )r. N)loaderrL c C s |j | _|j| _|j| _|j| _|j| _|j| _|j| _ |j
| _ |j oNt
jj | _|j| _|j| _t| j| _t
jf t
jdj|jdj | _ |j | _ d| _ dj | j!j"| _#d S )N)Zdtype)r: r z!enumerate(DataLoader)#{}.__next__)$r _datasetrC rI rE r
_drop_lastr\ r6 _num_workersr0 _prefetch_factorr7 r= cudaZ is_available _pin_memoryr8 _timeoutr _collate_fniter
_sampler_iteremptyZint64Zrandom_r: item
_base_seedr1 _persistent_workers _num_yieldedrD r) r
_profile_name)r( rt r r r r' s z _BaseDataLoaderIter.__init__)rL c C s | S )Nr )r( r r r r* s z _BaseDataLoaderIter.__iter__Fc C s t | j| _d| _|j| _d S )Nr )r} r\ r~ r rI )r( rt
first_iterr r r r[ s z_BaseDataLoaderIter._resetc C s
t | jS )N)nextr~ )r( r r r _next_index s z _BaseDataLoaderIter._next_indexc C s t d S )N)NotImplementedError)r( r r r
_next_data s z _BaseDataLoaderIter._next_datac C s t jjj| j | jd kr$| j | j }| jd7 _| j t
j kr| j d k r| j| j krdj
| j| j | j}| jdkr|d7 }tj| |S Q R X d S )Nr zwLength of IterableDataset {} was reported to be {} (when accessing len(dataloader)), but {} samples have been fetched. r zFor multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.)r= ZautogradZprofilerZrecord_functionr r~ r[ r r rC r r# rI rD ru rw rk rl )r( datard r r r __next__ s
z _BaseDataLoaderIter.__next__c C s
t | jS )N)r^ r\ )r( r r r ra s z_BaseDataLoaderIter.__len__c C s t d| jjd S )Nz{} cannot be pickled)r r) r )r( r r r __getstate__ s z _BaseDataLoaderIter.__getstate__)F)r r! r" r- r' r* r[ r r r r r rm ra r r r r r r. s
r. c s$ e Zd Z fddZdd Z ZS )rM c sN t t| j| | jdks t| jdks,ttj| j| j | j
| j | j | _
d S )Nr )r& rM r' r{ r@ rw r r rC ru rE r| rv _dataset_fetcher)r( rt )r) r r r' ' s
z%_SingleProcessDataLoaderIter.__init__c C s* | j }| jj|}| jr&tjj|}|S )N)r r r rz r r7 )r( indexr r r r r / s
z'_SingleProcessDataLoaderIter._next_data)r r! r" r' r r, r r )r) r rM & s rM c sv e Zd ZdZ fddZd fdd ZejfddZd d
Z d d Z
d
d Z dd Z dddZ
dd Zdd Z ZS )rN zHIterates once over the DataLoader's dataset, as specified by the samplerc s t t| j| | jdks t| jdks,t|jd kr s z:_MultiProcessingDataLoaderIter.__init__..)r )7r& rN r' rw r@ rx rA rQ r9 Z_worker_init_fn itertoolscyclerange_worker_queue_idx_cycleQueue_worker_result_queue_worker_pids_set _shutdownEvent_workers_done_event
_index_queues_workerscancel_join_threadProcessr workerZ _worker_looprC ru rE r| rv r r daemonstartappendrz threading _pin_memory_thread_done_eventqueue _data_queueThreadr7 Z_pin_memory_loopr= ry Zcurrent_device_pin_memory_threadsignal_handlingZ_set_worker_pidsidtupleZ_set_SIGCHLD_handlerr[ )r( rt rA iZ index_queuer Zpin_memory_thread)r) r r r' m sX
"
z'_MultiProcessingDataLoaderIter.__init__Fc s t j|| d| _d| _i | _d| _dd t| jD | _|sx(t| jD ]}| j | j
t j j
qLW | j}x:|dkr| j \}}t|t j j
rr|d kst|d8 }qrW x t| j| j D ] }| j qW d S )Nr c S s g | ]}d qS )Tr )r r r r r
s z9_MultiProcessingDataLoaderIter._reset..r )r& r[ _send_idx _rcvd_idx
_task_info_tasks_outstandingr rw _workers_statusr putr r Z_ResumeIteration _get_datarB r@ rx _try_put_index)r( rt r idxZresume_iteration_cntZ
return_idxZ return_data_)r) r r r[ s"
z%_MultiProcessingDataLoaderIter._resetc s2 y| j j|d}d|fS tk
r, } zg }x>t| jD ]0\}}| j| r:|j r:|j| | j| q:W t |dkrdj
dd |D }t dj ||t
|tjrd
S dd l dd l}y d } fd
d t| D }
W n< tk
r } z |j|jkrt d d W Y d d }~X nX W Y d d }~X nX d S )N)r8 Tr z, c s s | ]}t |jV qd S )N)strr )r r r r r r s z?_MultiProcessingDataLoaderIter._try_get_data..z1DataLoader worker (pid(s) {}) exited unexpectedlyF
c s g | ] } j qS r )NamedTemporaryFile)r r )tempfiler r r s z@_MultiProcessingDataLoaderIter._try_get_data..a Too many open files. Communication with the workers is no longer possible. Please increase the limit using `ulimit -n` in the shell or change the sharing strategy by calling `torch.multiprocessing.set_sharing_strategy('file_system')` at the beginning of your code)FN)r getri enumerater r is_aliver _mark_worker_as_unavailabler^ join RuntimeErrorrD rB r Emptyr errnor OSErrorEMFILE) r( r8 r eZfailed_workers worker_idr Zpids_strr Zfds_limit_marginfsr )r r
_try_get_data s2
z,_MultiProcessingDataLoaderIter._try_get_datac C s | j dkr4| j| j \}}|r"|S tdj| j nN| jrhxF| jj r\| j \}}|r<|S qP | j| j = | j d7 _ qW | jsh| j tt| j| j dkr| jj| j d }| j |S | j
r| j dkst | j
\}}| j d8 _ | jtjkrt|tjjr| jr d| j|j< n | j|j | j q|| j kr8| j| |f7 < q| j|= | j |S qW d S )Nr r/ r F)r r r r^ r r _shutdown_workers
StopIterationpop
_process_datar r r@ r rC r r# rB r r Z _IterableDatasetStopIterationr r r )r( infor r r r r r r s8
z)_MultiProcessingDataLoaderIter._next_datac C s | j | j| j k sty | j }W n tk
r6 d S X x,t| jD ]}t| j}| j | rDP qDW d S | j
| j | j |f |f| j
| j < | j d7 _ | j d7 _ d S )Nr )r rx rw r@ r r r r r r r r r r )r( r r Zworker_queue_idxr r r r s
z-_MultiProcessingDataLoaderIter._try_put_indexc C s, | j d7 _ | j t|tr(|j |S )Nr )r r rB r reraise)r( r r r r r s
z,_MultiProcessingDataLoaderIter._process_datac C sL | j | s| jr|st| j| }|jd d| j |< | jj |ksHtd S )NF)r r r@ r r r is_set)r( r shutdownqr r r r s
z:_MultiProcessingDataLoaderIter._mark_worker_as_unavailablec C s2 t j}|dks|d krd S | js.d| _zt| drh| jj | jjd | jj | jj
| jj | j j x4t
t| jD ]"}| js| j| r| j|dd qW x | jD ]}|j t jd qW x | jD ]}|j
|j qW W d | jrt jjt| d| _x"| jD ]}|j r|j qW X d S )NTr )r )r8 F)NN)r python_exit_statusr rg r setr r r r r closer r r^ r r r r MP_STATUS_CHECK_INTERVALr r r Z_remove_worker_pidsr r terminate)r( r r r r r r r r s6
z0_MultiProcessingDataLoaderIter._shutdown_workersc C s | j d S )N)r )r( r r r __del__/ s z&_MultiProcessingDataLoaderIter.__del__)F)F)r r! r" r+ r' r[ r r r r r r r r r r r, r r )r) r rN 7 s 7 E !1
ErN ).r+ rh r r rk r typingr r r r r r r rQ rT r= Ztorch.multiprocessingZ torch._utilsr Z
torch._sixr
rb r r
r r r r r r r rm rq rp rF rG r Zget_worker_infoobjectr r% r- r. rM rN r r r r s8 $
B