/usr/local/lib64/python3.6/site-packages/torch/utils/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/utils/__pycache__/checkpoint.cpython-36.pyc (10397B)
3
Eg/ @ s d dl Z d dlZd dlmZmZmZmZ eedf ee jdf dddZee ddddZ eee
ee j f d d
d Z dd d d
Z G dd de j
jZdd Zdd ZdS ) N)AnyIterableListTuple.)inputsreturnc C sj t | trVg }x>| D ]6}t |tjs0|j| q|j }|j|_|j| qW t|S tdt| j d S )Nz@Only tuple of tensors is supported. Got Unsupported input type: )
isinstancetupletorchTensorappenddetach
requires_grad RuntimeErrortype__name__)r outinpx r B/usr/local/lib64/python3.6/site-packages/torch/utils/checkpoint.pydetach_variable s
r c C s t dd | D s tjd d S )Nc s s | ]}t |tjr|jV qd S )N)r r
r r ).0r r r r
s z*check_backward_validity..zBNone of the inputs have requires_grad=True. Gradients will be None)anywarningswarn)r r r r check_backward_validity s r )r c
G sX t tdd | D }g }x4|D ],}tjj| |jtjj W d Q R X q W ||fS )Nc s s( | ] }t |tjr|jr|j V qd S )N)r r
r Zis_cudaZ
get_device)r argr r r r & s z$get_device_states..)listsetr
cudadevicer
get_rng_state)argsfwd_gpu_devicesfwd_gpu_statesr" r r r get_device_states# s
r' c
C s@ x:t | |D ],\}}tjj| tjj| W d Q R X q W d S )N)zipr
r! r"
set_rng_state)devicesZstatesr" stater r r set_device_states1 s r, c @ s$ e Zd Zedd Zedd ZdS )CheckpointFunctionc G s t | || _|| _tj | _|rPtj | _d| _tj j
rPd| _t | \| _ | _
g | _g | _g }xNt|D ]B\}}tj|r|j| | jj| | jjd qj| jj| qjW | j| tj || }W d Q R X |S )NFT)r run_functionpreserve_rng_stater
Zis_autocast_enabledhad_autocast_in_fwdr#
fwd_cpu_statehad_cuda_in_fwdr! _initializedr' r% r& r tensor_indices enumerate is_tensorr Zsave_for_backwardZno_grad)ctxr. r/ r$ Z
tensor_inputsir outputsr r r forward9 s.
zCheckpointFunction.forwardc
G s t jj stdt| j}| j}| j}x t|D ]\}}|| ||< q2W g }| j r`| j
r`| j }t j j
|| j dr | j rt j| j | j
rt| j | j tt|}t j , t jjj| j | j| } W d Q R X W d Q R X W d Q R X t| t jr| f} g }
g } xNt t | D ]>}t j | | r| | j r|
j | | | j || qW t |
dkrjtdt jj!|
| tdd |D } d| S )NzCheckpointing is not compatible with .grad() or when an `inputs` parameter is passed to .backward(). Please use .backward() and do not pass its `inputs` argument.)r* enabledr zInone of output has requires_grad=True, this checkpoint() is not necessaryc s s$ | ] }t |tjr|jnd V qd S )N)r r
r Zgrad)r r r r r r s z.CheckpointFunction.backward..)NN)"r
autogradZ_is_checkpoint_validr r r r4 Z
saved_tensorsr5 r/ r2 r% randomZfork_rngr) r1 r, r& r r Z enable_gradr! ampZautocastr0 r. r r rangelenr6 r r backward)
r7 r$ r r4 Ztensorsr8 idxZ rng_devicesZdetached_inputsr9 Zoutputs_with_gradZargs_with_gradZgradsr r r rA ] sD
(
zCheckpointFunction.backwardN)r
__module__ __qualname__ staticmethodr: rA r r r r r- 7 s $r- c O s> |j dd}|r,tddjdd |D tj| |f| S )a* Checkpoint a model or part of the model
Checkpointing works by trading compute for memory. Rather than storing all
intermediate activations of the entire computation graph for computing
backward, the checkpointed part does **not** save intermediate activations,
and instead recomputes them in backward pass. It can be applied on any part
of a model.
Specifically, in the forward pass, :attr:`function` will run in
:func:`torch.no_grad` manner, i.e., not storing the intermediate
activations. Instead, the forward pass saves the inputs tuple and the
:attr:`function` parameter. In the backwards pass, the saved inputs and
:attr:`function` is retrieved, and the forward pass is computed on
:attr:`function` again, now tracking the intermediate activations, and then
the gradients are calculated using these activation values.
The output of :attr:`function` can contain non-Tensor values and gradient
recording is only performed for the Tensor values. Note that if the output
consists of nested structures (ex: custom objects, lists, dicts etc.)
consisting of Tensors, these Tensors nested in custom structures will not
be considered as part of autograd.
.. warning::
Checkpointing currently only supports :func:`torch.autograd.backward`
and only if its `inputs` argument is not passed. :func:`torch.autograd.grad`
is not supported.
.. warning::
If :attr:`function` invocation during backward does anything different
than the one during forward, e.g., due to some global variable, the
checkpointed version won't be equivalent, and unfortunately it can't be
detected.
.. warning::
If checkpointed segment contains tensors detached from the computational
graph by `detach()` or `torch.no_grad()`, the backward pass will raise an
error. This is because `checkpoint` makes all the outputs require
gradients which causes issues when a tensor is defined to have no
gradient in the model. To circumvent this, detach the tensors outside of
the `checkpoint` function.
.. warning::
At least one of the inputs needs to have :code:`requires_grad=True` if
grads are needed for model inputs, otherwise the checkpointed part of the
model won't have gradients. At least one of the outputs needs to have
:code:`requires_grad=True` as well.
Args:
function: describes what to run in the forward pass of the model or
part of the model. It should also know how to handle the inputs
passed as the tuple. For example, in LSTM, if user passes
``(activation, hidden)``, :attr:`function` should correctly use the
first input as ``activation`` and the second input as ``hidden``
preserve_rng_state(bool, optional, default=True): Omit stashing and restoring
the RNG state during each checkpoint.
args: tuple containing inputs to the :attr:`function`
Returns:
Output of running :attr:`function` on :attr:`*args`
r/ Tz Unexpected keyword arguments: ,c s s | ]
}|V qd S )Nr )r r r r r r s z checkpoint..)pop
ValueErrorjoinr- apply)functionr$ kwargspreserver r r
checkpoint s > rN c K s |j dd}|r,tddjdd |D dd }t| tjjrNt| j } t | | }d }x>t
d
||d |D ]&}|| d }t |||| ||d }qtW ||d t | d | |S )
a A helper function for checkpointing sequential models.
Sequential models execute a list of modules/functions in order
(sequentially). Therefore, we can divide such a model in various segments
and checkpoint each segment. All segments except the last will run in
:func:`torch.no_grad` manner, i.e., not storing the intermediate
activations. The inputs of each checkpointed segment will be saved for
re-running the segment in the backward pass.
See :func:`~torch.utils.checkpoint.checkpoint` on how checkpointing works.
.. warning::
Checkpointing currently only supports :func:`torch.autograd.backward`
and only if its `inputs` argument is not passed. :func:`torch.autograd.grad`
is not supported.
.. warning:
At least one of the inputs needs to have :code:`requires_grad=True` if
grads are needed for model inputs, otherwise the checkpointed part of the
model won't have gradients.
.. warning:
Since PyTorch 1.4, it allows only one Tensor as the input and
intermediate outputs, just like :class:`torch.nn.Sequential`.
Args:
functions: A :class:`torch.nn.Sequential` or the list of modules or
functions (comprising the model) to run sequentially.
segments: Number of chunks to create in the model
input: A Tensor that is input to :attr:`functions`
preserve_rng_state(bool, optional, default=True): Omit stashing and restoring
the RNG state during each checkpoint.
Returns:
Output of running :attr:`functions` sequentially on :attr:`*inputs`
Example:
>>> model = nn.Sequential(...)
>>> input_var = checkpoint_sequential(model, chunks, input_var)
r/ Tz Unexpected keyword arguments: rF c s s | ]
}|V qd S )Nr )r r r r r r s z(checkpoint_sequential..c s fdd}|S )Nc s( x"t d D ]}| | } qW | S )N )r? )inputj)end functionsstartr r r: s z.run_function..forwardr )rT rR rS r: r )rR rS rT r r. s z+checkpoint_sequential..run_functionrO r )r/ ) rG rH rI r r
nnZ
Sequentialr childrenr@ r? rN ) rS segmentsrP rL rM r. Z segment_sizerR rT r r r checkpoint_sequential s * rY )r
r typingr r r r r r r intr' r, r< ZFunctionr- rN rY r r r r s " ZE