/usr/local/lib64/python3.6/site-packages/torch/autograd/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/autograd/__pycache__/functional.cpython-36.pyc (31543B)
3
Eg6 @ s d dl Z d dlmZmZ d dlmZ dd Zdd Zdd Zd
d Z d d
Z
dd Z d%ddZ dd Z
d&ddZd'ddZee jdf eedf ee jdf ddd Zd(d d Zd)d d Zd*d!d"Zd+d#d$ZdS ), N)TupleList)_vmapc C sx d}t | ts| f} d}xVt| D ]J\}}t |tjs"|rTtdj|||t|q"tdj|||t|q"W || fS )NTFzgThe {} given to {} must be either a Tensor or a tuple of Tensors but the value at index {} has type {}.z^The {} given to {} must be either a Tensor or a tuple of Tensors but the given {} has type {}.)
isinstancetuple enumeratetorchTensor TypeErrorformattype)inpZarg_namefn_nameZ is_inp_tupleiel r E/usr/local/lib64/python3.6/site-packages/torch/autograd/functional.py _as_tuple s
r c C sV t |trFt|dkst|d s4tdd | D } |d sR| d } n |sR| d } | S )N c s s | ]}|d V qdS )r Nr ).0r r r r
# s z%_tuple_postprocess..r )r r lenAssertionError)resZ to_unpackr r r _tuple_postprocess s
r c C s` g }xR| D ]J}|r@|j r@|js0|j|j| qT|j|j q
|j|j j| q
W t|S )N)
requires_gradZ is_sparseappendZview_asclonedetachZrequires_grad_r )inputs create_graph
need_graphr r
r r r _grad_preprocess+ s
r# c sF t | d tjr, s&tdd | D S | S nt fdd| D S d S )Nr c s s | ]}|j V qd S )N)r )r r
r r r r G s z$_grad_postprocess..c 3 s | ]}t | V qd S )N)_grad_postprocess)r r
)r! r r r K s )r r r r )r r! r )r! r r$ B s
r$ c C s t |t | kr6|r.tdjt |t | ntdxZtt| |D ]H\}\}}|j |j krFd}|rtdj|}tdj||j |j qFW d S )Nz8v is a tuple of invalid length: should be {} but got {}.z+The given v should contain a single Tensor. z Entry {} in z.{}v has invalid size: should be {} but got {}.)r RuntimeErrorr r zipsize)votherZis_other_tupleidxZel_vZel_otherprependr r r _validate_vM s
r- c C s |sd S |d krt dxt| D ]z\}}|d kr@t dj||js"|dkr^t dj|q"|dkrvt dj|q"|dkrt d j|q"t d
j|q"W d S ) Noutputs grad_inputsjacobianhessianz*Invalid input_type to _check_requires_gradzhThe output of the user-provided function is independent of input {}. This is not allowed in strict mode.zThe hessian of the user-provided function with respect to input {} is independent of the input. This is not allowed in strict mode. You should ensure that your function is thrice differentiable and that the hessian depends on the inputs.aP While computing the hessian, found that the jacobian of the user-provided function with respect to input {} is independent of the input. This is not allowed in strict mode. You should ensure that your function is twice differentiable and that the jacobian depends on the inputs (this would be violated by a linear function for example).zThe gradient with respect to input {} is independent of the inputs of the user-provided function. This is not allowed in strict mode.zOutput {} of the user-provided function does not require gradients. The outputs must be computed in a differentiable manner from the input when running in strict mode.)r. r/ r0 r1 )r& r r r )r Z
input_typestrictr r
r r r _check_requires_grad_ s(
r3 Fc C s t | tst|d kr"dt| }t |ts0tt| t|ksDtt }t }x8t| |D ]*\}}|d k r\|jr\||f7 }||f7 }q\W t|dkrdt| S tjj|||d||dS d S )Nr T)Z allow_unusedr! retain_graph)N)N) r r r r r' r r ZautogradZgrad) r. r grad_outputsr! r4 Z new_outputsZnew_grad_outputsoutZgrad_outr r r _autograd_grad s
r7 c C s |d
krt dj|t }xt| D ]\}}|d kr|r|dkrRt dj|n>|dkrjt dj|n&|dkrt dj|nt d j|tj|| }n6|r|r|j rd
|krt d j|nt d j|||f7 }q&W |S )Nback
back_trick double_backdouble_back_trickz-Invalid stage argument '{}' to _fill_in_zeroszhThe output of the user-provided function is independent of input {}. This is not allowed in strict mode.zThe gradient with respect to the input is independent of entry {} in the grad_outputs when using the double backward trick to compute forward mode gradients. This is not allowed in strict mode.zjThe jacobian of the user-provided function is independent of input {}. This is not allowed in strict mode.zThe hessian of the user-provided function is independent of entry {} in the grad_jacobian. This is not allowed in strict mode as it prevents from using the double backward trick to replace forward mode AD.doublezThe jacobian of the user-provided function is independent of input {}. This is not allowed in strict mode when create_graph=True.zThe hessian of the user-provided function is independent of input {}. This is not allowed in strict mode when create_graph=True.)r8 r9 r: r; )r& r r r r
zeros_liker )Zgradsrefsr2 r! Zstager r Zgrads_ir r r _fill_in_zeros s4
r? c C s t j t|dd\}}t||dd}| | }t|dd\}}t|d|d |dk rt|d d\}}t||d
d}t||| n$t|d ks|d j d krtd
W dQ R X |rdnt j } t j
| & t ||||d}
t |
|||d} W dQ R X t
||}t
| |} t||t| |fS )av
Function that computes the dot product between a vector ``v`` and the
Jacobian of the given function at the point given by the inputs.
Args:
func (function): a Python function that takes Tensor inputs and returns
a tuple of Tensors or a Tensor.
inputs (tuple of Tensors or Tensor): inputs to the function ``func``.
v (tuple of Tensors or Tensor): The vector for which the vector
Jacobian product is computed. Must be the same size as the output
of ``func``. This argument is optional when the output of ``func``
contains a single element and (if it is not provided) will be set
as a Tensor containing a single ``1``.
create_graph (bool, optional): If ``True``, both the output and result
will be computed in a differentiable way. Note that when ``strict``
is ``False``, the result can not require gradients or be
disconnected from the inputs. Defaults to ``False``.
strict (bool, optional): If ``True``, an error will be raised when we
detect that there exists an input such that all the outputs are
independent of it. If ``False``, we return a Tensor of zeros as the
vjp for said inputs, which is the expected mathematical value.
Defaults to ``False``.
Returns:
output (tuple): tuple with:
func_output (tuple of Tensors or Tensor): output of ``func(inputs)``
vjp (tuple of Tensors or Tensor): result of the dot product with
the same shape as the inputs.
Example:
>>> def exp_reducer(x):
... return x.exp().sum(dim=1)
>>> inputs = torch.rand(4, 4)
>>> v = torch.ones(4)
>>> vjp(exp_reducer, inputs, v)
(tensor([5.7817, 7.2458, 5.7830, 6.7782]),
tensor([[1.4458, 1.3962, 1.3042, 1.6354],
[2.1288, 1.0652, 1.5483, 2.5035],
[2.2046, 1.1292, 1.1432, 1.3059],
[1.3225, 1.6652, 1.7753, 2.0152]]))
>>> vjp(exp_reducer, inputs, v, create_graph=True)
(tensor([5.7817, 7.2458, 5.7830, 6.7782], grad_fn=),
tensor([[1.4458, 1.3962, 1.3042, 1.6354],
[2.1288, 1.0652, 1.5483, 2.5035],
[2.2046, 1.1292, 1.1432, 1.3059],
[1.3225, 1.6652, 1.7753, 2.0152]], grad_fn=))
>>> def adder(x, y):
... return 2 * x + 3 * y
>>> inputs = (torch.rand(2), torch.rand(2))
>>> v = torch.ones(2)
>>> vjp(adder, inputs, v)
(tensor([2.4225, 2.3340]),
(tensor([2., 2.]), tensor([3., 3.])))
r vjpT)r! r" z%outputs of the user-provided functionr. )r2 Nr) Fr r zjThe vector v can only be None if the user-provided function returns a single Tensor with a single element.)r! r8 )r enable_gradr r# r3 r- r nelementr& is_grad_enabledset_grad_enabledr7 r? r$ r ) funcr r) r! r2 is_inputs_tupler. is_outputs_tuple_rA grad_resr@ r r r r@ s& ;
r@ c
C sd t j t|dd\}}t||dd}|dk r\t|dd\}}t||dd}t||| n$t|dksx|d j dkrtd
| | }t|d d\}}t|d |d
t dd |D } t
||| dd}
t|
d|d
W dQ R X |r t j & t
|
| ||d} t | |||d} W dQ R X n t
|
| ||d} t | |||d} t ||}t | |} t
||t
| |fS )a$
Function that computes the dot product between the Jacobian of
the given function at the point given by the inputs and a vector ``v``.
Args:
func (function): a Python function that takes Tensor inputs and returns
a tuple of Tensors or a Tensor.
inputs (tuple of Tensors or Tensor): inputs to the function ``func``.
v (tuple of Tensors or Tensor): The vector for which the Jacobian
vector product is computed. Must be the same size as the input of
``func``. This argument is optional when the input to ``func``
contains a single element and (if it is not provided) will be set
as a Tensor containing a single ``1``.
create_graph (bool, optional): If ``True``, both the output and result
will be computed in a differentiable way. Note that when ``strict``
is ``False``, the result can not require gradients or be
disconnected from the inputs. Defaults to ``False``.
strict (bool, optional): If ``True``, an error will be raised when we
detect that there exists an input such that all the outputs are
independent of it. If ``False``, we return a Tensor of zeros as the
jvp for said inputs, which is the expected mathematical value.
Defaults to ``False``.
Returns:
output (tuple): tuple with:
func_output (tuple of Tensors or Tensor): output of ``func(inputs)``
jvp (tuple of Tensors or Tensor): result of the dot product with
the same shape as the output.
Example:
>>> def exp_reducer(x):
... return x.exp().sum(dim=1)
>>> inputs = torch.rand(4, 4)
>>> v = torch.ones(4, 4)
>>> jvp(exp_reducer, inputs, v)
(tensor([6.3090, 4.6742, 7.9114, 8.2106]),
tensor([6.3090, 4.6742, 7.9114, 8.2106]))
>>> jvp(exp_reducer, inputs, v, create_graph=True)
(tensor([6.3090, 4.6742, 7.9114, 8.2106], grad_fn=),
tensor([6.3090, 4.6742, 7.9114, 8.2106], grad_fn=))
>>> def adder(x, y):
... return 2 * x + 3 * y
>>> inputs = (torch.rand(2), torch.rand(2))
>>> v = (torch.ones(2), torch.ones(2))
>>> jvp(adder, inputs, v)
(tensor([2.2399, 2.5005]),
tensor([5., 5.]))
Note:
The jvp is currently computed by using the backward of the backward
(sometimes called the double backwards trick) as we don't have support
for forward mode AD in PyTorch at the moment.
r jvpT)r! r" Nr) Fr r zrThe vector v can only be None if the input to the user-provided function is a single Tensor with a single element.z%outputs of the user-provided functionr. )r2 c s s | ]}t j|d dV qdS )T)r N)r r= )r r6 r r r r i s zjvp..)r! r/ r9 )r rA r r# r- r rB r& r3 r r7 r? r$ r )
rE r r) r! r2 rF rH r. rG r5 r/ rI rJ r r r rJ s0 :
rJ .)tensors
tensor_numelsreturnc s t | t |kstt | dks$tt| dtj|jddd d j j }t fddt | |D }x&t ||D ]\}}|j
|j d qzW |S )Nr )dimr c 3 s | ]\}}|j |V qd S )N)Z new_zeros)r tensorZ tensor_numel) total_numelr r r s z0_construct_standard_basis_for..)r ) r r sumr rO ZcumsumnegZunbindr r' ZdiagonalZfill_)rK rL Zdiag_start_indiceschunkschunkZdiag_start_idxr )rP r _construct_standard_basis_for} s & rV c sz t j f tdd\}t dd| }t|dd\}}t|d|d |r8|rbtdtd d
|D }t||} td d
|D fd d
}
t|
| } g } xbt | D ]T\}
}g }x>> def exp_reducer(x):
... return x.exp().sum(dim=1)
>>> inputs = torch.rand(2, 2)
>>> jacobian(exp_reducer, inputs)
tensor([[[1.4917, 2.4352],
[0.0000, 0.0000]],
[[0.0000, 0.0000],
[2.4369, 2.3799]]])
>>> jacobian(exp_reducer, inputs, create_graph=True)
tensor([[[1.4917, 2.4352],
[0.0000, 0.0000]],
[[0.0000, 0.0000],
[2.4369, 2.3799]]], grad_fn=)
>>> def exp_adder(x, y):
... return 2 * x.exp() + 3 * y
>>> inputs = (torch.rand(2), torch.rand(2))
>>> jacobian(exp_adder, inputs)
(tensor([[2.8052, 0.0000],
[0.0000, 3.3963]]),
tensor([[3., 0.],
[0., 3.]]))
r r0 T)r! r" z%outputs of the user-provided functionr. )r2 ztorch.autograd.functional.jacobian: `strict=True` and `vectorized=True` are not supported together. Please either set `strict=False` or `vectorize=False`.c s s | ]}|j V qd S )N)Znumel)r outputr r r r s zjacobian..c s s | ]}|j dV qdS )r NrQ )reshape)r rW r r r r s c sN t t| d}x0t|D ]$\}}|d k r0q tj| ||< q W t|S )N)r! )listr7 r r r= r )Z grad_outputvjel_idxvj_el)r! flat_outputsr r r r@ s zjacobian..vjpr )rN c s s | ]
}g V qd S )Nr )r rH r r r r A s r )r4 r! NzThe jacobian of the user-provided function is independent of input {}. This is not allowed in strict mode when create_graph=True.zgOutput {} of the user-provided function is independent of input {}. This is not allowed in strict mode.c 3 s6 | ].\}}t j|d djj | j V qdS )r )rN N)r stackviewr( )r r[ jac_i_el)r r6 r r r V s rQ )r rA r r# r3 r& r rV r r' splitr_ shaper r$ r r ranger rB r7 rX r r r= ) rE r r! r2 vectorizerF r. rG Z
output_numelsr5 r@ Zjacobians_of_flat_outputZjacobian_input_outputjacZinput_iZjacobian_input_i_outputZoutput_jZjacobian_input_i_output_jZjacobian_output_inputr0 r Zjac_ijrZ r[ r` r\ Zinp_elmsgr )r! r] r r6 r r0 s` C
-
r0 c sL t |dd\}}fdd fdd}t||||d}t|||fS )a
Function that computes the Hessian of a given scalar function.
Args:
func (function): a Python function that takes Tensor inputs and returns
a Tensor with a single element.
inputs (tuple of Tensors or Tensor): inputs to the function ``func``.
create_graph (bool, optional): If ``True``, the Hessian will be computed in
a differentiable manner. Note that when ``strict`` is ``False``, the result can not
require gradients or be disconnected from the inputs.
Defaults to ``False``.
strict (bool, optional): If ``True``, an error will be raised when we detect that there exists an input
such that all the outputs are independent of it. If ``False``, we return a Tensor of zeros as the
hessian for said inputs, which is the expected mathematical value.
Defaults to ``False``.
vectorize (bool, optional): This feature is experimental, please use at
your own risk. When computing the hessian, usually we invoke
``autograd.grad`` once per row of the hessian. If this flag is
``True``, we use the vmap prototype feature as the backend to
vectorize calls to ``autograd.grad`` so we only invoke it once
instead of once per row. This should lead to performance
improvements in many use cases, however, due to this feature
being incomplete, there may be performance cliffs. Please
use `torch._C._debug_only_display_vmap_fallback_warnings(True)`
to show any performance warnings and file us issues if
warnings exist for your use case. Defaults to ``False``.
Returns:
Hessian (Tensor or a tuple of tuple of Tensors): if there is a single input,
this will be a single Tensor containing the Hessian for the input.
If it is a tuple, then the Hessian will be a tuple of tuples where
``Hessian[i][j]`` will contain the Hessian of the ``i``\th input
and ``j``\th input with size the sum of the size of the ``i``\th input plus
the size of the ``j``\th input. ``Hessian[i][j]`` will have the same
dtype and device as the corresponding ``i``\th input.
Example:
>>> def pow_reducer(x):
... return x.pow(3).sum()
>>> inputs = torch.rand(2, 2)
>>> hessian(pow_reducer, inputs)
tensor([[[[5.2265, 0.0000],
[0.0000, 0.0000]],
[[0.0000, 4.8221],
[0.0000, 0.0000]]],
[[[0.0000, 0.0000],
[1.9456, 0.0000]],
[[0.0000, 0.0000],
[0.0000, 3.2550]]]])
>>> hessian(pow_reducer, inputs, create_graph=True)
tensor([[[[5.2265, 0.0000],
[0.0000, 0.0000]],
[[0.0000, 4.8221],
[0.0000, 0.0000]]],
[[[0.0000, 0.0000],
[1.9456, 0.0000]],
[[0.0000, 0.0000],
[0.0000, 3.2550]]]], grad_fn=)
>>> def pow_adder_reducer(x, y):
... return (2 * x.pow(2) + 3 * y.pow(2)).sum()
>>> inputs = (torch.rand(2), torch.rand(2))
>>> hessian(pow_adder_reducer, inputs)
((tensor([[4., 0.],
[0., 4.]]),
tensor([[0., 0.],
[0., 0.]])),
(tensor([[0., 0.],
[0., 0.]]),
tensor([[6., 0.],
[0., 6.]])))
r r1 c s\ | }t |dd\}}t|dd |s8t|tj r@td|j dkrTtd|j S )Nz%outputs of the user-provided functionr1 r. )r2 z;The function given to hessian should return a single Tensorr zTThe Tensor returned by the function given to hessian should contain a single element)r r3 r r r r& rB Zsqueeze)r
r6 Z is_out_tupleZt_out)rE r2 r r ensure_single_output_function s z.hessian..ensure_single_output_functionc s t | dd}t|dd |S )NT)r! r0 )r2 )r0 r3 )r
re )rh r2 r r jac_func s zhessian..jac_func)r! r2 rd )r r0 r )rE r r! r2 rd rF ri r r )rh rE r2 r r1 ] s
L
r1 c
C sr t j t|dd\}}t||dd}|dk r\t|dd\}}t||dd}t||| n$t|dksx|d j dkrtd
| | }t|d d\}}t|d |d
|st |d t j
rtd|d j dkrtdt ||dd} t| d|d
W dQ R X |r dnt j }
t j
|
& t | |||d} t| |||d} W dQ R X t||}t| |} t||t| |fS )a Function that computes the dot product between a vector ``v`` and the
Hessian of a given scalar function at the point given by the inputs.
Args:
func (function): a Python function that takes Tensor inputs and returns
a Tensor with a single element.
inputs (tuple of Tensors or Tensor): inputs to the function ``func``.
v (tuple of Tensors or Tensor): The vector for which the vector Hessian
product is computed. Must be the same size as the input of
``func``. This argument is optional when ``func``'s input contains
a single element and (if it is not provided) will be set as a
Tensor containing a single ``1``.
create_graph (bool, optional): If ``True``, both the output and result
will be computed in a differentiable way. Note that when ``strict``
is ``False``, the result can not require gradients or be
disconnected from the inputs.
Defaults to ``False``.
strict (bool, optional): If ``True``, an error will be raised when we
detect that there exists an input such that all the outputs are
independent of it. If ``False``, we return a Tensor of zeros as the
vhp for said inputs, which is the expected mathematical value.
Defaults to ``False``.
Returns:
output (tuple): tuple with:
func_output (tuple of Tensors or Tensor): output of ``func(inputs)``
vhp (tuple of Tensors or Tensor): result of the dot product with the
same shape as the inputs.
Example:
>>> def pow_reducer(x):
... return x.pow(3).sum()
>>> inputs = torch.rand(2, 2)
>>> v = torch.ones(2, 2)
>>> vhp(pow_reducer, inputs, v)
(tensor(0.5591),
tensor([[1.0689, 1.2431],
[3.0989, 4.4456]]))
>>> vhp(pow_reducer, inputs, v, create_graph=True)
(tensor(0.5591, grad_fn=),
tensor([[1.0689, 1.2431],
[3.0989, 4.4456]], grad_fn=))
>>> def pow_adder_reducer(x, y):
... return (2 * x.pow(2) + 3 * y.pow(2)).sum()
>>> inputs = (torch.rand(2), torch.rand(2))
>>> v = (torch.zeros(2), torch.ones(2))
>>> vhp(pow_adder_reducer, inputs, v)
(tensor(4.8053),
(tensor([0., 0.]),
tensor([6., 6.])))
r vhpT)r! r" Nr) Fr r zrThe vector v can only be None if the input to the user-provided function is a single Tensor with a single element.z%outputs of the user-provided functionr. )r2 z7The function given to vhp should return a single TensorzPThe Tensor returned by the function given to vhp should contain a single element)r! r0 r: )r rA r r# r- r rB r& r3 r r r7 rC rD r? r$ r )
rE r r) r! r2 rF rH r. rG re rA rI rj r r r rj s2 7
rj c C s t j $ t|dd\}}t||dd}|dk r^t|dd\}}t||dd}t||| n$t|dksz|d j dkrtd
| | }t|d d\}}t|d |d
|st |d t j
rtd|d j dkrtdt ||dd} t| d|d
t dd |D }
t | ||
dd} t| d|d
W dQ R X |r>dnt j
} t j| & t | |
||d}
t|
|||d}W dQ R X t||}t||}t||t||fS )a
Function that computes the dot product between the Hessian of a given scalar
function and a vector ``v`` at the point given by the inputs.
Args:
func (function): a Python function that takes Tensor inputs and returns
a Tensor with a single element.
inputs (tuple of Tensors or Tensor): inputs to the function ``func``.
v (tuple of Tensors or Tensor): The vector for which the Hessian vector
product is computed. Must be the same size as the input of
``func``. This argument is optional when ``func``'s input contains
a single element and (if it is not provided) will be set as a
Tensor containing a single ``1``.
create_graph (bool, optional): If ``True``, both the output and result will be
computed in a differentiable way. Note that when ``strict`` is
``False``, the result can not require gradients or be disconnected
from the inputs. Defaults to ``False``.
strict (bool, optional): If ``True``, an error will be raised when we
detect that there exists an input such that all the outputs are
independent of it. If ``False``, we return a Tensor of zeros as the
hvp for said inputs, which is the expected mathematical value.
Defaults to ``False``.
Returns:
output (tuple): tuple with:
func_output (tuple of Tensors or Tensor): output of ``func(inputs)``
hvp (tuple of Tensors or Tensor): result of the dot product with
the same shape as the inputs.
Example:
>>> def pow_reducer(x):
... return x.pow(3).sum()
>>> inputs = torch.rand(2, 2)
>>> v = torch.ones(2, 2)
>>> hvp(pow_reducer, inputs, v)
(tensor(0.1448),
tensor([[2.0239, 1.6456],
[2.4988, 1.4310]]))
>>> hvp(pow_reducer, inputs, v, create_graph=True)
(tensor(0.1448, grad_fn=),
tensor([[2.0239, 1.6456],
[2.4988, 1.4310]], grad_fn=))
>>> def pow_adder_reducer(x, y):
... return (2 * x.pow(2) + 3 * y.pow(2)).sum()
>>> inputs = (torch.rand(2), torch.rand(2))
>>> v = (torch.zeros(2), torch.ones(2))
>>> hvp(pow_adder_reducer, inputs, v)
(tensor(2.3030),
(tensor([0., 0.]),
tensor([6., 6.])))
Note:
This function is significantly slower than `vhp` due to backward mode AD constraints.
If your functions is twice continuously differentiable, then hvp = vhp.t(). So if you
know that your function satisfies this condition, you should use vhp instead that is
much faster with the current implementation.
r hvpT)r! r" Nr) Fr r zrThe vector v can only be None if the input to the user-provided function is a single Tensor with a single element.z%outputs of the user-provided functionr. )r2 z7The function given to hvp should return a single TensorzPThe Tensor returned by the function given to hvp should contain a single element)r! r0 c s s | ]}t j|d dV qdS )T)r N)r r= )r r
r r r r u s zhvp..r1 r; )r rA r r# r- r rB r& r3 r r r7 r rC rD r? r$ r )rE r r) r! r2 rF rH r. rG re Zgrad_jacr: rA rI rk r r r rk s8 @
rk )NFN)NFF)NFF)FFF)FFF)NFF)NFF)r typingr r Ztorch._vmap_internalsr r r r# r$ r- r3 r7 r? r@ rJ r intrV r0 r1 rj rk r r r r s$
,
Y
b.
C
d
[