/usr/local/lib64/python3.6/site-packages/torch/nn/utils/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/nn/utils/__pycache__/parametrizations.cpython-36.pyc (14999B)
3
EgW @ s d dl mZmZ d dlZd dlmZ ddlmZ ddlmZ ddl m
Z d dl m
Z
d d d
Zd d ZG d
d deZG dd deZd ddeee
e eedddZG dd deZd eeeee
e edd d ZdS )! )EnumautoN)Tensor ) parametrize)Module)
functional)Optionalc C s` | j d| j d }}tj|| j| jd}d| tj| jj }tj| jdd j | ||dS )
Nr )dtypedeviceg $@)Zatolr
r )
sizetorcheyer r ZfinfoepsZallclose transposeconj)Qr nkId r K/usr/local/lib64/python3.6/site-packages/torch/nn/utils/parametrizations.py_is_orthogonal s r c C s< t j| \}}t jj||}||jdddj jd9 }|S )z Assume that A is a tall matrix.
Compute the Q factor s.t. A = QR (A may be complex) and diag(R) is real and non-negative
r r
)dim1dim2r
r r
)r geqrflinalghouseholder_productdiagonalZsgn unsqueeze)AXtaur r r r _make_orthogonal s r& c @ s e Zd Ze Ze Ze ZdS ) _OrthMapsN)__name__
__module__ __qualname__r
matrix_expcayley householderr r r r r' s r' c sb e Zd ZU eddedd fddZejejddd Zej j
ejejd
d d Z Z S )
_OrthogonalT)use_trivializationN)orthogonal_mapreturnc sF t j |j r$|tjkr$td|j| _|| _|rB| jdd d S )NzAThe householder parametrization does not support complex tensors.base) super__init__
is_complexr' r-
ValueErrorshaper0 register_buffer)selfweightr0 r/ ) __class__r r r4 ( s
z_Orthogonal.__init__)r$ r1 c C s |j d
|j d }}||k }|r8|jdd}|| }}| jtjksR| jtjkr$|j }||krtj||j ||| j
|j d d d gdd}||jddj }| jtjkrtj|}nH| jtjkrtj
||j|jd}tjjtj||ddtj||dd}||krr|dd |f }nN|jdd}d d
|| jdd }tjj||}||jdd d j jd }t| d r| j| }|r|jd d }|S ) Nr r
)dim)r r g ?)alpha.)r! g @g ?)r r r2 r
r r
r r
r r )r r r r
r g r r
r
r r
r
r )r r r0 r' r+ r, Ztrilr catZ new_zerosexpandr7 r r r r r Zsolveaddsumr r! intr" hasattrr2 ) r9 r$ r r Z
transposedr# r r r% r r r forwardC s4
2 &
z_Orthogonal.forward)r r1 c
C sz |j | j kr&td| j d|j d|}|jd
|jd }}||k }|rb|jdd}|| }}t| ds| jtjks| jtjkrt dt
j |\}}|j dddj
|j ddd|d k d9 < |r|jddS |S ||krt| st|}n|j }nHt
j|j d d ||| f |j|jd
}t
j||gdd }t|}|| _t
j|} | j dddjd | S d S ) Nz0Expected a matrix or batch of matrices of shape z. Got a tensor of shape .r r
r2 ztIt is not possible to assign to the matrix exponential or the Cayley parametrizations when use_trivialization=False.)r r g )r r )r< g ?r
r r
r r
r r
r r r
r r
r r
r g )r7 r6 r r rC r0 r' r, r+ NotImplementedErrorr r r! Zsign_r r& cloneZrandnr r r> r2 Z
zeros_likeZfill_)
r9 r ZQ_initr r r r# r% NZneg_Idr r r
right_inversek s4
.
z_Orthogonal.right_inverse)
r( r) r* r r2 r' r4 r rD autogradno_gradrI
__classcell__r r )r; r r. % s
(r. r: T)r/ )modulenamer0 r/ r1 c C s t | |d}t|ts&tdj| ||jdk rBtd|j d|dkrn|jd
|jdksf|j rjdnd}t t|d}|dkrtd | t |||d
}t
j | ||d d | S )a Applies an orthogonal or unitary parametrization to a matrix or a batch of matrices.
Letting :math:`\mathbb{K}` be :math:`\mathbb{R}` or :math:`\mathbb{C}`, the parametrized
matrix :math:`Q \in \mathbb{K}^{m \times n}` is **orthogonal** as
.. math::
\begin{align*}
Q^{\text{H}}Q &= \mathrm{I}_n \mathrlap{\qquad \text{if }m \geq n}\\
QQ^{\text{H}} &= \mathrm{I}_m \mathrlap{\qquad \text{if }m < n}
\end{align*}
where :math:`Q^{\text{H}}` is the conjugate transpose when :math:`Q` is complex
and the transpose when :math:`Q` is real-valued, and
:math:`\mathrm{I}_n` is the `n`-dimensional identity matrix.
In plain words, :math:`Q` will have orthonormal columns whenever :math:`m \geq n`
and orthonormal rows otherwise.
If the tensor has more than two dimensions, we consider it as a batch of matrices of shape `(..., m, n)`.
The matrix :math:`Q` may be parametrized via three different ``orthogonal_map`` in terms of the original tensor:
- ``"matrix_exp"``/``"cayley"``:
the :func:`~torch.matrix_exp` :math:`Q = \exp(A)` and the `Cayley map`_
:math:`Q = (\mathrm{I}_n + A/2)(\mathrm{I}_n - A/2)^{-1}` are applied to a skew-symmetric
:math:`A` to give an orthogonal matrix.
- ``"householder"``: computes a product of Householder reflectors
(:func:`~torch.linalg.householder_product`).
``"matrix_exp"``/``"cayley"`` often make the parametrized weight converge faster than
``"householder"``, but they are slower to compute for very thin or very wide matrices.
If ``use_trivialization=True`` (default), the parametrization implements the "Dynamic Trivialization Framework",
where an extra matrix :math:`B \in \mathbb{K}^{n \times n}` is stored under
``module.parametrizations.weight[0].base``. This helps the
convergence of the parametrized layer at the expense of some extra memory use.
See `Trivializations for Gradient-Based Optimization on Manifolds`_ .
Initial value of :math:`Q`:
If the original tensor is not parametrized and ``use_trivialization=True`` (default), the initial value
of :math:`Q` is that of the original tensor if it is orthogonal (or unitary in the complex case)
and it is orthogonalized via the QR decomposition otherwise (see :func:`torch.linalg.qr`).
Same happens when it is not parametrized and ``orthogonal_map="householder"`` even when ``use_trivialization=False``.
Otherwise, the initial value is the result of the composition of all the registered
parametrizations applied to the original tensor.
.. note::
This function is implemented using the parametrization functionality
in :func:`~torch.nn.utils.parametrize.register_parametrization`.
.. _`Cayley map`: https://en.wikipedia.org/wiki/Cayley_transform#Matrix_map
.. _`Trivializations for Gradient-Based Optimization on Manifolds`: https://arxiv.org/abs/1909.09501
Args:
module (nn.Module): module on which to register the parametrization.
name (str, optional): name of the tensor to make orthogonal. Default: ``"weight"``.
orthogonal_map (str, optional): One of the following: ``"matrix_exp"``, ``"cayley"``, ``"householder"``.
Default: ``"matrix_exp"`` if the matrix is square or complex, ``"householder"`` otherwise.
use_trivialization (bool, optional): whether to use the dynamic trivialization framework.
Default: ``True``.
Returns:
The original module with an orthogonal parametrization registered to the specified
weight
Example::
>>> orth_linear = orthogonal(nn.Linear(20, 40))
>>> orth_linear
ParametrizedLinear(
in_features=20, out_features=40, bias=True
(parametrizations): ModuleDict(
(weight): ParametrizationList(
(0): _Orthogonal()
)
)
)
>>> Q = orth_linear.weight
>>> torch.dist(Q.T @ Q, torch.eye(20))
tensor(4.9332e-07)
Nz5Module '{}' has no parameter ot buffer with name '{}'r z8Expected a matrix or batch of matrices. Got a tensor of z dimensions.r
r+ r- zLorthogonal_map has to be one of "matrix_exp", "cayley", "householder". Got: )r/ T)Zunsafer
r ) getattr
isinstancer r6 formatndimr r5 r' r. r register_parametrization)rM rN r0 r/ r: Z orth_enumZorthr r r
orthogonal s W
$ rT c s e Zd Zdejeeedd fdd
Zejejdd d
Zej j
ejedd d d
Z ejejdddZ ejejdddZ
ZS )
_SpectralNormr
r -q=N)r: n_power_iterationsr< r r1 c s t j |j}||ks"|| k rBtd| d|d d| d|dkrXtdj||dkrd|n|| | _|| _|dkr|| _| j |}|j
\}}|j |j dd} |j |j dd}
| j
dtj| d| jd | j
d
tj|
d| jd | j|d d S ) Nz5Dimension out of range (expected to be in range of [-z, r
z
] but got )r zIExpected n_power_iterations to be positive, but got n_power_iterations={}_u)r< r _v )r3 r4 rR
IndexErrorr6 rQ r< r rW _reshape_weight_to_matrixr Z new_emptyZnormal_r8 F normalize
_power_method) r9 r: rW r< r rR
weight_mathwuv)r; r r r4 s$
z_SpectralNorm.__init__)r: r1 c sJ |j dkst jdkr@|j jf fddt|j D }|jdS )Nr
r c 3 s | ]}| j kr|V qd S )N)r< ).0d)r9 r r
D s z:_SpectralNorm._reshape_weight_to_matrix..)rR AssertionErrorr< Zpermuterangeflatten)r9 r: r )r9 r r] > s
(z'_SpectralNorm._reshape_weight_to_matrix)ra rW r1 c C sl |j dkstxXt|D ]L}tjtj|| jd| j| j d| _ tjtj|j
| j d| j| jd| _qW d S )Nr
r )r< r out) rR ri rj r^ r_ r mvrZ r rY t)r9 ra rW _r r r r` H s "z_SpectralNorm._power_methodc C sz |j dkr tj|d| jdS | j|}| jr:| j|| j | jj t
j d}| j j t
j d}t
j
|t
j||}|| S d S )Nr
r )r< r )Z
memory_format)rR r^ r_ r r] Ztrainingr` rW rY rG r Zcontiguous_formatrZ dotrm )r9 r: ra rd re sigmar r r rD u s
z_SpectralNorm.forward)valuer1 c C s |S )Nr )r9 rr r r r rI s z_SpectralNorm.right_inverse)r
r rV )r( r) r* r r rB floatr4 r] rJ rK r` rD rI rL r r )r; r rU s
,rU r
-q=)rM rN rW r r< r1 c C sp t | |d}t|ts&tdj| ||dkrTt| tjjtjjtjj frPd}nd}t
j | |t |||| | S )a Applies spectral normalization to a parameter in the given module.
.. math::
\mathbf{W}_{SN} = \dfrac{\mathbf{W}}{\sigma(\mathbf{W})},
\sigma(\mathbf{W}) = \max_{\mathbf{h}: \mathbf{h} \ne 0} \dfrac{\|\mathbf{W} \mathbf{h}\|_2}{\|\mathbf{h}\|_2}
When applied on a vector, it simplifies to
.. math::
\mathbf{x}_{SN} = \dfrac{\mathbf{x}}{\|\mathbf{x}\|_2}
Spectral normalization stabilizes the training of discriminators (critics)
in Generative Adversarial Networks (GANs) by reducing the Lipschitz constant
of the model. :math:`\sigma` is approximated performing one iteration of the
`power method`_ every time the weight is accessed. If the dimension of the
weight tensor is greater than 2, it is reshaped to 2D in power iteration
method to get spectral norm.
See `Spectral Normalization for Generative Adversarial Networks`_ .
.. _`power method`: https://en.wikipedia.org/wiki/Power_iteration
.. _`Spectral Normalization for Generative Adversarial Networks`: https://arxiv.org/abs/1802.05957
.. note::
This function is implemented using the parametrization functionality
in :func:`~torch.nn.utils.parametrize.register_parametrization`. It is a
reimplementation of :func:`torch.nn.utils.spectral_norm`.
.. note::
When this constraint is registered, the singular vectors associated to the largest
singular value are estimated rather than sampled at random. These are then updated
performing :attr:`n_power_iterations` of the `power method`_ whenever the tensor
is accessed with the module on `training` mode.
.. note::
If the `_SpectralNorm` module, i.e., `module.parametrization.weight[idx]`,
is in training mode on removal, it will perform another power iteration.
If you'd like to avoid this iteration, set the module to eval mode
before its removal.
Args:
module (nn.Module): containing module
name (str, optional): name of weight parameter. Default: ``"weight"``.
n_power_iterations (int, optional): number of power iterations to
calculate spectral norm. Default: ``1``.
eps (float, optional): epsilon for numerical stability in
calculating norms. Default: ``1e-12``.
dim (int, optional): dimension corresponding to number of outputs.
Default: ``0``, except for modules that are instances of
ConvTranspose{1,2,3}d, when it is ``1``
Returns:
The original module with a new parametrization registered to the specified
weight
Example::
>>> snm = spectral_norm(nn.Linear(20, 40))
>>> snm
ParametrizedLinear(
in_features=20, out_features=40, bias=True
(parametrizations): ModuleDict(
(weight): ParametrizationList(
(0): _SpectralNorm()
)
)
)
>>> torch.linalg.matrix_norm(snm.weight, 2)
tensor(1.0000, grad_fn=)
Nz5Module '{}' has no parameter or buffer with name '{}'r
r )
rO rP r r6 rQ r nnZConvTranspose1dZConvTranspose2dZConvTranspose3dr rS rU )rM rN rW r r< r: r r r
spectral_norm s L
rv )N)r: N)r: r
rt N)enumr r r r utilsr modulesr r r^ typingr r r& r' r. strboolrT rU rB rs rv r r r r s&
mq