/usr/local/lib64/python3.6/site-packages/torch/optim/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/optim/__pycache__/rmsprop.cpython-36.pyc (6543B)
3
Eg @ s4 d dl Z ddlmZ ddlmZ G dd deZdS ) N ) _functional) Optimizerc s@ e Zd ZdZd fdd Z fd d
Zej dd d
Z Z S )RMSpropa Implements RMSprop algorithm.
.. math::
\begin{aligned}
&\rule{110mm}{0.4pt} \\
&\textbf{input} : \alpha \text{ (alpha)},\: \gamma \text{ (lr)},
\: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)} \\
&\hspace{13mm} \lambda \text{ (weight decay)},\: \mu \text{ (momentum)},\: centered\\
&\textbf{initialize} : v_0 \leftarrow 0 \text{ (square average)}, \:
\textbf{b}_0 \leftarrow 0 \text{ (buffer)}, \: g^{ave}_0 \leftarrow 0 \\[-1.ex]
&\rule{110mm}{0.4pt} \\
&\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\
&\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\
&\hspace{5mm}if \: \lambda \neq 0 \\
&\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\
&\hspace{5mm}v_t \leftarrow \alpha v_{t-1} + (1 - \alpha) g^2_t
\hspace{8mm} \\
&\hspace{5mm} \tilde{v_t} \leftarrow v_t \\
&\hspace{5mm}if \: centered \\
&\hspace{10mm} g^{ave}_t \leftarrow g^{ave}_{t-1} \alpha + (1-\alpha) g_t \\
&\hspace{10mm} \tilde{v_t} \leftarrow \tilde{v_t} - \big(g^{ave}_{t} \big)^2 \\
&\hspace{5mm}if \: \mu > 0 \\
&\hspace{10mm} \textbf{b}_t\leftarrow \mu \textbf{b}_{t-1} +
g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \\
&\hspace{10mm} \theta_t \leftarrow \theta_{t-1} - \gamma \textbf{b}_t \\
&\hspace{5mm} else \\
&\hspace{10mm}\theta_t \leftarrow \theta_{t-1} -
\gamma g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \hspace{3mm} \\
&\rule{110mm}{0.4pt} \\[-1.ex]
&\bf{return} \: \theta_t \\[-1.ex]
&\rule{110mm}{0.4pt} \\[-1.ex]
\end{aligned}
For further details regarding the algorithm we refer to
`lecture notes
`_ by G. Hinton.
and centered version `Generating Sequences
With Recurrent Neural Networks `_.
The implementation here takes the square root of the gradient average before
adding epsilon (note that TensorFlow interchanges these two operations). The effective
learning rate is thus :math:`\gamma/(\sqrt{v} + \epsilon)` where :math:`\gamma`
is the scheduled learning rate and :math:`v` is the weighted moving average
of the squared gradient.
Args:
params (iterable): iterable of parameters to optimize or dicts defining
parameter groups
lr (float, optional): learning rate (default: 1e-2)
momentum (float, optional): momentum factor (default: 0)
alpha (float, optional): smoothing constant (default: 0.99)
eps (float, optional): term added to the denominator to improve
numerical stability (default: 1e-8)
centered (bool, optional) : if ``True``, compute the centered RMSProp,
the gradient is normalized by an estimation of its variance
weight_decay (float, optional): weight decay (L2 penalty) (default: 0)
{Gz?Gz?:0yE>r Fc s d|kst dj|d|ks,t dj|d|ksBt dj|d|ksXt dj|d|ksnt dj|t||||||d}tt| j|| d S )Ng zInvalid learning rate: {}zInvalid epsilon value: {}zInvalid momentum value: {}z Invalid weight_decay value: {}zInvalid alpha value: {})lrmomentumalphaepscentered weight_decay)
ValueErrorformatdictsuperr __init__) selfparamsr r r r r
r
defaults) __class__ ?/usr/local/lib64/python3.6/site-packages/torch/optim/rmsprop.pyr @ s zRMSprop.__init__c s< t t| j| x&| jD ] }|jdd |jdd qW d S )Nr
r r
F)r r __setstate__ param_groups
setdefault)r stategroup)r r r r O s zRMSprop.__setstate__Nc C s d}|dk r&t j | }W dQ R X xb| jD ]V}g }g }g }g }g }x|d D ]} | jdkrfqV|j| | jjrtd|j| j | j| }
t|
dkrd|
d< t j | t j
d|
d< |d dkrt j | t j
d|
d < |d
rt j | t j
d|
d < |j|
d |d dkr$|j|
d |d
r<|j|
d |
d d 7 < qVW t j ||||||d
|d |d |d |d |d
d q0W |S )zPerforms a single optimization step.
Args:
closure (callable, optional): A closure that reevaluates the model
and returns the loss.
Nr z)RMSprop does not support sparse gradientsr step)Z
memory_formatZ
square_avgr
Zmomentum_bufferr
Zgrad_avgr r r r r )r r r r r
r
)
torchZ enable_gradr ZgradappendZ is_sparse RuntimeErrorr lenZ
zeros_likeZpreserve_formatFZrmsprop) r ZclosureZlossr Zparams_with_gradZgradsZ square_avgsZ grad_avgsZmomentum_buffer_listpr r r r r U sV
z RMSprop.step)r r r r r F)N)
__name__
__module__ __qualname____doc__r r r Zno_gradr
__classcell__r r )r r r s
8 r )r r r$ Z optimizerr r r r r r s