Name	Size	Mode	Actions
adadelta.cpython-36.pyc	4829	0644	editdlrm
adagrad.cpython-36.pyc	4767	0644	editdlrm
adam.cpython-36.pyc	6250	0644	editdlrm
adamax.cpython-36.pyc	4869	0644	editdlrm
adamw.cpython-36.pyc	6258	0644	editdlrm
asgd.cpython-36.pyc	2848	0644	editdlrm
lbfgs.cpython-36.pyc	8802	0644	editdlrm
lr_scheduler.cpython-36.pyc	63792	0644	editdlrm
nadam.cpython-36.pyc	5727	0644	editdlrm
optimizer.cpython-36.pyc	11827	0644	editdlrm
radam.cpython-36.pyc	5712	0644	editdlrm
rmsprop.cpython-36.pyc	6543	0644	editdlrm
rprop.cpython-36.pyc	5182	0644	editdlrm
sgd.cpython-36.pyc	6267	0644	editdlrm
sparse_adam.cpython-36.pyc	3329	0644	editdlrm
swa_utils.cpython-36.pyc	11370	0644	editdlrm
_functional.cpython-36.pyc	10407	0644	editdlrm
__init__.cpython-36.pyc	1049	0644	editdlrm

3 �Eg��@s4dd
lZddl
mZ
ddlmZ
Gdd�de�Zd
S)�N�
)
�_functional)
� Optimizercs<eZ
dZd
Zd�f
dd� Zdd�Zej�d d d�
�
Z�Z S)�Adagrada� Implements Adagrad algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)}, \: \lambda \text{ (weight decay)}, \\ &\hspace{12mm} \tau \text{ (initial accumulator value)}, \: \eta\text{ (lr decay)}\\ &\textbf{initialize} : state\_sum_0 \leftarrow 0 \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm} \tilde{\gamma} \leftarrow \gamma / (1 +(t-1) \eta) \\ &\hspace{5mm} \textbf{if} \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}state\_sum_t \leftarrow state\_sum_{t-1} + g^2_t \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1}- \tilde{\gamma} \frac{g_t}{\sqrt{state\_sum_t}+\epsilon} \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Adaptive Subgradient Methods for Online Learning and Stochastic Optimization`_. Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-2) lr_decay (float, optional): learning rate decay (default: 0) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-10) .. _Adaptive Subgradient Methods for Online Learning and Stochastic Optimization: http://jmlr.org/papers/v12/duchi11a.html �{�G�z�?r
绽���|�=cs�d
|k
stdj
|�
�
�
d
|k
s,tdj
|�
�
�
d
|k
sBtdj
|�
�
�
d
|k
sXtdj
|�
�
�
d
|k
sntdj
|�
�
�
t|||||d�}tt|�j|
|�
xH|jD]>}x8|dD],} |j| } d | d <tj | |tj d�| d<q�Wq�WdS) NgzInvalid learning rate: {}zInvalid lr_decay value: {}zInvalid weight_decay value: {}z+Invalid initial_accumulator_value value: {}zInvalid epsilon value: {})�lr�lr_decay�eps�weight_decay�initial_accumulator_value�paramsr
�step)
Z memory_format�sum)� ValueError�format�dict�superr�__init__�param_groups�state�torchZ full_likeZpreserve_format)�selfr rr rrr �defaults�group�
pr)
� __class__��?/usr/local/lib64/python3.6/site-packages/torch/optim/adagrad.pyr.s$















zAdagrad.__init__c
Cs<x6|jD],}
x&|
d
D]}|j
|}|dj�
qWqWdS)Nr r)rrZ share_memory_)rrrrrrr�share_memoryDs



zAdagrad.share_memoryNc Cs�d
}|
d
k r&tj
��
|
�}Wd
QRXx�|jD]�}g}g}g}g}xf|dD]Z}|jd
k rL|j|�

|j|j�

|j|} |j| d�

| dd7<|j| d�

qLWtj|||||d|d|d|d d �
q.W|S)z�Performs a single optimization step. Args: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nr rrrrrr r )rrr r )rZenable_gradrZgrad�appendr�
FZadagrad) rZclosureZlossrZparams_with_gradZgradsZ state_sumsZstate_stepsrrrrrrJs4


















zAdagrad.step)rr
r
r
r)
N) �__name__� __module__�__qualname__�__doc__rrrZno_gradr� __classcell__rr)
rrrs &
r)r�rr!Z optimizerrrrrrr�
s