Name	Size	Mode	Actions
adadelta.cpython-36.pyc	4829	0644	editdlrm
adagrad.cpython-36.pyc	4767	0644	editdlrm
adam.cpython-36.pyc	6250	0644	editdlrm
adamax.cpython-36.pyc	4869	0644	editdlrm
adamw.cpython-36.pyc	6258	0644	editdlrm
asgd.cpython-36.pyc	2848	0644	editdlrm
lbfgs.cpython-36.pyc	8802	0644	editdlrm
lr_scheduler.cpython-36.pyc	63792	0644	editdlrm
nadam.cpython-36.pyc	5727	0644	editdlrm
optimizer.cpython-36.pyc	11827	0644	editdlrm
radam.cpython-36.pyc	5712	0644	editdlrm
rmsprop.cpython-36.pyc	6543	0644	editdlrm
rprop.cpython-36.pyc	5182	0644	editdlrm
sgd.cpython-36.pyc	6267	0644	editdlrm
sparse_adam.cpython-36.pyc	3329	0644	editdlrm
swa_utils.cpython-36.pyc	11370	0644	editdlrm
_functional.cpython-36.pyc	10407	0644	editdlrm
__init__.cpython-36.pyc	1049	0644	editdlrm

3 �Eg"�@s4dd
lZddl
mZ
ddlmZ
Gdd�de�Zd
S)�N�
)
�_functional)
� Optimizercs4eZ
dZd
Zd�f
dd� Zej�dd d�
�
Z�ZS)�RpropaGImplements the resilient backpropagation algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \theta_0 \in \mathbf{R}^d \text{ (params)},f(\theta) \text{ (objective)}, \\ &\hspace{13mm} \eta_{+/-} \text{ (etaplus, etaminus)}, \Gamma_{max/min} \text{ (step sizes)} \\ &\textbf{initialize} : g^0_{prev} \leftarrow 0, \: \eta_0 \leftarrow \text{lr (learning rate)} \\ &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm} \textbf{for} \text{ } i = 0, 1, \ldots, d-1 \: \mathbf{do} \\ &\hspace{10mm} \textbf{if} \: g^i_{prev} g^i_t > 0 \\ &\hspace{15mm} \eta^i_t \leftarrow \mathrm{min}(\eta^i_{t-1} \eta_{+}, \Gamma_{max}) \\ &\hspace{10mm} \textbf{else if} \: g^i_{prev} g^i_t < 0 \\ &\hspace{15mm} \eta^i_t \leftarrow \mathrm{max}(\eta^i_{t-1} \eta_{-}, \Gamma_{min}) \\ &\hspace{10mm} \textbf{else} \: \\ &\hspace{15mm} \eta^i_t \leftarrow \eta^i_{t-1} \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1}- \eta_t \mathrm{sign}(g_t) \\ &\hspace{5mm}g_{prev} \leftarrow g_t \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to the paper `A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm `_. Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-2) etas (Tuple[float, float], optional): pair of (etaminus, etaplis), that are multiplicative increase and decrease factors (default: (0.5, 1.2)) step_sizes (Tuple[float, float], optional): a pair of minimal and maximal allowed step sizes (default: (1e-6, 50)) �{�G�z�?��?�333333�?�����ư>�2cs|d
|k
stdj
|�
�
�
d
|dko:dko:|dkn
sXtdj
|d|d��
�
t|||d�}tt|�j|
|�
dS)NgzInvalid learning rate: {}r
g�?rzInvalid eta values: {}, {})�lr�etas� step_sizes)� ValueError�format�dict�superr�__init__)�self�paramsrrr �defaults)
� __class__��=/usr/local/lib64/python3.6/site-packages/torch/optim/rprop.pyr4s


*

zRprop.__init__NcCs@
d
}|
d
k r&tj
��
|
�}Wd
QRX�
x|jD�
]}g}g}g}g}x�|dD]�}|jd
kr`qP|j|�

|j} | jr~td�
�
|j| �

|j|} t| �
dkr�d| d<tj |tj d�| d<| j�j| �
j |d�
| d <|j| d�

|j| d �

|d \}}|d\} }| dd7<qPWtj||||| |||d �
q0W|S)z�Performs a single optimization step. Args: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrz'Rprop does not support sparse gradientsr
�step)
Z memory_format�prevrZ step_sizerr r)� step_size_min� step_size_max�etaminus�etaplus)�torchZenable_gradZparam_groups�grad�appendZ is_sparse�RuntimeError�state�lenZ zeros_likeZpreserve_format�newZ resize_as_Zfill_�
FZrprop)rZclosureZloss�grouprZgradsZprevsr �
pr r#rrrrrrrr=sH

























z Rprop.step�rr�r r )rr)r*)
N) �__name__� __module__�__qualname__�__doc__rrZno_gradr� __classcell__rr)
rrrs, 
r)r�rr&Z optimizerrrrrrr�
s