/usr/local/lib64/python3.6/site-packages/torch/optim/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/optim/__pycache__/optimizer.cpython-36.pyc (11827B)
3
Ege. @ sj d dl mZmZ d dlZd dlmZ d dlmZ d dl Z d dl
Z
G dd de Z e Z
G dd de ZdS ) ) defaultdictabcN)deepcopy)chainc @ s e Zd ZdZdd ZdS )_RequiredParameterzCSingleton class representing a required parameter for an Optimizer.c C s dS )Nz
)selfr r A/usr/local/lib64/python3.6/site-packages/torch/optim/optimizer.py__repr__ s z_RequiredParameter.__repr__N)__name__
__module__ __qualname____doc__r
r r r r r
s r c @ sh e Zd ZdZdd Zdd Zdd Zdd Zd
d Zd d
Z dd Z
de dddZ dd Z
dd ZdS ) OptimizeraX Base class for all optimizers.
.. warning::
Parameters need to be specified as collections that have a deterministic
ordering that is consistent between runs. Examples of objects that don't
satisfy those properties are sets and iterators over values of dictionaries.
Args:
params (iterable): an iterable of :class:`torch.Tensor` s or
:class:`dict` s. Specifies what Tensors should be optimized.
defaults: (dict): a dict containing default values of optimization
options (used when a parameter group doesn't specify them).
c C s t jjd || _| j t|t jr8tdt j| t t
| _ g | _ t
|}t|dkrdtdt|d t
s|d|ig}x|D ]}| j| qW d S )Nzpython.optimizerzZparams argument given to the optimizer should be an iterable of Tensors or dicts, but got r z%optimizer got an empty parameter listparams)torch_CZ_log_api_usage_oncedefaults_hook_for_profile
isinstanceTensor TypeErrortypenamer dictstate param_groupslistlen
ValueErroradd_param_group)r r r r param_groupr r r __init__! s
zOptimizer.__init__c C s | j | j| jdS )N)r r r )r r r )r r r r __getstate__8 s zOptimizer.__getstate__c C s | j j| | j d S )N)__dict__updater )r r r r r __setstate__? s zOptimizer.__setstate__c C sx | j jd }x^t| jD ]P\}}|d7 }|dj|7 }x0t|j D ] }|dkrD|dj||| 7 }qDW qW |d7 }|S )Nz (
zParameter Group {0}
r z
{0}: {1}
)) __class__r enumerater formatsortedkeys)r
format_stringigroupkeyr r r r
C s zOptimizer.__repr__c C sJ dj | jj| _dd }t| jjdd }|sF|| jj| j_d| jj_d S )Nz Optimizer.zero_grad#{}.zero_gradc s t j fdd}|S )Nc
s: | ^}}dj |jj}tjjj| | |S Q R X d S )NzOptimizer.step#{}.step)r* r( r r autogradprofilerrecord_function)argskwargsobj_Z profile_name)funcr r wrapperS s zGOptimizer._hook_for_profile..profile_hook_step..wrapper) functoolswraps)r8 r9 r )r8 r profile_hook_stepQ s z6Optimizer._hook_for_profile..profile_hook_stephookedT)r* r( r _zero_grad_profile_namegetattrstepr= )r r< r= r r r r N s
zOptimizer._hook_for_profilec sL i dfdd fdd| j D }fdd| jj D }||dS ) aK Returns the state of the optimizer as a :class:`dict`.
It contains two entries:
* state - a dict holding current optimization state. Its content
differs between optimizer classes.
* param_groups - a list containing all parameter groups where each
parameter group is a dict
r c sb dd | j D } j fddt| d D fdd| d D |d< t|d 7 |S )Nc S s i | ]\}}|d kr||qS )r r ).0kvr r r
p s z.pack_group..c s& i | ] \}}t | kr|t |qS r )id)rA r. p)param_mappingsr r rD q s r c s g | ]} t | qS r )rE )rA rF )rG r r
s s z.pack_group..)itemsr$ r) r )r/ packed)rG start_indexr r
pack_groupn s
"z(Optimizer.state_dict..pack_groupc s g | ] } |qS r r )rA g)rL r r rH v s z(Optimizer.state_dict..c s. i | ]&\}}|t |tjr& t| n|qS r )r r r rE )rA rB rC )rG r r rD x s z(Optimizer.state_dict..)r r )r r rI )r r Z packed_stater )rL rG rK r
state_dict` s
zOptimizer.state_dictc s( t |}| j}|d }t|t|kr.tddd |D }dd |D }tdd t||D rjtddd ttjd
d |D tjd d |D D } fd d
tt }x@|d j
D ]0\}} ||kr|| }
|
| ||
< q| ||< qW dd fddt||D } | j || d dS )zLoads the optimizer state.
Args:
state_dict (dict): optimizer state. Should be an object returned
from a call to :meth:`state_dict`.
r z s z,Optimizer.load_state_dict..c s s | ]}t |d V qdS )r N)r )rA rM r r r rO s c s s | ]\}}||kV qd S )Nr )rA Zp_lenZs_lenr r r rO s z]loaded state dict contains a parameter group that doesn't match the size of optimizer's groupc S s i | ]\}}||qS r r )rA Zold_idrF r r r rD s z-Optimizer.load_state_dict..c s s | ]}|d V qdS )r Nr )rA rM r r r rO s c s s | ]}|d V qdS )r Nr )rA rM r r r rO s c s t |tjr0 j r |j j}|j j}|S t |trR fdd|j D S t |t j
rzt | fdd|D S |S dS )zBMake a deep copy of value, casting all tensors to device of param.c s i | ]\}} ||qS r r )rA rB rC )castparamr r rD s z;Optimizer.load_state_dict..cast..c 3 s | ]} |V qd S )Nr )rA rC )rP rQ r r rO s z:Optimizer.load_state_dict..cast..N) r r r Zis_floating_pointZtoZdtypeZdevicer rI container_abcsIterabletype)rQ value)rP )rQ r rP s
z'Optimizer.load_state_dict..castr c S s | d |d< |S )Nr r )r/ Z new_groupr r r update_group s z/Optimizer.load_state_dict..update_groupc s g | ]\}} ||qS r r )rA rM Zng)rV r r rH s z-Optimizer.load_state_dict..)r r N) r r r r anyzipr
from_iterabler r rI r% ) r rN groupsZ saved_groupsZ
param_lensZ
saved_lensZid_mapr rB rC rQ r r )rP rV r load_state_dict s, zOptimizer.load_state_dictF) set_to_nonec C s t | ds| j tjjj| jl xd| jD ]Z}xT|d D ]H}|jdk r:|rTd|_q:|jj dk rl|jj
n |jj d |jj q:W q,W W dQ R X dS )a Sets the gradients of all optimized :class:`torch.Tensor` s to zero.
Args:
set_to_none (bool): instead of setting to zero, set the grads to None.
This will in general have lower memory footprint, and can modestly improve performance.
However, it changes certain behaviors. For example:
1. When the user tries to access a gradient and perform manual ops on it,
a None attribute or a Tensor full of 0s will behave differently.
2. If the user requests ``zero_grad(set_to_none=True)`` followed by a backward pass, ``.grad``\ s
are guaranteed to be None for params that did not receive a gradient.
3. ``torch.optim`` optimizers have a different behavior if the gradient is 0 or None
(in one case it does the step with a gradient of 0 and in the other it skips
the step altogether).
r> r NF)
hasattrr r r1 r2 r3 r> r ZgradZgrad_fnZdetach_Zrequires_grad_Zzero_)r r\ r/ rF r r r zero_grad s
zOptimizer.zero_gradc C s t dS )af Performs a single optimization step (parameter update).
Args:
closure (callable): A closure that reevaluates the model and
returns the loss. Optional for most optimizers.
.. note::
Unless otherwise specified, this function should not modify the
``.grad`` field of the parameters.
N)NotImplementedError)r Zclosurer r r r@ s zOptimizer.stepc C sT t |tstd|d }t |tjr2|g|d< n t |trFtdn t||d< x<|d D ]0}t |tjs~tdtj| |j s\t
dq\W x@| j j D ]2\}}|t
kr||krt
d| q|j|| qW |d }t|tt|k rtjddd t }x"| jD ]}|jt|d q W |jt|d sDt
d
| jj| d S ) a Add a param group to the :class:`Optimizer` s `param_groups`.
This can be useful when fine tuning a pre-trained network as frozen layers can be made
trainable and added to the :class:`Optimizer` as training progresses.
Args:
param_group (dict): Specifies what Tensors should be optimized along with group
specific optimization options.
zparam group must be a dictr zoptimizer parameters need to be organized in ordered collections, but the ordering of tensors in sets will change between runs. Please use a list instead.z>optimizer can only optimize Tensors, but one of the params is z can't optimize a non-leaf TensorzJparameter group didn't specify a value of required optimization parameter zoptimizer contains a parameter group with duplicate parameters; in future, this will cause an error; see github.com/pytorch/pytorch/issues/40967 for more information )
stacklevelz7some parameters appear in more than one parameter groupN)r r AssertionErrorr r setr r r Zis_leafr r rI required
setdefaultr warningswarnr r$
isdisjointappend)r r r rQ namedefaultZ param_setr/ r r r r s8
zOptimizer.add_param_groupN)F)r r r
r r! r" r% r
r rN r[ boolr^ r@ r r r r r r s
>
r ) collectionsr r rR r copyr itertoolsr rf r: objectr rd r r r r r s