/usr/local/lib64/python3.6/site-packages/caffe2/python/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/caffe2/python/__pycache__/regularizer.cpython-36.pyc (18604B)
3
EgR @ s, d dl mZmZ d dlZG dd deZG dd deZG dd deZG d d
d
eZ G d d d eZ
G d
d deZ G dd deZ G dd deZ
G dd deZG dd deZG dd deZG dd deZG dd d eZG d d d eZG d d d eZG d!d" d"eZG d#d$ d$eZdS )% )coreutilsNc @ s e Zd ZdZdZdS )RegularizationByZafter_optimizerZon_lossN)__name__
__module__ __qualname__ZAFTER_OPTIMIZERZON_LOSS r r E/usr/local/lib64/python3.6/site-packages/caffe2/python/regularizer.pyr s r c @ sB e Zd Zdd ZdddZdddZdd Zd
d Zdd
dZdS ) Regularizerc C s
d| _ d S )Ng& .>)kEpsilon)selfr r r __init__ s zRegularizer.__init__Nc C sv t |tjsttjt}||j ks>tdj| j ||j d| }t
| |sbtdj| j |t | |||||S )Nz>Regularizer of type {} is called with invalid by={}, not in {}Z_run_z5Regularizer of type {} does not implement function {})
isinstancer Z
BlobReferenceAssertionErrorr ZEnumClassKeyValsr valuesformat __class__hasattrgetattr)r netparam_init_netparamgradZbyZby_enumrun_funcr r r __call__ s
zRegularizer.__call__c C s d S )Nr )r r r r r r r r _run_on_loss' s zRegularizer._run_on_lossc C s d S )Nr )r r r r r r r r _run_after_optimizer* s z Regularizer._run_after_optimizerc C sL |j ||g|jdg}|j|g|jdg}|j|g|jdgdd}|S )N param_mul
param_reducedgrouped_feature_weight_vecg ?)exponent)MulNextScopedBlobReduceFrontSumPow)r r r r r r r r r _feature_grouping- s
z Regularizer._feature_groupingFc
C st |d k r|s|r|| j n|}|d k r8|s.|r8|| j n|}t|tjrV||j|jgn|g} |j| |g||d d S )N)minmax)r r r
GradientSliceindicesr Z
EnsureClipped)
r r r r r& r'
open_range left_open
right_openZ input_blobsr r r _ensure_clipped= s
zRegularizer._ensure_clipped)NN)N)NNNFFF) r r r r
r r r r% r- r r r r r
s
r
c s& e Zd Z fddZdddZ ZS )L1Normc s( t t| j |dks td|| _d S )Nr z6factor ahead of regularization should be 0 or positive)superr. r
r
reg_lambda)r r0 )r r r r
[ s zL1Norm.__init__Nc C s< |j |d }|j|g|gdd |j|g|g| jd |S )NZ_l1_regularization )p)scale)r" LpNormScaler0 )r r r r r output_blobr r r r a s zL1Norm._run_on_loss)N)r r r r
r
__classcell__r r )r r r. Z s r. c s( e Zd Zd fdd ZdddZ ZS ) r4 ?c s> t t| j |dks td|dks.td|| _|| _dS )a
reg_lambda: parameter to scale regularization by
p_value: determines what type of Lp norm to calculate. If p > 0,
we will calculate Lp norm with the formula:
pow( sum_i { pow(theda_i, p) } , 1/p)
r z7factor ahead of regularization should be greater than 0z'p_value factor should be greater than 0N)r/ r4 r
r p_valuer0 )r r0 r9 )r r r r
h s
zLpNorm.__init__Nc
C s |j |d }| j||}|j|g|j dg| jd}|j|g|j dg}|j|g|j dgd| j d} |j| g|g| jd |S )N _dense_feature_regularization
lp_vec_raised)r
lp_vec_summedZlp_vecr1 )r3 )r" r% r$ r9 r# r5 r0 )
r r r r r r6 r r; r< Zlp_normr r r r w s
zLpNorm._run_on_loss)r8 )N)r r r r
r r7 r r )r r r4 g s r4 c s( e Zd Zd fdd Zd ddZ ZS )
L0ApproxNorm{Gz?r c sX t t| j |dks td|dks.td|dks>td|| _|| _t|| _dS )a9
reg_lambda: parameter to scale regularization by
alpha: hyper parameter to tune that is only used in the calculation
of approximate L0 norm
budget: desired number of features. If the number of features is greater
than the budget amount, then the least important features will
be penalized. If there are fewer features than the desired
budget, no penalization will be applied. Optional parameter, if
0, then no budget is used
r z7factor ahead of regularization should be greater than 0z4alpha factor must be a positive value greater than 0z0budget factor must be greater than or equal to 0N)r/ r= r
r r0 alphafloatbudget)r r0 r? rA )r r r r
s
zL0ApproxNorm.__init__Nc C s |j |d }| j||}|j|g|j dg}|j|g|j dg| jd}|j|g|j dg} |j| g|j dgd| j d}
| jr|jg d dg| jd
} |j |
| g|j d g} |j
| g|j d g}
|j|
g|g| j d n|j|
g|g| j d |S )
Nr: l0_absl0_min)r' l0_summedl0_normr1 )r3 rA )shapevalueZ l0_budgetrelu_l0_sub_budget) r" r% AbsClipr? r# r5 rA ConstantFillSubZRelur0 )r r r r r r6 r rB rC rD rE Z budget_blobZ
l0_sub_budgetrH r r r r s zL0ApproxNorm._run_on_loss)r> r )N)r r r r
r r7 r r )r r r= s r= c s* e Zd ZdZ fddZdddZ ZS )
L1NormTrimmedzV
The Trimmed Lasso: Sparsity and Robustness. https://arxiv.org/abs/1708.04527
c sP t t| j |dks tdt|ts0td|dks@td|| _|| _d S )Nr z6factor ahead of regularization should be 0 or positivez6k should be an interger as expected #. after selectionr1 zk should be larger than 1)r/ rM r
r r intr0 k)r r0 rO )r r r r
s zL1NormTrimmed.__init__Nc C s |j |d }|j|g|j dg}|j|g|j dgdd}|j|g|j d|j d|j dg| jd \}} } |j|g|j d
gdd}
|j||
g|g |j|g|g| jd |S ) NZ_l1_trimmed_regularizationabssum_absF)averagetopkidflat_id)rO topk_sum)r3 )r" rI SumElementsTopKrO rL r5 r0 ) r r r r r r6 rP rQ rS _rV r r r r s 2zL1NormTrimmed._run_on_loss)N)r r r __doc__r
r r7 r r )r r rM s rM c s& e Zd Z fddZdddZ ZS )L2Normc s( t t| j |dks td|| _d S )Nr z6factor ahead of regularization should be 0 or positive)r/ r[ r
r r0 )r r0 )r r r r
s zL2Norm.__init__Nc C s< |j |d }|j|g|gdd |j|g|g| jd |S )NZ_l2_regularization )r2 )r3 )r" r4 r5 r0 )r r r r r r6 r r r r s zL2Norm._run_on_loss)N)r r r r
r r7 r r )r r r[ s r[ c s& e Zd Z fddZdddZ ZS )
ElasticNetc s t t| j || _|| _d S )N)r/ r] r
l1l2)r r^ r_ )r r r r
s zElasticNet.__init__Nc C s |j |d }|j |d }|j |d }|j|g|gdd |j|g|gdd |j|g|g| jd |j|g|g| jd |j||g|g |S )NZ_elastic_net_regularization_l2_blob_l1_blobr\ )r2 r1 )r3 )r" r4 r5 r_ r^ Add)r r r r r r6 l2_blobl1_blobr r r r s zElasticNet._run_on_loss)N)r r r r
r r7 r r )r r r] s r] c s& e Zd Z fddZdddZ ZS )ElasticNetL1NormTrimmedc s$ t t| j || _|| _|| _d S )N)r/ re r
r^ r_ rO )r r^ r_ rO )r r r r
s z ElasticNetL1NormTrimmed.__init__Nc
C s |j |d }|j |d }|j|g|gdd |j|g|g| jd |j |d }|j|g|j dg}|j|g|j dgd d
} |j|g|j d |j d |j d
g| jd\}
} } |j|
g|j dgd d
} |j| | g|g |j|g|g| j d |j
||g|g |S )NZ&_elastic_net_l1_trimmed_regularizationr` r\ )r2 )r3 ra rP rQ F)rR rS rT rU )rO rV ) r" r4 r5 r_ rI rW rX rO rL r^ rb )
r r r r r r6 rc rd rP rQ rS rY rV r r r r s 2z$ElasticNetL1NormTrimmed._run_on_loss)N)r r r r
r r7 r r )r r re s re c s& e Zd Zd fdd Zdd Z ZS )MaxNorm ?Nc s t t| j || _|| _d S )N)r/ rf r
normdtype)r rh ri )r r r r
s zMaxNorm.__init__c C sv | j dkstdt|tjrj| jrL| jdkrL|j||jg|gd| j d qr|j||jg|gd| j d nt dd S )Nr z norm should be bigger than 0.Zfp16T) use_max_normrh z-MaxNorm is not supported for dense parameters)
rh r r r r( ri ZFloat16SparseNormalizer) SparseNormalizeNotImplementedError)r r r r r r r r r ! s z MaxNorm._run_after_optimizer)rg N)r r r r
r r7 r r )r r rf s rf c s& e Zd Zd fdd Zdd Z ZS ) ConstantNorm ?c s t t| j || _d S )N)r/ rm r
rh )r rh )r r r r
7 s zConstantNorm.__init__c C sH | j dkstdt|tjr<|j||jg|gd| j d ntdd S )Nr z norm should be bigger than 0.F)rj rh z2ConstantNorm is not supported for dense parameters)rh r r r r( rk r) rl )r r r r r r r r r ; s z!ConstantNorm._run_after_optimizer)rn )r r r r
r r7 r r )r r rm 6 s rm c s$ e Zd Z fddZdd Z ZS ) SparseLpNormc s> t t| j |dks td|dks.td|| _|| _d S )N ? @zBSparse Lp regularization only implemented for p = 1.0 and p = 2.0.r z8factor ahead of regularization should be greater than 0.)rp rq )r/ ro r
r r2 r0 )r r2 r0 )r r r r
K s
zSparseLpNorm.__init__c C s8 t |tjr,|j||jg|g| j| jd ntdd S )N)r2 r0 z2SparseLpNorm is not supported for dense parameters)r r r( ZSparseLpRegularizerr) r2 r0 rl )r r r r r r r r r R s z!SparseLpNorm._run_after_optimizer)r r r r
r r7 r r )r r ro J s ro c s e Zd Z fddZ ZS ) SparseL1Normc s t t| jd|d d S )Ng ?)r2 r0 )r/ rr r
)r r0 )r r r r
_ s zSparseL1Norm.__init__)r r r r
r7 r r )r r rr ^ s rr c s e Zd Z fddZ ZS ) SparseL2Normc s t t| jd|d d S )Ng @)r2 r0 )r/ rs r
)r r0 )r r r r
d s zSparseL2Norm.__init__)r r r r
r7 r r )r r rs c s rs c s4 e Zd ZdZd
fdd Zd ddZdd Z ZS )
LogBarrierzr
Wright, S., & Nocedal, J. (1999). Numerical optimization. Springer Science,
35(67-68), 7. Chapter 19
invNc s> t t| j |dks td|| _|| _|p6ddd| _dS )z
discount is a positive weight that is decreasing, and here it is implemented
similar to the learning rate. It is specified by a learning rate policy and
corresponding options
r z6factor ahead of regularization should be 0 or positiveg ?)gammapowerN)r/ rt r
r r0 discount_policydiscount_options)r r0 rx ry )r r r r
n s
zLogBarrier.__init__c C s t j||}|j|d }|j|g|gf| j | jd| j |j|d }|j|g|g| jd |j|d }|j |g|g |j|d } |j
|g| g |j|d }
|j | |g|
gdd |
S )
NZ_log_barrier_discount)Zbase_lrpolicyZ_non_neg)r& _logZ_log_sumZ _log_barrierr1 ) broadcast) r ZBuildUniqueMutexIterr" Z LearningRater0 rx ry rJ r LogrW r! ) r r r r r iterationZdiscountZ
param_non_negZ param_logZ
param_log_sumr6 r r r r z s"
zLogBarrier._run_on_lossc C s | j |||ddd d S )Nr T)r& r* )r- )r r r r r r r r r s z LogBarrier._run_after_optimizer)ru N)N)r r r rZ r
r r r7 r r )r r rt h s
rt c s* e Zd ZdZd fdd Zdd Z ZS ) BoundedGradientProjectionzr
Wright, S., & Nocedal, J. (1999). Numerical optimization. Springer Science,
35(67-68), 7. Chapter 16
NFc s t t| j |d k r t|nd }|d k r2t|nd }|d k rFt|n| j}|dksdtdj|d|d ks|d ks||r~|nd ||r|nd kstdj|||rdnd|rdnd |d
|| _|| _|| _|| _ || _
d S ) Nr z2Bounded Gradient Projection with invalid eps={eps})epsg zLBounded Gradient Projection with invalid {lp}ub={ub}, lb={lb}{rp}, eps={eps}([)])lbubZlprpr ) r/ r r
r@ r r r r+ r, r r )r r r r+ r, epsilon)r r r r
s*
z"BoundedGradientProjection.__init__c C s$ | j |||| j| j| j| jd d S )N)r& r' r+ r, )r- r r r+ r, )r r r r r r r r r s z.BoundedGradientProjection._run_after_optimizer)NNFFN)r r r rZ r
r r7 r r )r r r s r c s, e Zd ZdZd fdd Zd ddZ ZS )
GroupL1Norma
Scardapane, Simone, et al. "Group sparse regularization for deep neural networks."
Neurocomputing 241 (2017): 81-89.
This regularizer computes l1 norm of a weight matrix based on groups.
There are essentially three stages in the computation:
1. Compute the l2 norm on all the members of each group
2. Scale each l2 norm by the size of each group
3. Compute the l1 norm of the scaled l2 norms
r c sF t t| j |dks tdt|ts0td|| _|| _|| _dS )a
Args:
reg_lambda: The weight of the regularization term.
groups: A list of integers describing the size of each group.
The length of the list is the number of groups.
Optional Args:
stabilizing_val: The computation of GroupL1Norm involves the Sqrt
operator. When values are small, its gradient can be numerically
unstable and causing gradient explosion. Adding this term to
stabilize gradient calculation. Recommended value of this term is
1e-8, but it depends on the specific scenarios. If the implementation
of the gradient operator of Sqrt has taken into stability into
consideration, this term won't be necessary.
r z-regularization weight should be 0 or positivezgroups needs to be a listN) r/ r r
r r listr0 groupsstabilizing_val)r r0 r r )r r r r
s
zGroupL1Norm.__init__Nc C s |j |}|j|dgdd}|j||jg dt| jg| jdg}| jrl|j||jg d| jdg|gdd |j |}|j
||j g t| jgt j
| j| j dgdg} |j| dgdd }
|
S )
a
Args:
param: The input blob to regularize. It should be a weight matrix
blob with shape (output_dim, input_dim). input_dim should be
equal to the sum of self.groups.
Returns:
group_l1_norm: The output blob after applying regularization.
These are the steps of computation:
1. square all elements
2. sum by row
3. lengthssum by group
4. square_root all elements
5. normalize each group based on group size
6. compute l1 norm of each group
7. scale the result with the regularization lambda
r )ZaxesZkeepdimsr1 )rF r )rG )r| Znormalized_l2_norm_scaledZ
group_l1_nrom)r2 )ZSqrZ ReduceSumZ
LengthsSumZGivenTensorIntFilllenr r rb rK ZSqrtr! ZGivenTensorFillnpsqrtr0 r4 ) r r r r r ZsquaredZ reduced_sumZ lengths_sumr Z l2_scaledZ
group_l1_normr r r r s*
zGroupL1Norm._run_on_loss)r )N)r r r rZ r
r r7 r r )r r r s
r )Z
caffe2.pythonr r Znumpyr objectr r
r. r4 r= rM r[ r] re rf rm ro rr rs rt r r r r r r
s$ L
,7.3