/usr/local/lib64/python3.6/site-packages/torch/nn/utils/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/nn/utils/__pycache__/prune.cpython-36.pyc (45557B)
3
Eg| @ s> d Z ddlZddlmZmZ ddlmZ ddlmZ ddl Z G dd deZ
G dd d e
Z G d
d d e
Z G d d
d
e
Z
G dd de
ZG dd de
ZG dd de
ZG dd de
Zdd Zdd Zd4ddZd d Zd5d d Zd6d d!Zd"d# Zd$d% Zd&d' Zd(d) Zd*d+ Z d,d- Z d.d/ Z d0d1 Z d2d3 Z dS )7z
Pruning methods
N)ABCabstractmethod)Iterable)Tuplec @ s^ e Zd ZU dZedd Zdd Zedd Z dd Z
e d
d d d
Z dddZ
dd Zd
S )BasePruningMethodzAbstract base class for creation of new pruning techniques.
Provides a skeleton for customization requiring the overriding of methods
such as :meth:`compute_mask` and :meth:`apply`.
c C s d S )N )selfr r @/usr/local/lib64/python3.6/site-packages/torch/nn/utils/prune.py__init__ s zBasePruningMethod.__init__c C s t || j| j| dS )aW Multiplies the mask (stored in ``module[name + '_mask']``)
into the original tensor (stored in ``module[name + '_orig']``)
and stores the result into ``module[name]`` by using
:meth:`apply_mask`.
Args:
module (nn.Module): module containing the tensor to prune
inputs: not used.
N)setattr _tensor_name
apply_mask)r moduleinputsr r r __call__ s
zBasePruningMethod.__call__c C s dS )a Computes and returns a mask for the input tensor ``t``.
Starting from a base ``default_mask`` (which should be a mask of ones
if the tensor has not been pruned yet), generate a random mask to
apply on top of the ``default_mask`` according to the specific pruning
method recipe.
Args:
t (torch.Tensor): tensor representing the importance scores of the
parameter to prune.
default_mask (torch.Tensor): Base mask from previous pruning
iterations, that need to be respected after the new mask is
applied. Same dims as ``t``.
Returns:
mask (torch.Tensor): mask to apply to ``t``, of same dims as ``t``
Nr )r t default_maskr r r compute_mask# s z BasePruningMethod.compute_maskc C sN | j dk stdj|t|| j d }t|| j d }|j|jd| }|S )a Simply handles the multiplication between the parameter being
pruned and the generated mask.
Fetches the mask and the original tensor from the module
and returns the pruned version of the tensor.
Args:
module (nn.Module): module containing the tensor to prune
Returns:
pruned_tensor (torch.Tensor): pruned version of the input tensor
NzModule {} has to be pruned_mask_orig)dtype)r AssertionErrorformatgetattrtor )r r maskorigZ
pruned_tensorr r r r
7 s z BasePruningMethod.apply_maskN)importance_scoresc O s8 dd }|| ||f||}t ||}|dk rL|j|jksPtdj||n|}t|ts~|j|d | |j|= tj |} n t ||d j
j tj d} y>|j
|| d}
|j|d |
t|||j| |j| W nZ tk
r2 } z
._get_composite_methodNzRimportance_scores should have the same shape as parameter {} of {}r r )
memory_format)r )r shaper r r! r# Zregister_parameter _parameterstorch ones_likedetachclonecontiguous_formatr Zregister_bufferr r
Zregister_forward_pre_hook Exception) r% r r& r r' r( r. r, r r r er r r applyM s4 1
zBasePruningMethod.applyc C sJ |dk r |j |j ks"tdn|}|dk r.|ntj|}|| j||d S )a Computes and returns a pruned version of input tensor ``t``
according to the pruning rule specified in :meth:`compute_mask`.
Args:
t (torch.Tensor): tensor to prune (of same dimensions as
``default_mask``).
importance_scores (torch.Tensor): tensor of importance scores (of
same shape as ``t``) used to compute mask for pruning ``t``.
The values in this tensor indicate the importance of the
corresponding elements in the ``t`` that is being pruned.
If unspecified or None, the tensor ``t`` will be used in its place.
default_mask (torch.Tensor, optional): mask from previous pruning
iteration, if any. To be considered when determining what
portion of the tensor that pruning should act on. If None,
default to a mask of ones.
Returns:
pruned version of tensor ``t``.
Nz8importance_scores should have the same shape as tensor t)r )r0 r r2 r3 r )r r r r r r r prune s zBasePruningMethod.prunec C s | j dk stdj|| j|}t|| j r:t|| j |j| j d }|j|_|j| j d = |j| j d = t || j | dS )ar Removes the pruning reparameterization from a module. The pruned
parameter named ``name`` remains permanently pruned, and the parameter
named ``name+'_orig'`` is removed from the parameter list. Similarly,
the buffer named ``name+'_mask'`` is removed from the buffers.
Note:
Pruning itself is NOT undone or reversed!
NzCModule {} has to be pruned before pruning can be removedr r )
r r r r
hasattrdelattrr1 dataZ_buffersr )r r Zweightr r r r remove s
zBasePruningMethod.remove)NN)__name__
__module__ __qualname____doc__strr r
r r r r
classmethodr9 r: r> r r r r r s
r c @ s@ e Zd ZdZdd Zdd Zdd Zdd Zd
d Zd d
Z dS )r# a Container holding a sequence of pruning methods for iterative pruning.
Keeps track of the order in which pruning methods are applied and handles
combining successive pruning calls.
Accepts as argument an instance of a BasePruningMethod or an iterable of
them.
c G sj t | _t|ts&|j| _| j| n@t|dkrN|d j| _| j|d nx|D ]}| j| qTW d S )Nr r )tuple_pruning_methodsr! r r r$ len)r r' r, r r r r
s
zPruningContainer.__init__c C sn t |t r(|dk r(tdjt|n2|dk rZ| j|jkrZtdj| j| dj|j | j|f7 _dS )zAdds a child pruning ``method`` to the container.
Args:
method (subclass of BasePruningMethod): child pruning method
to be added to the container.
Nz&{} is not a BasePruningMethod subclasszWCan only add pruning methods acting on the parameter named '{}' to PruningContainer {}.z Found '{}')r! r TypeErrorr typer
ValueErrorrF )r r, r r r r$ ! s z#PruningContainer.add_pruning_methodc C s
t | jS )N)rG rF )r r r r __len__8 s zPruningContainer.__len__c C s
t | jS )N)iterrF )r r r r __iter__; s zPruningContainer.__iter__c C s
| j | S )N)rF )r idxr r r __getitem__> s z PruningContainer.__getitem__c C s" dd }| j d }||||}|S )a Applies the latest ``method`` by computing the new partial masks
and returning its combination with the ``default_mask``.
The new partial mask should be computed on the entries or channels
that were not zeroed out by the ``default_mask``.
Which portions of the tensor ``t`` the new mask will be calculated from
depends on the ``PRUNING_TYPE`` (handled by the type handler):
* for 'unstructured', the mask will be computed from the raveled
list of nonmasked entries;
* for 'structured', the mask will be computed from the nonmasked
channels in the tensor;
* for 'global', the mask will be computed across all entries.
Args:
t (torch.Tensor): tensor representing the parameter to prune
(of same dimensions as ``default_mask``).
default_mask (torch.Tensor): mask from previous pruning iteration.
Returns:
mask (torch.Tensor): new mask that combines the effects
of the ``default_mask`` and the new mask from the current
pruning ``method`` (of same dimensions as ``default_mask`` and
``t``).
c s |}|j |jd}| jdkr&|dk}n| jdkrt| dsBtd|j }| j dk r`| dk rvtdj||j fd d
t |D d dk}t
d g| }|| < n4| jd
krt |j }t
d g| }nt
dj| j| j|| || d}|j |jd||< |S )aJ
Args:
method (a BasePruningMethod subclass): pruning method
currently being applied.
t (torch.Tensor): tensor representing the parameter to prune
(of same dimensions as mask).
mask (torch.Tensor): mask from previous pruning iteration
Returns:
new_mask (torch.Tensor): new mask that combines the effects
of the old mask and the new mask from the current
pruning method (of same dimensions as mask and t).
)r unstructuredr
structureddimzVPruning methods of PRUNING_TYPE "structured" need to have the attribute `dim` defined.r z4Index is out of bounds for tensor with dimensions {}c s g | ]}| kr|qS r r ).0d)rR r r
s zIPruningContainer.compute_mask.._combine_masks..)rR Nglobalz Unrecognized PRUNING_TYPE {})r )r r PRUNING_TYPEr; AttributeErrorrR
IndexErrorr sumrangeslicerG r0 rJ r )r, r r Znew_maskslcZn_dimsZ keep_channelZ partial_maskr )rR r _combine_masks] s6
"
z5PruningContainer.compute_mask.._combine_masksr )rF )r r r r^ r, r r r r r A s B
z PruningContainer.compute_maskN)
r? r@ rA rB r
r$ rK rM rO r r r r r r# s r# c s0 e Zd ZdZdZdd Ze fddZ ZS )Identityz|Utility pruning method that does not prune any units but generates the
pruning parametrization with a mask of ones.
rP c C s |}|S )Nr )r r r r r r r r s zIdentity.compute_maskc s t t| j||S )ag Adds the forward pre-hook that enables pruning on the fly and
the reparametrization of a tensor in terms of the original tensor
and the pruning mask.
Args:
module (nn.Module): module containing the tensor to prune
name (str): parameter name within ``module`` on which pruning
will act.
)superr` r9 )r% r r& ) __class__r r r9 s zIdentity.apply) r? r@ rA rB rW r rD r9
__classcell__r r )rb r r` s r` c s8 e Zd ZdZdZdd Zdd Ze fddZ Z S ) RandomUnstructureda Prune (currently unpruned) units in a tensor at random.
Args:
name (str): parameter name within ``module`` on which pruning
will act.
amount (int or float): quantity of parameters to prune.
If ``float``, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If ``int``, it represents the
absolute number of parameters to prune.
rP c C s t | || _d S )N) _validate_pruning_amount_initamount)r rf r r r r
s zRandomUnstructured.__init__c C sf |j }t| j|}t|| |jtjd}|dkrbtj|}tj|j d|d}d|j d|j
< |S )N)r/ r r )r* r_ r_ ) nelement_compute_nparams_toprunerf _validate_pruning_amountr5 r2 r6 Z rand_liketopkviewindices)r r r tensor_sizenparams_topruner probrj r r r r s
z RandomUnstructured.compute_maskc s t t| j|||dS )a Adds the forward pre-hook that enables pruning on the fly and
the reparametrization of a tensor in terms of the original tensor
and the pruning mask.
Args:
module (nn.Module): module containing the tensor to prune
name (str): parameter name within ``module`` on which pruning
will act.
amount (int or float): quantity of parameters to prune.
If ``float``, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If ``int``, it represents the
absolute number of parameters to prune.
)rf )ra rd r9 )r% r r& rf )rb r r r9 s zRandomUnstructured.apply)
r? r@ rA rB rW r
r rD r9 rc r r )rb r rd s
rd c s: e Zd ZdZdZdd Zdd Zed
fdd Z Z S ) L1Unstructuredaz Prune (currently unpruned) units in a tensor by zeroing out the ones
with the lowest L1-norm.
Args:
amount (int or float): quantity of parameters to prune.
If ``float``, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If ``int``, it represents the
absolute number of parameters to prune.
rP c C s t | || _d S )N)re rf )r rf r r r r
s zL1Unstructured.__init__c C sd |j }t| j|}t|| |jtjd}|dkr`tjtj|j d|dd}d|j d|j
< |S )N)r/ r r F)r* largestr_ r_ ) rg rh rf ri r5 r2 r6 rj absrk rl )r r r rm rn r rj r r r r s
zL1Unstructured.compute_maskNc s t t| j||||dS )a Adds the forward pre-hook that enables pruning on the fly and
the reparametrization of a tensor in terms of the original tensor
and the pruning mask.
Args:
module (nn.Module): module containing the tensor to prune
name (str): parameter name within ``module`` on which pruning
will act.
amount (int or float): quantity of parameters to prune.
If ``float``, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If ``int``, it represents the
absolute number of parameters to prune.
importance_scores (torch.Tensor): tensor of importance scores (of same
shape as module parameter) used to compute mask for pruning.
The values in this tensor indicate the importance of the corresponding
elements in the parameter being pruned.
If unspecified or None, the module parameter will be used in its place.
)rf r )ra rp r9 )r% r r& rf r )rb r r r9 s
zL1Unstructured.apply)N)
r? r@ rA rB rW r
r rD r9 rc r r )rb r rp s rp c s< e Zd ZdZdZd ddZdd Zed
fdd Z Z S )RandomStructureda Prune entire (currently unpruned) channels in a tensor at random.
Args:
amount (int or float): quantity of parameters to prune.
If ``float``, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If ``int``, it represents the
absolute number of parameters to prune.
dim (int, optional): index of the dim along which we define
channels to prune. Default: -1.
rQ r c C s t | || _|| _d S )N)re rf rR )r rf rR r r r r
E s zRandomStructured.__init__c C sr t | t|| j |j| j }t| j|}t|| dd }|dkrL|}n"||| j||}||j|jd9 }|S )a Computes and returns a mask for the input tensor ``t``.
Starting from a base ``default_mask`` (which should be a mask of ones
if the tensor has not been pruned yet), generate a random mask to
apply on top of the ``default_mask`` by randomly zeroing out channels
along the specified dim of the tensor.
Args:
t (torch.Tensor): tensor representing the parameter to prune
default_mask (torch.Tensor): Base mask from previous pruning
iterations, that need to be respected after the new mask is
applied. Same dims as ``t``.
Returns:
mask (torch.Tensor): mask to apply to ``t``, of same dims as ``t``
Raises:
IndexError: if ``self.dim >= len(t.shape)``
c S sT t j|}t j||dj}||k}t j| }td gt| j }|||< d||< |S )N)r* r )r2 ZrandZkthvaluevalues
zeros_liker\ rG r0 ) r rR Z nchannelsZnchannels_toprunero thresholdZ channel_maskr r] r r r make_maskr s
z0RandomStructured.compute_mask..make_maskr )r ) _validate_structured_pruning_validate_pruning_dimrR r0 rh rf ri r r )r r r rm rn rw r r r r r K s
z RandomStructured.compute_maskc s t t| j||||dS )a Adds the forward pre-hook that enables pruning on the fly and
the reparametrization of a tensor in terms of the original tensor
and the pruning mask.
Args:
module (nn.Module): module containing the tensor to prune
name (str): parameter name within ``module`` on which pruning
will act.
amount (int or float): quantity of parameters to prune.
If ``float``, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If ``int``, it represents the
absolute number of parameters to prune.
dim (int, optional): index of the dim along which we define
channels to prune. Default: -1.
)rf rR )ra rs r9 )r% r r& rf rR )rb r r r9 s zRandomStructured.applyr_ )r_ r_ )r_ )
r? r@ rA rB rW r
r rD r9 rc r r )rb r rs 7 s
>rs c s< e Zd ZdZdZd ddZdd Zed
fd d
Z Z S ) LnStructuredai Prune entire (currently unpruned) channels in a tensor based on their
L\ ``n``-norm.
Args:
amount (int or float): quantity of channels to prune.
If ``float``, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If ``int``, it represents the
absolute number of parameters to prune.
n (int, float, inf, -inf, 'fro', 'nuc'): See documentation of valid
entries for argument ``p`` in :func:`torch.norm`.
dim (int, optional): index of the dim along which we define
channels to prune. Default: -1.
rQ r c C s t | || _|| _|| _d S )N)re rf nrR )r rf r{ rR r r r r
s zLnStructured.__init__c
C s t | t|| j |j| j }t| j|}|| }t|| t|| j| j}t j
||dd}dd }|dkrt|} n"||| j|j } | |j | j
d9 } | S )a Computes and returns a mask for the input tensor ``t``.
Starting from a base ``default_mask`` (which should be a mask of ones
if the tensor has not been pruned yet), generate a mask to apply on
top of the ``default_mask`` by zeroing out the channels along the
specified dim with the lowest L\ ``n``-norm.
Args:
t (torch.Tensor): tensor representing the parameter to prune
default_mask (torch.Tensor): Base mask from previous pruning
iterations, that need to be respected after the new mask is
applied. Same dims as ``t``.
Returns:
mask (torch.Tensor): mask to apply to ``t``, of same dims as ``t``
Raises:
IndexError: if ``self.dim >= len(t.shape)``
T)r* rq c S s2 t j| }td gt| j }|||< d||< |S )Nr )r2 ru r\ rG r0 )r rR rl r r] r r r rw s
z,LnStructured.compute_mask..make_maskr )r )rx ry rR r0 rh rf ri
_compute_normr{ r2 rj rl r r )
r r r rm rn Znparams_tokeepnormrj rw r r r r r s
zLnStructured.compute_maskNc s t t| j||||||dS )a Adds the forward pre-hook that enables pruning on the fly and
the reparametrization of a tensor in terms of the original tensor
and the pruning mask.
Args:
module (nn.Module): module containing the tensor to prune
name (str): parameter name within ``module`` on which pruning
will act.
amount (int or float): quantity of parameters to prune.
If ``float``, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If ``int``, it represents the
absolute number of parameters to prune.
n (int, float, inf, -inf, 'fro', 'nuc'): See documentation of valid
entries for argument ``p`` in :func:`torch.norm`.
dim (int): index of the dim along which we define channels to
prune.
importance_scores (torch.Tensor): tensor of importance scores (of same
shape as module parameter) used to compute mask for pruning.
The values in this tensor indicate the importance of the corresponding
elements in the parameter being pruned.
If unspecified or None, the module parameter will be used in its place.
)rf r{ rR r )ra rz r9 )r% r r& rf r{ rR r )rb r r r9 s
zLnStructured.applyr_ )r_ )N)
r? r@ rA rB rW r
r rD r9 rc r r )rb r rz s
Erz c s4 e Zd ZdZdd Zdd Ze fddZ ZS )CustomFromMaskrV c C s
|| _ d S )N)r )r r r r r r
s zCustomFromMask.__init__c C s* |j | jj kst|| jj|jd }|S )N)r )r0 r r r r )r r r r r r r r # s zCustomFromMask.compute_maskc s t t| j|||dS )ag Adds the forward pre-hook that enables pruning on the fly and
the reparametrization of a tensor in terms of the original tensor
and the pruning mask.
Args:
module (nn.Module): module containing the tensor to prune
name (str): parameter name within ``module`` on which pruning
will act.
)r )ra r~ r9 )r% r r& r )rb r r r9 ( s zCustomFromMask.apply) r? r@ rA rW r
r rD r9 rc r r )rb r r~ s r~ c C s t j| | | S )a Applies pruning reparametrization to the tensor corresponding to the
parameter called ``name`` in ``module`` without actually pruning any
units. Modifies module in place (and also return the modified module)
by:
1) adding a named buffer called ``name+'_mask'`` corresponding to the
binary mask applied to the parameter ``name`` by the pruning method.
2) replacing the parameter ``name`` by its pruned version, while the
original (unpruned) parameter is stored in a new parameter named
``name+'_orig'``.
Note:
The mask is a tensor of ones.
Args:
module (nn.Module): module containing the tensor to prune.
name (str): parameter name within ``module`` on which pruning
will act.
Returns:
module (nn.Module): modified (i.e. pruned) version of the input module
Examples:
>>> m = prune.identity(nn.Linear(2, 3), 'bias')
>>> print(m.bias_mask)
tensor([1., 1., 1.])
)r` r9 )r r& r r r identity6 s r c C s t j| || | S )a Prunes tensor corresponding to parameter called ``name`` in ``module``
by removing the specified ``amount`` of (currently unpruned) units
selected at random.
Modifies module in place (and also return the modified module) by:
1) adding a named buffer called ``name+'_mask'`` corresponding to the
binary mask applied to the parameter ``name`` by the pruning method.
2) replacing the parameter ``name`` by its pruned version, while the
original (unpruned) parameter is stored in a new parameter named
``name+'_orig'``.
Args:
module (nn.Module): module containing the tensor to prune
name (str): parameter name within ``module`` on which pruning
will act.
amount (int or float): quantity of parameters to prune.
If ``float``, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If ``int``, it represents the
absolute number of parameters to prune.
Returns:
module (nn.Module): modified (i.e. pruned) version of the input module
Examples:
>>> m = prune.random_unstructured(nn.Linear(2, 3), 'weight', amount=1)
>>> torch.sum(m.weight_mask == 0)
tensor(1)
)rd r9 )r r& rf r r r random_unstructuredV s r c C s t j| |||d | S )ax Prunes tensor corresponding to parameter called ``name`` in ``module``
by removing the specified `amount` of (currently unpruned) units with the
lowest L1-norm.
Modifies module in place (and also return the modified module)
by:
1) adding a named buffer called ``name+'_mask'`` corresponding to the
binary mask applied to the parameter ``name`` by the pruning method.
2) replacing the parameter ``name`` by its pruned version, while the
original (unpruned) parameter is stored in a new parameter named
``name+'_orig'``.
Args:
module (nn.Module): module containing the tensor to prune
name (str): parameter name within ``module`` on which pruning
will act.
amount (int or float): quantity of parameters to prune.
If ``float``, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If ``int``, it represents the
absolute number of parameters to prune.
importance_scores (torch.Tensor): tensor of importance scores (of same
shape as module parameter) used to compute mask for pruning.
The values in this tensor indicate the importance of the corresponding
elements in the parameter being pruned.
If unspecified or None, the module parameter will be used in its place.
Returns:
module (nn.Module): modified (i.e. pruned) version of the input module
Examples:
>>> m = prune.l1_unstructured(nn.Linear(2, 3), 'weight', amount=0.2)
>>> m.state_dict().keys()
odict_keys(['bias', 'weight_orig', 'weight_mask'])
)rf r )rp r9 )r r& rf r r r r l1_unstructuredx s #r c C s t j| ||| | S )a Prunes tensor corresponding to parameter called ``name`` in ``module``
by removing the specified ``amount`` of (currently unpruned) channels
along the specified ``dim`` selected at random.
Modifies module in place (and also return the modified module)
by:
1) adding a named buffer called ``name+'_mask'`` corresponding to the
binary mask applied to the parameter ``name`` by the pruning method.
2) replacing the parameter ``name`` by its pruned version, while the
original (unpruned) parameter is stored in a new parameter named
``name+'_orig'``.
Args:
module (nn.Module): module containing the tensor to prune
name (str): parameter name within ``module`` on which pruning
will act.
amount (int or float): quantity of parameters to prune.
If ``float``, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If ``int``, it represents the
absolute number of parameters to prune.
dim (int): index of the dim along which we define channels to prune.
Returns:
module (nn.Module): modified (i.e. pruned) version of the input module
Examples:
>>> m = prune.random_structured(
nn.Linear(5, 3), 'weight', amount=3, dim=1
)
>>> columns_pruned = int(sum(torch.sum(m.weight, dim=0) == 0))
>>> print(columns_pruned)
3
)rs r9 )r r& rf rR r r r random_structured s "r c C s t j| |||||d | S )aO Prunes tensor corresponding to parameter called ``name`` in ``module``
by removing the specified ``amount`` of (currently unpruned) channels
along the specified ``dim`` with the lowest L\ ``n``-norm.
Modifies module in place (and also return the modified module)
by:
1) adding a named buffer called ``name+'_mask'`` corresponding to the
binary mask applied to the parameter ``name`` by the pruning method.
2) replacing the parameter ``name`` by its pruned version, while the
original (unpruned) parameter is stored in a new parameter named
``name+'_orig'``.
Args:
module (nn.Module): module containing the tensor to prune
name (str): parameter name within ``module`` on which pruning
will act.
amount (int or float): quantity of parameters to prune.
If ``float``, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If ``int``, it represents the
absolute number of parameters to prune.
n (int, float, inf, -inf, 'fro', 'nuc'): See documentation of valid
entries for argument ``p`` in :func:`torch.norm`.
dim (int): index of the dim along which we define channels to prune.
importance_scores (torch.Tensor): tensor of importance scores (of same
shape as module parameter) used to compute mask for pruning.
The values in this tensor indicate the importance of the corresponding
elements in the parameter being pruned.
If unspecified or None, the module parameter will be used in its place.
Returns:
module (nn.Module): modified (i.e. pruned) version of the input module
Examples:
>>> m = prune.ln_structured(
nn.Conv2d(5, 3, 2), 'weight', amount=0.3, dim=1, n=float('-inf')
)
)r )rz r9 )r r& rf r{ rR r r r r
ln_structured s &r c s t | tstd dk r ni t ts4tdtjjj fdd| D }tjjjdd | D }t }d|_ |f |}d|_ |j
dkrtd j ||j
|j | |j
||}d
} xN| D ]F\}
} t|
| } | j }
|| | |
j| }t|
| |d | |
7 } qW dS ) aD
Globally prunes tensors corresponding to all parameters in ``parameters``
by applying the specified ``pruning_method``.
Modifies modules in place by:
1) adding a named buffer called ``name+'_mask'`` corresponding to the
binary mask applied to the parameter ``name`` by the pruning method.
2) replacing the parameter ``name`` by its pruned version, while the
original (unpruned) parameter is stored in a new parameter named
``name+'_orig'``.
Args:
parameters (Iterable of (module, name) tuples): parameters of
the model to prune in a global fashion, i.e. by aggregating all
weights prior to deciding which ones to prune. module must be of
type :class:`nn.Module`, and name must be a string.
pruning_method (function): a valid pruning function from this module,
or a custom one implemented by the user that satisfies the
implementation guidelines and has ``PRUNING_TYPE='unstructured'``.
importance_scores (dict): a dictionary mapping (module, name) tuples to
the corresponding parameter's importance scores tensor. The tensor
should be the same shape as the parameter, and is used for computing
mask for pruning.
If unspecified or None, the parameter will be used in place of its
importance scores.
kwargs: other keyword arguments such as:
amount (int or float): quantity of parameters to prune across the
specified parameters.
If ``float``, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If ``int``, it represents the
absolute number of parameters to prune.
Raises:
TypeError: if ``PRUNING_TYPE != 'unstructured'``
Note:
Since global structured pruning doesn't make much sense unless the
norm is normalized by the size of the parameter, we now limit the
scope of global pruning to unstructured methods.
Examples:
>>> net = nn.Sequential(OrderedDict([
('first', nn.Linear(10, 4)),
('second', nn.Linear(4, 1)),
]))
>>> parameters_to_prune = (
(net.first, 'weight'),
(net.second, 'weight'),
)
>>> prune.global_unstructured(
parameters_to_prune,
pruning_method=prune.L1Unstructured,
amount=10,
)
>>> print(sum(torch.nn.utils.parameters_to_vector(net.buffers()) == 0))
tensor(10, dtype=torch.uint8)
z4global_unstructured(): parameters is not an IterableNz=global_unstructured(): importance_scores must be of type dictc s& g | ] \}} j ||ft||qS r )getr )rS r r& )r r r rU 9 s z'global_unstructured..c S s, g | ]$\}}t ||d tjt ||qS )r )r r2 r3 )rS r r& r r r rU A s temprP z_Only "unstructured" PRUNING_TYPE supported for the `pruning_method`. Found method {} of type {}r )r )r! r rH dictr2 nnutilsZparameters_to_vectorr# r rW r r$ r r ZnumelZview_ascustom_from_mask)
parametersZpruning_methodr r( Zrelevant_importance_scoresr r- r, Z
final_maskpointerr r& paramZ num_paramZ
param_maskr )r r global_unstructured s8 <
r c C s t j| || | S )a Prunes tensor corresponding to parameter called ``name`` in ``module``
by applying the pre-computed mask in ``mask``.
Modifies module in place (and also return the modified module)
by:
1) adding a named buffer called ``name+'_mask'`` corresponding to the
binary mask applied to the parameter ``name`` by the pruning method.
2) replacing the parameter ``name`` by its pruned version, while the
original (unpruned) parameter is stored in a new parameter named
``name+'_orig'``.
Args:
module (nn.Module): module containing the tensor to prune
name (str): parameter name within ``module`` on which pruning
will act.
mask (Tensor): binary mask to be applied to the parameter.
Returns:
module (nn.Module): modified (i.e. pruned) version of the input module
Examples:
>>> m = prune.custom_from_mask(
nn.Linear(5, 3), name='bias', mask=torch.tensor([0, 1, 0])
)
>>> print(m.bias_mask)
tensor([0., 1., 0.])
)r~ r9 )r r& r r r r r k s r c C sV x@| j j D ]2\}}t|tr |j|kr |j| | j |= | S q W tdj|| dS )a Removes the pruning reparameterization from a module and the
pruning method from the forward hook. The pruned
parameter named ``name`` remains permanently pruned, and the parameter
named ``name+'_orig'`` is removed from the parameter list. Similarly,
the buffer named ``name+'_mask'`` is removed from the buffers.
Note:
Pruning itself is NOT undone or reversed!
Args:
module (nn.Module): module containing the tensor to prune
name (str): parameter name within ``module`` on which pruning
will act.
Examples:
>>> m = random_unstructured(nn.Linear(5, 7), name='weight', amount=0.2)
>>> m = remove(m, name='weight')
zJParameter '{}' of module {} has to be pruned before pruning can be removedN)r r r! r r r> rJ r )r r& r* r+ r r r r> s
r> c C s@ x:| j D ].\}}x$|jj D ]\}}t|tr dS q W q
W dS )a Check whether ``module`` is pruned by looking for
``forward_pre_hooks`` in its modules that inherit from the
:class:`BasePruningMethod`.
Args:
module (nn.Module): object that is either pruned or unpruned
Returns:
binary answer to whether ``module`` is pruned.
Examples:
>>> m = nn.Linear(5, 7)
>>> print(prune.is_pruned(m))
False
>>> prune.random_unstructured(m, name='weight', amount=0.2)
>>> print(prune.is_pruned(m))
True
TF)Z
named_modulesr r r! r )r _ submoduler+ r r r is_pruned s
r c C sf t | tjstdj| t | tjr.| dk sTt | tj rbt| dksTt| dk rbtdj| dS )a Validation helper to check the range of amount at init.
Args:
amount (int or float): quantity of parameters to prune.
If float, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If int, it represents the
absolute number of parameters to prune.
Raises:
ValueError: if amount is a float not in [0, 1], or if it's a negative
integer.
TypeError: if amount is neither a float nor an integer.
Note:
This does not take into account the number of parameters in the
tensor to be pruned, which is known only at prune.
z2Invalid type for amount: {}. Must be int or float.r g ?g zPamount={} should either be a float in the range [0, 1] or a non-negative integerN)r! numbersRealrH r IntegralfloatrJ )rf r r r re s re c C s( t | tjr$| |kr$tdj| |dS )a Validation helper to check that the amount of parameters to prune
is meaningful wrt to the size of the data (`tensor_size`).
Args:
amount (int or float): quantity of parameters to prune.
If float, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If int, it represents the
absolute number of parameters to prune.
tensor_size (int): absolute number of parameters in the tensor
to prune.
zEamount={} should be smaller than the number of parameters to prune={}N)r! r r rJ r )rf rm r r r ri s ri c C s* | j }t|dkr&tdj|t|dS )a# Validation helper to check that the tensor to be pruned is multi-
dimensional, such that the concept of "channels" is well-defined.
Args:
t (torch.Tensor): tensor representing the parameter to prune
Raises:
ValueError: if the tensor `t` is not at least 2D.
r ziStructured pruning can only be applied to multidimensional tensors. Found tensor of shape {} with {} dimsN)r0 rG rJ r )r r0 r r r rx s
rx c C s$ t | tjr| S tt| | S dS )a Since amount can be expressed either in absolute value or as a
percentage of the number of units/channels in a tensor, this utility
function converts the percentage to absolute value to standardize
the handling of pruning.
Args:
amount (int or float): quantity of parameters to prune.
If float, should be between 0.0 and 1.0 and represent the
fraction of parameters to prune. If int, it represents the
absolute number of parameters to prune.
tensor_size (int): absolute number of parameters in the tensor
to prune.
Returns:
int: the number of units to prune in the tensor
N)r! r r intround)rf rm r r r rh s rh c C s" || j kr tdj|| jdS )z
Args:
t (torch.Tensor): tensor representing the parameter to prune
dim (int): index of the dim along which we define channels to prune
z&Invalid index {} for tensor of size {}N)rR rY r r0 )r rR r r r ry ) s ry c C s> t t| j }|dk r || }|j| tj| ||d}|S )a Compute the L_n-norm across all entries in tensor `t` along all dimension
except for the one identified by dim.
Example: if `t` is of shape, say, 3x2x4 and dim=2 (the last dim),
then norm will have Size [4], and each entry will represent the
`L_n`-norm computed using the 3x2=6 entries for each of the 4 channels.
Args:
t (torch.Tensor): tensor representing the parameter to prune
n (int, float, inf, -inf, 'fro', 'nuc'): See documentation of valid
entries for argument p in torch.norm
dim (int): dim identifying the channels to prune
Returns:
norm (torch.Tensor): L_n norm computed across all dimensions except
for `dim`. By construction, `norm.shape = t.shape[-1]`.
r )prR )listr[ rR r> r2 r} )r r{ rR Zdimsr} r r r r| 3 s
r| )N)N)N)!rB r abcr r collections.abcr typingr r2 r r# r` rd rp rs rz r~ r r r r r r r r> r re ri rx rh ry r| r r r r s> 9Af "
)&
,
x! "