/usr/local/lib64/python3.6/site-packages/torch/nn/modules/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/nn/modules/__pycache__/adaptive.cpython-36.pyc (9725B)
3
Eg+ @ s~ d dl mZ d dlZd dlmZ d dlmZmZ ddlmZm Z m
Z
ddl m Z dd l
mZ ed
d d gZG d
d de ZdS ) )
namedtupleN)Tensor)ListSequence )
Sequential
ModuleListLinear)Module ) log_softmax
_ASMoutputoutputlossc s e Zd ZU dZeeee e
e e
edeeee e e dd fdd
Zddd d
Zeeed d d
Zdd ZeedddZeedddZ ZS )AdaptiveLogSoftmaxWithLossu Efficient softmax approximation as described in
`Efficient softmax approximation for GPUs by Edouard Grave, Armand Joulin,
Moustapha Cissé, David Grangier, and Hervé Jégou
`__.
Adaptive softmax is an approximate strategy for training models with large
output spaces. It is most effective when the label distribution is highly
imbalanced, for example in natural language modelling, where the word
frequency distribution approximately follows the `Zipf's law`_.
Adaptive softmax partitions the labels into several clusters, according to
their frequency. These clusters may contain different number of targets
each.
Additionally, clusters containing less frequent labels assign lower
dimensional embeddings to those labels, which speeds up the computation.
For each minibatch, only clusters for which at least one target is
present are evaluated.
The idea is that the clusters which are accessed frequently
(like the first one, containing most frequent labels), should also be cheap
to compute -- that is, contain a small number of assigned labels.
We highly recommend taking a look at the original paper for more details.
* :attr:`cutoffs` should be an ordered Sequence of integers sorted
in the increasing order.
It controls number of clusters and the partitioning of targets into
clusters. For example setting ``cutoffs = [10, 100, 1000]``
means that first `10` targets will be assigned
to the 'head' of the adaptive softmax, targets `11, 12, ..., 100` will be
assigned to the first cluster, and targets `101, 102, ..., 1000` will be
assigned to the second cluster, while targets
`1001, 1002, ..., n_classes - 1` will be assigned
to the last, third cluster.
* :attr:`div_value` is used to compute the size of each additional cluster,
which is given as
:math:`\left\lfloor\frac{\texttt{in\_features}}{\texttt{div\_value}^{idx}}\right\rfloor`,
where :math:`idx` is the cluster index (with clusters
for less frequent words having larger indices,
and indices starting from :math:`1`).
* :attr:`head_bias` if set to True, adds a bias term to the 'head' of the
adaptive softmax. See paper for details. Set to False in the official
implementation.
.. warning::
Labels passed as inputs to this module should be sorted according to
their frequency. This means that the most frequent label should be
represented by the index `0`, and the least frequent
label should be represented by the index `n_classes - 1`.
.. note::
This module returns a ``NamedTuple`` with ``output``
and ``loss`` fields. See further documentation for details.
.. note::
To compute log-probabilities for all classes, the ``log_prob``
method can be used.
Args:
in_features (int): Number of features in the input tensor
n_classes (int): Number of classes in the dataset
cutoffs (Sequence): Cutoffs used to assign targets to their buckets
div_value (float, optional): value used as an exponent to compute sizes
of the clusters. Default: 4.0
head_bias (bool, optional): If ``True``, adds a bias term to the 'head' of the
adaptive softmax. Default: ``False``
Returns:
``NamedTuple`` with ``output`` and ``loss`` fields:
* **output** is a Tensor of size ``N`` containing computed target
log probabilities for each example
* **loss** is a Scalar representing the computed negative
log likelihood loss
Shape:
- input: :math:`(N, \texttt{in\_features})`
- target: :math:`(N)` where each value satisfies :math:`0 <= \texttt{target[i]} <= \texttt{n\_classes}`
- output1: :math:`(N)`
- output2: ``Scalar``
.. _Zipf's law: https://en.wikipedia.org/wiki/Zipf%27s_law
@FN) in_features n_classescutoffs div_value head_biasreturnc
sn ||d}t t| j t|}|t|ksnt|dksnt||d ksntt|t|ksnt dd |D rvt
d|| _ || _ ||g | _
|| _|| _| j
d | _t| j
d | _| j| j | _t| j | jfd| ji|| _t | _x~t| jD ]p} t| j | j| d }
| j
| d | j
| } tt| j |
fddi|t|
| fddi|} | jj| qW d S ) N)devicedtyper r c S s g | ]}t ||kqS )int).0cr r E/usr/local/lib64/python3.6/site-packages/torch/nn/modules/adaptive.py
s z7AdaptiveLogSoftmaxWithLoss.__init__..zcutoffs should be a sequence of unique, positive integers sorted in an increasing order, where each value is between 1 and n_classes-1ZbiasF)superr __init__listsortedminmaxlensetany
ValueErrorr r r r r shortlist_sizeZ
n_clustersZ head_sizer headr tailranger r append)
selfr r r r r r r Zfactory_kwargsiZhszZoszZ
projection) __class__r r r! p s6
z#AdaptiveLogSoftmaxWithLoss.__init__)r c C s2 | j j x"| jD ]\}}|j |j qW d S )N)r+ reset_parametersr, )r/ Zi2hZh2or r r r2 s
z+AdaptiveLogSoftmaxWithLoss.reset_parameters)inputtargetr c C s |j d|j dkr tdd}|j d}|j|}|j|}dg| j }xtt|d D ]}|| } ||d }
|| k||
k @ } | j j } | j dkrq\|dkr|j
d| || nx|| | }
|j d| }| j |d |}| j
| d }|jd| | t|dd}|jd|
jd}|j
d| |jd || j 7 }q\W ||krttdj| jd |j j |j j | j|}t|dd}||jd|jdj 7 }| j }t||S )Nr zBInput and target should have the same size in the batch dimension.r )dimzMTarget values should be in [0, {}], but values in range [{}, {}] were found. )size RuntimeErrorZ new_zeros new_emptyr r- r& ZnonzeroZsqueezeZnumelZ index_copy_Z index_selectr, r* Z index_fill_r gather unsqueezeformatr r$ itemr% r+ Zmeanr
)r/ r3 r4 Z used_rowsZ
batch_sizer Z gather_indsZ
cutoff_valuesr0 Zlow_idxZhigh_idxZ target_maskZ row_indicesZrelative_targetZ input_subsetcluster_outputZ
cluster_indexcluster_logprobZ
local_logprob head_output head_logprobr r r r forward sF
z"AdaptiveLogSoftmaxWithLoss.forwardc C s |j |jd| jf}t|dd}|ddd| jf |ddd| jf< xxtt| j| jdd D ]Z\}\}}| j| |}t|dd} | |dd| j| f j d }
|
|dd||f< qfW |S )za Given input tensor, and output of `self.head`,
compute the log of the full distribution r r )r5 N)
r8 r6 r r r* enumeratezipr r, r: ) r/ r3 r? outr@ r0 Z start_idxZstop_idxr= r> Zoutput_logprobr r r _get_full_log_prob s (( z-AdaptiveLogSoftmaxWithLoss._get_full_log_prob)r3 r c C s | j |}| j||S )a Computes log probabilities for all :math:`\texttt{n\_classes}`
Args:
input (Tensor): a minibatch of examples
Returns:
log-probabilities of for each class :math:`c`
in range :math:`0 <= c <= \texttt{n\_classes}`, where :math:`\texttt{n\_classes}` is a
parameter passed to ``AdaptiveLogSoftmaxWithLoss`` constructor.
Shape:
- Input: :math:`(N, \texttt{in\_features})`
- Output: :math:`(N, \texttt{n\_classes})`
)r+ rE )r/ r3 r? r r r log_prob s
z#AdaptiveLogSoftmaxWithLoss.log_probc C s | j |}tj|dd}|| jk}|j }|r4|S |j rV| j||}tj|ddS | j|| || }tj|dd||< |S dS )a This is equivalent to `self.log_pob(input).argmax(dim=1)`,
but is more efficient in some cases.
Args:
input (Tensor): a minibatch of examples
Returns:
output (Tensor): a class with the highest probability for each example
Shape:
- Input: :math:`(N, \texttt{in\_features})`
- Output: :math:`(N)`
r )r5 N)r+ torchZargmaxr* r( allrE )r/ r3 r? r Znot_in_shortlistZall_in_shortlistrF r r r predict s
z"AdaptiveLogSoftmaxWithLoss.predict)r FNN)__name__
__module__ __qualname____doc__r r r r r floatr boolr r r+ r r, r r! r2 r r
rA rE rF rI
__classcell__r r )r1 r r s$
T *7r ) collectionsr rG r typingr r r r r moduler
Z
functionalr r
r r r r r s