/usr/local/lib/python3.6/site-packages/sacremoses/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/sacremoses/__pycache__/truecase.cpython-36.pyc (11778B)
3
EgH @ s d dl mZ d dlZd dlmZmZ d dlmZ d dlm Z d dl
m Z d dl m
Z
d dlmZmZ d dlZejd d k rd dlZd dlZejZejed
e
ZG d d d eZG d
d deZd dgZ dS ) )print_functionN) defaultdictCounter)partial)chain) text_type) Perluniprops)parallelize_preprocessgrouper zTYou should really be using Python3!!! Tick tock, tick tock, https://pythonclock.org/c s e Zd ZdZedjejdZedjejdZ edjejdZ
d$ fdd Z d%d
d Z d&d
dZ
d'ddZd(ddZd)ddZd*ddZd+ddZedd Zd d Zd d Zd d! Zd"d# Z ZS ),MosesTruecaserz
This is a Python port of the Moses Truecaser from
https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/train-truecaser.perl
https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/truecase.perl
Lowercase_LetterUppercase_LetterNutf8c sz t t| j tjdj| j| j| j| _ tjd| _
ddddh| _ ddd d
d d d
dh| _ || _
|| _|rv| j|| _dS )a.
:param load_from:
:type load_from:
:param is_asr: A flag to indicate that model is for ASR. ASR input has
no case, make sure it is lowercase, and make sure known are cased
eg. 'i' to be uppercased even if i is known.
:type is_asr: bool
z[{}{}{}]z (<.*(?<=>))(.*)((?=]*>).:?!(["'z'z"z[z]N)superr __init__recompileformatr r Titlecase_LetterSKIP_LETTERS_REGEXZXML_SPLIT_REGXSENT_ENDDELAYED_SENT_STARTencodingis_asr _load_modelmodel)selfZ load_fromr# r" ) __class__ =/usr/local/lib/python3.6/site-packages/sacremoses/truecase.pyr - s$ zMosesTruecaser.__init__Fc C s d}g }xt |D ]\}}tjd|r*qn || jkr6q| rL|| jkrLd}q| jj|s^d}qd}|sld}n"|r|d j rd}n |dkrd}d}|dkr|j|j ||f qW |S )z
This function checks through each tokens in a sentence and returns the
appropriate weight of each surface token form.
Tz (<\S[^>]*>)Fr g?) enumerater searchr! r r islowerappendlower)r& tokenspossibly_use_first_token
is_first_wordZtruecase_weightsitokenZcurrent_word_weightr( r( r) learn_truecase_weightsT s2
z%MosesTruecaser.learn_truecase_weightsr* c C sl t t}t| j|d}tt||||d }x&|D ] \} }
} || |
| 7 < q0W |rb| j|| | j|S )aN
:param document_iterator: The input document, each outer list is a sentence,
the inner list is the list of tokens for each sentence.
:type document_iterator: iter(list(str))
:param possibly_use_first_token: When True, on the basis that the first
word of a sentence is always capitalized; if this option is provided then:
a) if a sentence-initial token is *not* capitalized, then it is counted, and
b) if a capitalized sentence-initial token is the only token of the segment,
then it is counted, but with only 10% of the weight of a normal token.
:type possibly_use_first_token: bool
:returns: A dictionary of the best, known objects as values from `_casing_to_model()`
:rtype: {'best': dict, 'known': Counter}
)r1 ) progress_bar)r r r r5 r r _save_model_from_casing_casing_to_model) r& document_iteratorsave_tor1 processesr6 casingZtrain_truecaserZ
token_weightsZlowercase_tokenZ
surface_tokenZweightr( r( r) _train s zMosesTruecaser._trainc C s" d| _ | j|||||d| _ | j S )z\
Default duck-type of _train(), accepts list(list(str)) as input documents.
N)r6 )r% r= )r& Z documentsr: r1 r; r6 r( r( r) train s
zMosesTruecaser.trainc C sN t || jd}dd |j D }W dQ R X d| _| j|||||d| _| jS )zj
Duck-type of _train(), accepts a filename to read as a `iter(list(str))`
object.
)r" c s s | ]}|j V qd S )N)split).0liner( r( r)
s z1MosesTruecaser.train_from_file..N)r6 )openr" readlinesr% r= )r& filenamer: r1 r; r6 finr9 r( r( r) train_from_file s
z MosesTruecaser.train_from_filec C s4 dd |j D }d| _| j|||||d| _| jS )zm
Duck-type of _train(), accepts a file object to read as a `iter(list(str))`
object.
c s s | ]}|j V qd S )N)r? )r@ rA r( r( r) rB s z8MosesTruecaser.train_from_file_object..N)r6 )rD r% r= )r& Z file_objectr: r1 r; r6 r9 r( r( r) train_from_file_object s
z%MosesTruecaser.train_from_file_objectc
C s( t d}t| dst|d}g }| j|}xt|D ]\}} tjd| rV|j| q6| dksh| jdrt|j| q6tjd| j \} }
| j
r| j } | j d j
| j d} | j d j
| d} |r| r| } n| r|r| n| } n| r| } | |
} |j| | | jk}| | jkr6d
}q6W |r$d j|S |S ) a
Truecase a single sentence / line of text.
:param text: A single string, i.e. sentence text.
:type text: str
:param use_known: Use the known case if a word is a known word but not the first word.
:type use_known: bool
zV
Use Truecaser.train() to train a model.
Or use Truecaser('modefile') to load a model.r% Tz (<\S[^>]*>)|z
^([^\|]+)(.*)bestNknownF )strhasattrAssertionError split_xmlr+ r r, r.
startswithgroupsr# r/ r% getr r! join)
r& text
return_strZ use_knownZcheck_model_messager2 truecased_tokensr0 r3 r4 Z
other_factorsZ best_caseZ
known_caser( r( r) truecase s<
zMosesTruecaser.truecaseTc
c sN t || jd6}x.|D ]&}| j|j }|r6dj|n|V qW W d Q R X d S )N)r" rL )rC r" rX striprT )r& rE rV rF rA rW r( r( r)
truecase_file> s
z MosesTruecaser.truecase_filec C s0 | j } g }x | r*tjd| }tjd| }tjd| }|r|j \}}tjd| rt|dkrtjd|d r|d
|7 < tjd|}|r|d |jd7 < |jd }n|j|d
|} nP|r|j|jd |jd } n0|r
|j|jd |jd } ntd j| |d j |d< qW |S )a
Python port of split_xml function in Moses' truecaser:
https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/truecaser.perl
:param line: Input string, should be tokenized, separated by space.
:type line: str
z^\s*(<\S[^>]*>)(.*)$z^\s*([^\s<>]+)(.*)$z^\s*(\S+)(.*)$z^\Sr z\|$r* z ^(\|+)(.*)$ rL zERROR: huh? {}r\ r\ r\ r\ ) rY r r, rR lengroupr. Exceptionr )rA r0 Zhas_xmlZ
is_non_xmlZ xml_cognatesZ
potential_xmlZ line_nextZ is_factorr( r( r) rP E s:
zMosesTruecaser.split_xmlc C st i }t }xX|D ]P}|| j }|d d ||< | jsx(|dd D ]\}}|| d7 < qDW qW |||d}|S )zg
:returns: A tuple of the (best, known) objects.
:rtype: tuple(dict, Counter)
r r* N)rJ rK r< )r most_commonr# ) r& r< rJ rK Z token_lowerr0 r4 countr% r( r( r) r8 x s
z MosesTruecaser._casing_to_modelc C s | j | jd | d S )Nr< )r7 r% )r& rE r( r( r)
save_model s zMosesTruecaser.save_modelc
C s t |d| jd}x|D ]~}t|| j }g }xPt|| j D ]<\}\}}|dkrfdj|||} ndj|||} |j| qBW tdj |d|d qW W d Q R X d S )
a"
Outputs the truecaser model file in the same output format as
https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/train-truecaser.perl
:param casing: The dictionary of tokens counter from `train()`.
:type casing: default(Counter)
w)r" r z
{} ({}/{})z{} ({})rL
)endfileN)
rC r" sumvaluesr+ r` r r. printrT )
r& r< rE Zfoutr4 Ztotal_token_countZ
tokens_countsr3 ra Z out_tokenr( r( r) r7 s
z&MosesTruecaser._save_model_from_casingc C s t t}t|| jd`}xX|D ]P}|j j }x>t|dD ]0\}}|jdd jd}t|||j |< q:W q W W dQ R X | j |S )z
Loads pre-trained truecasing file.
:returns: A dictionary of the best, known objects as values from `_casing_to_model()`
:rtype: {'best': dict, 'known': Counter}
)r" r[ /r z()N)
r r rC r" rY r? r
intr/ r8 )r& rE r< rF rA r4 ra r( r( r) r$ s
&zMosesTruecaser._load_model)NNr )F)NFr* F)NFr* F)NFr* F)NFr* F)FF)T)__name__
__module__ __qualname____doc__r rT perlunipropscharsr r r r r5 r= r> rG rH rX rZ staticmethodrP r8 rb r7 r$
__classcell__r( r( )r' r) r ! s: '
1
'
H
3r c s& e Zd Z fddZdddZ ZS )MosesDetruecaserc # s t t| j ddddh| _ddddd d
d d h| _d
ddddddddddddddd d d d d d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/h#| _d S )0Nr r r r r r r r z'z"z[z]aafterZagainstzal-.+andanyasZatbeZbecauseZbetweenZbyZduringzel-.+forfromZhisinisZitslastnotZofoffonZthanZtheZtheirthisZtowaswerewhichZwillwith)r rt r r r! ALWAYS_LOWER)r& )r' r( r) r sZ zMosesDetruecaser.__init__Fc s g }d}x\|j D ]P}|r6|dd j |dd n|}|j| | jkrTd}q| jkrd}qW |r| fdd|D }|rdj|S |S )z
Detruecase the translated files from a model that learnt from truecased
tokens.
:param text: A single string, i.e. sentence text.
:type text: str
TNr* Fc s6 g | ].}| j kr|n|d d j |dd qS )Nr* )r upper)r@ r4 )r& r( r)
s z/MosesDetruecaser.detruecase..rL )r? r r. r r! rT )r& rU Z is_headlinerV Z cased_tokensZsentence_startr4 r( )r& r)
detruecase s $
zMosesDetruecaser.detruecase)FF)rl rm rn r r rs r( r( )r' r) rt s 6rt )
__future__r r collectionsr r functoolsr itertoolsr sixr Zsacremoses.corpusr Zsacremoses.utilr r
sys version_infoiowarningsrC warnrM rp objectr rt __all__r( r( r( r) s, T