Name	Size	Mode	Actions
chinese.cpython-36.pyc	14710	0644	editdlrm
cli.cpython-36.pyc	6519	0644	editdlrm
corpus.cpython-36.pyc	4915	0644	editdlrm
indic.cpython-36.pyc	949	0644	editdlrm
normalize.cpython-36.pyc	4829	0644	editdlrm
sent_tokenize.cpython-36.pyc	721	0644	editdlrm
subwords.cpython-36.pyc	4650	0644	editdlrm
tokenize.cpython-36.pyc	16593	0644	editdlrm
truecase.cpython-36.pyc	11778	0644	editdlrm
util.cpython-36.pyc	5130	0644	editdlrm
__init__.cpython-36.pyc	299	0644	editdlrm
__main__.cpython-36.pyc	224	0644	editdlrm

3 �Eg��@s:dd
lZdd
l
Z
ddlmZ
ddlmZ
Gdd�d�Zd
S)�N)
� text_type)
�chainc%@s
eZ
dZd
Zd�d�d�d�d�d�d�d�d�d�g Zd�d�gZd�d�d�d�d�d�d�d�d�d�d�d�d�d�d�gZd�d�d�d�d�d�gZd�d�d�d�d�d�d�d�d�d�g Zd�g
Z d�d�gZ d�g
Zd�g
Zd�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�d�g$Z d�dd��
Zd�d��Zd�d��Zd�d��Zd�S)��MosesPunctNormalizerz� This is a Python port of the Moses punctuation normalizer from https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/normalize-punctuation.perl �\r��\(� (�\)�) � +�
� \) ([.!:?;,])�)\g<1>�\( �
(� \)�
)�(\d) %�\g<1>%� :�
:� ;�
;�
`�
'�''� " �„�
"�“�”�–�
-�—� - �´�([a-zA-Z])‘([a-zA-Z])�\g<1>'\g<2>�([a-zA-Z])’([a-zA-Z])�‘�‚�’�´´�…�...� « �« �«� » � »�»� %�
%�nº �nº � :� ºC� ºC� cm� cm� \?�
?� \!�
!� ;�, �, �"([,.]+)�\g<1>"�,"�",�(\.+)"(\s*[^<])�"\g<1>\g<2>� (\d) (\d)�\g<1>,\g<2>�\g<1>.\g<2>�，�
,�。\s*�. �、�∶�：�？�《�》�）�！�（�；�」�「�０�
0�１�
1�２�
2�３�
3�４�
4�５�
5�６�
6�７�
7�８�
8�９�
9�．\s*�～�
~�━�〈�
<�〉�
>�【�
[�】�
]�％�enTFcCs�|j|j
|j|jg|_|r*|jjd
|j�
|r\|
dkrF|jj|j�

n|
d kr\|jj|j �

|r�|
d krx|jj|j �

n|jj|j�

tt |j��
|_||_||_dS)ah
:param language: The two-letter language code. :type lang: str :param penn: Normalize Penn Treebank style quotations. :type penn: bool :param norm_quote_commas: Normalize quotations and commas :type norm_quote_commas: bool :param norm_numbers: Normalize numbers :type norm_numbers: bool �
r�de�es�fr�cz�csN)r�r�r�)r�r�r�r�r�)�EXTRA_WHITESPACE�NORMALIZE_UNICODE� FRENCH_QUOTES�HANDLE_PSEUDO_SPACES� substitutions�insert�NORMALIZE_UNICODE_IF_NOT_PENN�append�EN_QUOTATION_FOLLOWED_BY_COMMA�$DE_ES_FR_QUOTATION_FOLLOWED_BY_COMMA�DE_ES_CZ_CS_FR�OTHER�listr�pre_replace_unicode_punct�post_remove_control_chars)�self�langZpennZnorm_quote_commasZnorm_numbersr�r��r��>/usr/local/lib/python3.6/site-packages/sacremoses/normalize.py�__init__�s$










zMosesPunctNormalizer.__init__cCsN|jr|j
|
�
}
x$|jD]\}}tj||t|
�
�}
qW|jrF|j|
�
}
|
j�S)
z? Returns a string with normalized punctuation. ) r��replace_unicode_punctr��re�subrr��remove_control_chars�strip)r��text�regexp�substitutionr�r�r�� normalize�s

zMosesPunctNormalizer.normalizecCs*x$|jD]\}}t
j||t|
�
�}
qW|
S)
N)�REPLACE_UNICODE_PUNCTUATIONr�r�r)r�r�r�r�r�r�r�r��s


z*MosesPunctNormalizer.replace_unicode_punctcCstj
d
d|
�S)Nz\p{C}r)�regexr�)r�r�r�r�r�r��s
z)MosesPunctNormalizer.remove_control_charsN)rr)rr)r r )rr)r r)rr)rr)rr)rr)rr)rr)rr)rr)rr)r r)r!r")r#r$)rr)r%r)r&r')r(r')r)r)r*r)r+r)rr)r,r)r-r.)r/r)r0r)r1r)r2r)r3r)r4r)r5r6)r7r8)r9r)r:r;)r<r=)r>r?)r@rA)rBr)rCrD)rr)rErF)rGrH)rIrJ)rKrL)rKrM)rNrO)rPrQ)rRrO)r r)rr)rSr)rTr)rUr?)rVr)rWr)rXr)rYrA)rZr)r[r)r\r)r]r)r^r_)r`ra)rbrc)rdre)rfrg)rhri)rjrk)rlrm)rnro)rprq)rrrQ)rsrt)r+r)r-r.)rur")rvrw)rxry)rzr{)r|r})r~r6)rTTTFF)�__name__� __module__�__qualname__�__doc__r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�rs�













































































'r)r�r��sixr� itertoolsrrr�r�r�r��s