Name	Size	Mode	Actions
chinese.cpython-36.pyc	14710	0644	editdlrm
cli.cpython-36.pyc	6519	0644	editdlrm
corpus.cpython-36.pyc	4915	0644	editdlrm
indic.cpython-36.pyc	949	0644	editdlrm
normalize.cpython-36.pyc	4829	0644	editdlrm
sent_tokenize.cpython-36.pyc	721	0644	editdlrm
subwords.cpython-36.pyc	4650	0644	editdlrm
tokenize.cpython-36.pyc	16593	0644	editdlrm
truecase.cpython-36.pyc	11778	0644	editdlrm
util.cpython-36.pyc	5130	0644	editdlrm
__init__.cpython-36.pyc	299	0644	editdlrm
__main__.cpython-36.pyc	224	0644	editdlrm

3 �Eg!�@s�ydd
lm
Z

Wn ek r0


ddlmZ

YnXddlmZ
ddlmZmZ
ddlm Z m Z
ddlmZ
Gdd�de�Z d d �Zdd�Zd d�Zdd�Zddd�
Zddd�
ZdS)�)
�zip_longest)
�izip_longest)
�tee)�escape�unescape)�Parallel�delayed)
�tqdmc@sZeZ
dZd
ZdZdZdZdZdZd Z d!Z d"Zd#Zd$Z d%Zd&Zeeeeee e ee eegZdS)'�CJKCharsz� An object that enumerates the code points of the CJK characters as listed on http://en.wikipedia.org/wiki/Basic_Multilingual_Plane#Basic_Multilingual_Plane ����.�Ϥ�@�������������0��O��e������o
��o
�p
���
��
�/�
�p�
���
����N)rr)r r)rr)rr)rr)rr)rr)rr)rr)rr)rr )r!r")�__name__� __module__�__qualname__�__doc__ZHangul_JamoZCJK_RadicalsZPhags_PaZHangul_SyllablesZCJK_Compatibility_IdeographsZCJK_Compatibility_FormsZKatakana_Hangul_HalfwidthZ#Ideographic_Symbols_And_PunctuationZTangutZKana_SupplementZNushuZSupplementary_Ideographic_Plane�ranges�r(r(�9/usr/local/lib/python3.6/site-packages/sacremoses/util.pyr s4











r c

st�f
d
d�d$D�
�
S)%u�
This checks for CJK character. >>> CJKChars().ranges [(4352, 4607), (11904, 42191), (43072, 43135), (44032, 55215), (63744, 64255), (65072, 65103), (65381, 65500), (94208, 101119), (110592, 110895), (110960, 111359), (131072, 196607)] >>> is_cjk(u'㏾') True >>> is_cjk(u'﹟') False :param character: The character that needs to be checked. :type character: char :return: bool c
s,g|]$\}
}|
t��
k
o"|k
n
�qSr()
�ord)�.0�start�end)
� characterr(r)� ts
zis_cjk..����.�Ϥ�@�������������0��O��e�����p
���
��
�/�
�p�
���
�����r0r1�r2r3�r4r5�r6r7�r8r9�r:r;�r<r=�r>r?�r@rA�rBrC�rDrE)rFrGrHrIrJrKrLrMrNrOrP)
�any)
r.r()
r.r)�is_cjkcs 









rRc

Cst|d
ddddd�d�S)a� This function transforms the input text into an "escaped" version suitable for well-formed XML formatting. Note that the default xml.sax.saxutils.escape() function don't escape some characters that Moses does so we have to manually add them to the entities dictionary. >>> input_str = ''')| & < > ' " ] [''' >>> expected_output = ''')| & < > ' " ] [''' >>> escape(input_str) == expected_output True >>> xml_escape(input_str) ')| & < > ' " ] [' :param text: The text that needs to be escaped. :type text: str :rtype: str z'z"z|z[z])�
'�
"�
|�
[�
])
�entities)
r)
�textr(r(r)� xml_escape�s




rZc

Cst|d
ddddd�d�S)ai This function transforms the "escaped" version suitable for well-formed XML formatting into humanly-readable string. Note that the default xml.sax.saxutils.unescape() function don't unescape some characters that Moses does so we have to manually add them to the entities dictionary. >>> from xml.sax.saxutils import unescape >>> s = ')| & < > ' " ] [' >>> expected = ''')| & < > ' " ] [''' >>> xml_unescape(s) == expected True :param text: The text that needs to be unescaped. :type text: str :rtype: str rSrTrUrVrW)z'z"z|z[z])
rX)
r)
rYr(r(r)�xml_unescape�s




r[c
Cs t|�
\}
}t
|d
�
t|
|�S)zp From https://docs.python.org/3/library/itertools.html#recipes s -> (s0,s1), (s1,s2), (s2, s3), ... N)r�next�zip)�iterable�
a�
br(r(r)�pairwise�s

raNcCst|�
g
|
}t
|d
|i
�
S)ziCollect data into fixed-length chunks or blocks from https://stackoverflow.com/a/16789869/610569 � fillvalue)�iterr)r^�
nrb�argsr(r(r)�grouper�s
rfFcs>|rt|
�
n|
}
|d
k
r"t
�|
�St|d�
�f
dd�|
D�
�
S)N�
)
Zn_jobsc
3s|]}
t��
|
�
V
qdS)
N)
r)r+�line)
�funcr(r)� �sz)parallelize_preprocess..)r �mapr)ri�iterator� processes�progress_barr()
rir)�parallelize_preprocess�s



ro)
N)
F)� itertoolsr�ImportErrorrr�xml.sax.saxutilsrrZjoblibrrr �objectr rRrZr[rarfror(r(r(r)�s




S#