Name	Size	Mode	Actions
big5freq.cpython-38.opt-1.pyc	27143	0644	editdlrm
big5freq.cpython-38.pyc	27143	0644	editdlrm
big5prober.cpython-38.opt-1.pyc	1098	0644	editdlrm
big5prober.cpython-38.pyc	1098	0644	editdlrm
chardistribution.cpython-38.opt-1.pyc	6184	0644	editdlrm
chardistribution.cpython-38.pyc	6184	0644	editdlrm
charsetgroupprober.cpython-38.opt-1.pyc	2215	0644	editdlrm
charsetgroupprober.cpython-38.pyc	2215	0644	editdlrm
charsetprober.cpython-38.opt-1.pyc	3447	0644	editdlrm
charsetprober.cpython-38.pyc	3447	0644	editdlrm
codingstatemachine.cpython-38.opt-1.pyc	2874	0644	editdlrm
codingstatemachine.cpython-38.pyc	2874	0644	editdlrm
compat.cpython-38.opt-1.pyc	319	0644	editdlrm
compat.cpython-38.pyc	319	0644	editdlrm
cp949prober.cpython-38.opt-1.pyc	1105	0644	editdlrm
cp949prober.cpython-38.pyc	1105	0644	editdlrm
enums.cpython-38.opt-1.pyc	2612	0644	editdlrm
enums.cpython-38.pyc	2612	0644	editdlrm
escprober.cpython-38.opt-1.pyc	2597	0644	editdlrm
escprober.cpython-38.pyc	2597	0644	editdlrm
escsm.cpython-38.opt-1.pyc	7438	0644	editdlrm
escsm.cpython-38.pyc	7438	0644	editdlrm
eucjpprober.cpython-38.opt-1.pyc	2415	0644	editdlrm
eucjpprober.cpython-38.pyc	2415	0644	editdlrm
euckrfreq.cpython-38.opt-1.pyc	12027	0644	editdlrm
euckrfreq.cpython-38.pyc	12027	0644	editdlrm
euckrprober.cpython-38.opt-1.pyc	1106	0644	editdlrm
euckrprober.cpython-38.pyc	1106	0644	editdlrm
euctwfreq.cpython-38.opt-1.pyc	27147	0644	editdlrm
euctwfreq.cpython-38.pyc	27147	0644	editdlrm
euctwprober.cpython-38.opt-1.pyc	1106	0644	editdlrm
euctwprober.cpython-38.pyc	1106	0644	editdlrm
gb2312freq.cpython-38.opt-1.pyc	19071	0644	editdlrm
gb2312freq.cpython-38.pyc	19071	0644	editdlrm
gb2312prober.cpython-38.opt-1.pyc	1114	0644	editdlrm
gb2312prober.cpython-38.pyc	1114	0644	editdlrm
hebrewprober.cpython-38.opt-1.pyc	2987	0644	editdlrm
hebrewprober.cpython-38.pyc	2987	0644	editdlrm
jisfreq.cpython-38.opt-1.pyc	22099	0644	editdlrm
jisfreq.cpython-38.pyc	22099	0644	editdlrm
jpcntx.cpython-38.opt-1.pyc	37572	0644	editdlrm
jpcntx.cpython-38.pyc	37572	0644	editdlrm
langbulgarianmodel.cpython-38.opt-1.pyc	23596	0644	editdlrm
langbulgarianmodel.cpython-38.pyc	23596	0644	editdlrm
langcyrillicmodel.cpython-38.opt-1.pyc	29060	0644	editdlrm
langcyrillicmodel.cpython-38.pyc	29060	0644	editdlrm
langgreekmodel.cpython-38.opt-1.pyc	23554	0644	editdlrm
langgreekmodel.cpython-38.pyc	23554	0644	editdlrm
langhebrewmodel.cpython-38.opt-1.pyc	22181	0644	editdlrm
langhebrewmodel.cpython-38.pyc	22181	0644	editdlrm
langhungarianmodel.cpython-38.opt-1.pyc	23585	0644	editdlrm
langhungarianmodel.cpython-38.pyc	23585	0644	editdlrm
langthaimodel.cpython-38.opt-1.pyc	22160	0644	editdlrm
langthaimodel.cpython-38.pyc	22160	0644	editdlrm
langturkishmodel.cpython-38.opt-1.pyc	22183	0644	editdlrm
langturkishmodel.cpython-38.pyc	22183	0644	editdlrm
latin1prober.cpython-38.opt-1.pyc	3367	0644	editdlrm
latin1prober.cpython-38.pyc	3367	0644	editdlrm
mbcharsetprober.cpython-38.opt-1.pyc	2230	0644	editdlrm
mbcharsetprober.cpython-38.pyc	2230	0644	editdlrm
mbcsgroupprober.cpython-38.opt-1.pyc	1095	0644	editdlrm
mbcsgroupprober.cpython-38.pyc	1095	0644	editdlrm
mbcssm.cpython-38.opt-1.pyc	16718	0644	editdlrm
mbcssm.cpython-38.pyc	16718	0644	editdlrm
sbcharsetprober.cpython-38.opt-1.pyc	2983	0644	editdlrm
sbcharsetprober.cpython-38.pyc	2983	0644	editdlrm
sbcsgroupprober.cpython-38.opt-1.pyc	1593	0644	editdlrm
sbcsgroupprober.cpython-38.pyc	1593	0644	editdlrm
sjisprober.cpython-38.opt-1.pyc	2451	0644	editdlrm
sjisprober.cpython-38.pyc	2451	0644	editdlrm
universaldetector.cpython-38.opt-1.pyc	5795	0644	editdlrm
universaldetector.cpython-38.pyc	5795	0644	editdlrm
utf8prober.cpython-38.opt-1.pyc	1956	0644	editdlrm
utf8prober.cpython-38.pyc	1956	0644	editdlrm
version.cpython-38.opt-1.pyc	403	0644	editdlrm
version.cpython-38.pyc	403	0644	editdlrm
__init__.cpython-38.opt-1.pyc	814	0644	editdlrm
__init__.cpython-38.pyc	814	0644	editdlrm

U <_9Y�0�@s�dZd
dl
Z
d
dlZd
dlZddlmZ
ddlmZmZm Z
ddl mZ
ddlm Z
ddlmZ
dd lmZ
Gd d�de�ZdS)a
Module containing the UniversalDetector detector class, which is the primary class a user of ``chardet`` should use. :author: Mark Pilgrim (initial port to Python) :author: Shy Shalom (original C code) :author: Dan Blanchard (major refactoring for 3.0) :author: Ian Cordasco �N�
)
�CharSetGroupProber)� InputState�LanguageFilter�ProbingState)
�EscCharSetProber)
�Latin1Prober)
�MBCSGroupProber)
�SBCSGroupProberc @sneZ
dZd
ZdZe�d�
Ze�d�
Ze�d�
Z dddd d ddd d�Z ejf
dd�
Z dd�Zdd�Zdd�ZdS)�UniversalDetectoraq
The ``UniversalDetector`` class underlies the ``chardet.detect`` function and coordinates all of the different charset probers. To get a ``dict`` containing an encoding and its confidence, you can simply run: .. code:: u = UniversalDetector() u.feed(some_bytes) u.close() detected = u.result g�������?s[�-�]s(|~{)s[�-�]zWindows-1252zWindows-1250zWindows-1251zWindows-1256zWindows-1253zWindows-1255zWindows-1254zWindows-1257)z iso-8859-1z iso-8859-2z iso-8859-5z iso-8859-6z iso-8859-7z iso-8859-8z iso-8859-9ziso-8859-13cCsNd|_g|_
d|_d|_d|_d|_d|_|
|_t� t �
|_d|_|� �
dS)
N)�_esc_charset_prober�_charset_probers�result�done� _got_data�_input_state� _last_char�lang_filter�logging� getLogger�__name__�logger�_has_win_bytes�reset)�selfr�r�J/opt/alt/python38/lib/python3.8/site-packages/chardet/universaldetector.py�__init__Qs










zUniversalDetector.__init__c
CsVd
dd
d�|_d|_
d|_d|_tj|_d|_|jr>|j� �
|j D]}
|
� �
qDd
S)z� Reset the UniversalDetector and all of its probers back to their initial states. This is called by ``__init__``, so you only need to call this directly in between analyses of different documents. N���encoding� confidence�languageF�)rrrrr� PURE_ASCIIrrrrr )r�proberrrrr^s








zUniversalDetector.resetcCs>|jr d
St
|
�
sd
St|
t�s(t|
�
}
|js�|
�tj�
rJdddd�|_nv|
�tj tj f�
rldddd�|_nT|
�d�
r�dddd�|_n:|
�d �
r�d ddd�|_n |
�tjtjf�
r�dddd�|_d|_|jd d
k r�d|_d
S|j tjk�
r.|j�|
�
�
rtj|_ n*|j tjk�
r.|j�|j|
�
�
r.tj|_ |
dd
�|_|j tjk�
r�|j�
s^t|j�
|_|j�|
�
tjk�r:|jj|j��|jjd�|_d|_n�|j tjk�r:|j�
s�t |j�
g
|_|jt!j"@�
r�|j�#t$��

|j�#t%��

|jD]:}|�|
�
tjk�
r�|j|��|jd�|_d|_
�q&�
q�|j&�|
�
�r:d|_'d
S)a�
Takes a chunk of a document and feeds it through all of the relevant charset probers. After calling ``feed``, you can check the value of the ``done`` attribute to see if you need to continue feeding the ``UniversalDetector`` more data, or if it has made a prediction (in the ``result`` attribute). .. note:: You should always call ``close`` when you're done feeding in your document if ``done`` is not already ``True``. Nz UTF-8-SIG��?�rzUTF-32s��zX-ISO-10646-UCS-4-3412s��zX-ISO-10646-UCS-4-2143zUTF-16Tr �����)(r�len� isinstance� bytearrayr� startswith�codecs�BOM_UTF8r�BOM_UTF32_LE�BOM_UTF32_BE�BOM_LE�BOM_BErrr$�HIGH_BYTE_DETECTOR�search� HIGH_BYTE�ESC_DETECTORrZ ESC_ASCIIrrr�feedrZFOUND_IT�charset_name�get_confidencer"r r rZNON_CJK�appendr r�WIN_BYTE_DETECTORr)rZbyte_strr%rrrr7os�




� 
�

�  

�  

� 

�






�


�








�


zUniversalDetector.feedc
Cst
|jr|j
Sd
|_|js&|j�d�

n�|jtjkrBdddd�|_
n�|jtjkr�d}
d}d}|j D]"}|sjq`|� �}
|
|kr`|
}|}q`|r�||jkr�|j}|j� �}|� �}|�d �
r�|jr�|j�||�}|||jd�|_
|j��tjk�
rn|j
d dk�
rn|j�d�

|j D]`}|�
s�
qt|t��
rP|jD] }|j�d|j|j|� ��
�
q,n|j�d|j|j|� ��
�
q|j
S) z� Stop analyzing the current document and come up with a final prediction. :returns: The ``result`` attribute, a ``dict`` with the keys `encoding`, `confidence`, and `language`. Tzno data received!�asciir&r'rNrziso-8859r z no probers hit minimum thresholdz%s %s confidence = %s)rrrr�debugrrr$r5r r9�MINIMUM_THRESHOLDr8�lowerr,r�ISO_WIN_MAP�getr"�getEffectiveLevelr�DEBUGr*rZprobers) rZprober_confidenceZmax_prober_confidenceZ max_proberr%r8Zlower_charset_namer!Zgroup_proberrrr�close�sj 





� 
















�

�










� 


�zUniversalDetector.closeN)r� __module__�__qualname__�__doc__r>�re�compiler3r6r;r@rZALLrrr7rDrrrrr3s$











�  mr)rGr-rrHZcharsetgroupproberrZenumsrrrZ escproberrZlatin1proberrZmbcsgroupproberr Zsbcsgroupproberr �objectrrrrr�s