/usr/local/lib/python3.6/site-packages/datasets/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/datasets/__pycache__/metric.cpython-36.pyc (23217B)
3
<%Egk @ s. d Z ddlZddlZddlZddlmZmZmZmZm Z m
Z
ddl Z ddl
ZddlmZ ddlmZ ddlmZ ddlmZ dd lmZ dd
lmZ dd lm Z dd l m Z m Z dd
l m!Z! ddl"m#Z#m$Z$m%Z% ddl&m'Z' ddl(m)Z)m*Z* e'e+Z,G dd de#Z-dd Z.G dd dZ/G dd de/Z0dS )z Metrics base class. N)AnyDictListOptionalTupleUnion )config)Dataset) ArrowReader) ArrowWriter)DownloadConfig)DownloadManager)Features) DatasetInfo
MetricInfo)camelcase_to_snakecase) BaseFileLockFileLockTimeout)
get_logger)copyfunc temp_seedc s0 e Zd ZdZ fddZdd Zdd Z ZS ) FileFreeLockz-Thread lock until a file **cannot** be lockedc s" t || _t j|f|| d S )N)r filelocksuper__init__)self lock_fileargskwargs) __class__ 9/usr/local/lib/python3.6/site-packages/datasets/metric.pyr - s
zFileFreeLock.__init__c C sH y| j jddd W n tk
r2 | j j| _Y nX | j j d | _d S )Ng{Gz?g{Gz?)timeoutZpoll_intervall)r acquirer r
_lock_file_fdrelease)r r" r" r# _acquire1 s
zFileFreeLock._acquirec C s
d | _ d S )N)r& )r r" r" r# _release< s zFileFreeLock._release)__name__
__module__ __qualname____doc__r r( r)
__classcell__r" r" )r! r# r * s r c C sR t | tk st| dkr | S dd }d|| d d d|| dd dS ) N c S s dj dd | D S )Nz, c s s | ]}t |V qd S )N)repr).0xr" r" r#
G s z?summarize_if_long_list..format_chunk..)join)chunkr" r" r# format_chunkF s z,summarize_if_long_list..format_chunk[ z, ..., ])typelistlen)objr6 r" r" r# summarize_if_long_listB s r? c @ s e Zd ZdZedddZedd Zeeddd Z ee
e dd
d Z eedd d
Z eedddZ
eedddZeedddZee
e dddZeedddZee
ee dddZee
ee dddZeedd d Zee
e dd d Zd S )!MetricInfoMixinzoThis base class exposes some attributes of MetricInfo
at the base level of the Metric for easy access.
)infoc C s
|| _ d S )N) _metric_info)r rA r" r" r# r Q s zMetricInfoMixin.__init__c C s | j S )zN:class:`datasets.MetricInfo` object containing all the metadata in the metric.)rB )r r" r" r# rA T s zMetricInfoMixin.info)returnc C s | j jS )N)rB metric_name)r r" r" r# nameY s zMetricInfoMixin.namec C s | j jS )N)rB
experiment_id)r r" r" r# rF ] s z MetricInfoMixin.experiment_idc C s | j jS )N)rB description)r r" r" r# rG a s zMetricInfoMixin.descriptionc C s | j jS )N)rB citation)r r" r" r# rH e s zMetricInfoMixin.citationc C s | j jS )N)rB features)r r" r" r# rI i s zMetricInfoMixin.featuresc C s | j jS )N)rB inputs_description)r r" r" r# rJ m s z"MetricInfoMixin.inputs_descriptionc C s | j jS )N)rB homepage)r r" r" r# rK q s zMetricInfoMixin.homepagec C s | j jS )N)rB license)r r" r" r# rL u s zMetricInfoMixin.licensec C s | j jS )N)rB
codebase_urls)r r" r" r# rM y s z MetricInfoMixin.codebase_urlsc C s | j jS )N)rB reference_urls)r r" r" r# rN } s z MetricInfoMixin.reference_urlsc C s | j jS )N)rB
streamable)r r" r" r# rO s zMetricInfoMixin.streamablec C s | j jS )N)rB format)r r" r" r# rP s zMetricInfoMixin.formatN)r* r+ r, r- r r propertyrA strrE r rF rG rH r rI rJ rK rL r rM rN boolrO rP r" r" r" r# r@ L s6 r@ c @ sH e Zd ZdZd1ee eee eeee ee eeee f d d d
Z
d d Z d
d Z dd Z
d2eeef dddZeee ee f dddZdd Zdd Zdd Zddd ee dd d Zddd d d Zddd!d"d#Zd3d$d%Zedd&d'Z d4ee ee d(d)d*Z d+d, Z ddd e!ee"f dd-d.Z#d/d0 Z$dS )5Metricar A Metric is the base class and common API for all metrics.
Args:
config_name (``str``): This is used to define a hash specific to a metrics computation script and prevents the metric's data
to be overridden when the metric loading script is modified.
keep_in_memory (:obj:`bool`): keep all predictions and references in memory. Not possible in distributed settings.
cache_dir (``str``): Path to a directory in which temporary prediction/references data will be stored.
The data directory should be located on a shared file-system in distributed setups.
num_process (``int``): specify the total number of nodes in a distributed settings.
This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).
process_id (``int``): specify the id of the current process in a distributed setup (between 0 and num_process-1)
This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).
seed (:obj:`int`, optional): If specified, this will temporarily set numpy's random seed when :func:`datasets.Metric.compute` is run.
experiment_id (``str``): A specific experiment id. This is used if several distributed evaluations share the same file system.
This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).
max_concurrent_cache_files (``int``): Max number of concurrent metrics cache files (default 10000).
timeout (``Union[int, float]``): Timeout in second for distributed setting synchronization.
NFr r ' d ) config_namekeep_in_memory cache_dir num_process
process_idseedrF max_concurrent_cache_filesr$ c
K s |pd| _ | j } t| jj| _| j | _ |p.d| _tj| | t |t
sR|dk rZt dt |t
sn||krvt d|r|dkrt d|| _ || _
|| _|| _tjj|ptj| _| j | _|d krtjj ^} }}
} |
dk r||
n|d | _n|| _| | _ t j t | j | | _ t j t | j!| | _!t j t | j"| | _"| j j# j$| j%j&7 _$| j!j# j$| j%j&7 _$| j"j# j$| j%j&7 _$d | _'d | _(d | _)d | _*d | _+d | _,d | _-d | _.d | _/d S ) NdefaultZdefault_experimentr z.'process_id' should be a number greater than 0z8'num_process' should be a number greater than process_idr zPUsing 'keep_in_memory' is not possible in distributed setting (num_process > 1).ip )0rW _infor r! r* rD rF r@ r
isinstanceint
ValueErrorrZ r[ r] rX ospath
expanduserr ZHF_METRICS_CACHE_data_dir_root_build_data_dirdata_dirnprandomZ get_stater\ r$ types
MethodTyper compute add_batchadd__func__r- rA rJ
buf_writerwriterwriter_batch_sizedatacache_file_namer rendez_vous_lock
file_paths filelocks)r rW rX rY rZ r[ r\ rF r] r$ r rA _posr" r" r# r sL
zMetric.__init__c C s | j dkrdS t| j S )zReturn the number of examples (predictions or predictions/references pair)
currently stored in the metric's cache.
Nr )rr r= )r r" r" r# __len__ s zMetric.__len__c C s( d| j d| j d| j dt| d S )NzMetric(name: "z
", features: z , usage: """z""", stored examples: ))rE rI rJ r= )r r" r" r# __repr__ s zMetric.__repr__c C s, | j }tjj|| j| j}tj|dd |S )a Path of this metric in cache_dir:
Will be:
self._data_dir_root/self.name/self.config_name/self.hash (if not none)/
If any of these element is missing or if ``with_version=False`` the corresponding subfolders are dropped.
T)exist_ok)rf rc rd r4 rE rW makedirs)r Zbuilder_data_dirr" r" r# rg s zMetric._build_data_dir)rC c C s t jj| j| j d| j d| j d}d}xt| jD ]}t |d }y|j
|d W n t k
r | jdkrt d| d| j d d|| jd krt d
| j d dt
tj }t jj| j| j d| d| j d| j d}Y q8X P q8W ||fS ) zTCreate a new cache file. If the default cache file is used, we generated a new hash.-z.arrowNz.lock)r$ r z^Error in _create_cache_file: another metric instance is already using the local cache file at z.. Please specify an experiment_id (currently: z:) to avoid collision between distributed metric instances.zCannot acquire lock, too many metric instance are operating concurrently on this file system.You should set a larger value of max_concurrent_cache_files when creating the metric (current value is z).)rc rd r4 rh rF rZ r[ ranger] r r% r rb rR uuiduuid4)r r$ file_pathr iZ file_uuidr" r" r# _create_cache_file s( (
.zMetric._create_cache_filec
s j dkr& jdkr td jg}n fddt j D }g }xt|D ]t\}}|dkrj|j j qLt|d }y|j j d W n, t
k
r td | d
| d dY qLX |j| qLW ||fS ) zGet a lock on all the cache files in a distributed setup.
We wait for timeout second to let all the distributed node finish their tasks (default is 100 seconds).
r Nz|Metric cache file doesn't exist. Please make sure that you call `add` or `add_batch` at least once before calling `compute`.c
s2 g | ]*}t jj j j d j d | dqS )r z.arrow)rc rd r4 rh rF rZ )r1 r[ )r r" r#
( s z/Metric._get_all_cache_files..r z.lock)r$ z#Cannot acquire lock on cached file z
for process .) rZ ru rb r enumerateappendr r r% r$ r )r rw rx r[ r r r" )r r# _get_all_cache_files s(
zMetric._get_all_cache_filesc
s| fddt jD }x^|D ]V}t|}y|j jd W n. tk
rj td| d j dd Y q X |j q W d S )Nc
s2 g | ]*}t jj j j d j d | dqS )r z .arrow.lock)rc rd r4 rh rF rZ )r1 r[ )r r" r# r @ s z5Metric._check_all_processes_locks..)r$ z Expected to find locked file z from process z but it doesn't exist.) r rZ r r% r$ r rb r[ r' )r Zexpected_lock_file_namesexpected_lock_file_name
nofilelockr" )r r# _check_all_processes_locks> s
z!Metric._check_all_processes_locksc C s t jj| j| j d| j d}t|}y|j| jd W n. t k
rh t
d| d| j dd Y n
X |j t jj| j| j d| j d}t
|}y|j| jd W n. t k
r t
d| d| j d d Y n
X |j d S )
Nr z
-0.arrow.lock)r$ z Expected to find locked file z from process z but it doesn't exist.z -rdv.lockzCouldn't acquire lock on r )rc rd r4 rh rF rZ r r% r$ r rb r[ r' r )r r r Zlock_file_namerv r" r" r# _check_rendez_vousN s
zMetric._check_rendez_vousc C s | j dk r| j j d| _ | jdk r8| jdkr8| jj | jrht| jt| j dd}t
j | j j
| _nv| jdkr| j \}}y4tdt| j dd}t
f |jdd |D | _W n tk
r tddY nX || _|| _dS ) zClose all the writing process and load/gather the data
from all the nodes if main node or all_process is True.
Nr )rI )rd rA c S s g | ] }d |iqS )filenamer" )r1 fr" r" r# r y s z$Metric._finalize..zError in finalize: another metric instance is already using the local cache file. Please specify an experiment_id to avoid collision between distributed metric instances.)rr finalizer r[ r' rX r rh r rI r
from_bufferrq getvaluert r Z
read_filesFileNotFoundErrorrb rw rx )r readerrw rx r" r" r# _finalizeb s&
zMetric._finalize) predictions
referencesc s ||d |dkr>|dkr> fddj D } j| n2 fddj D }|rptd| dtj fd dj D }fd
dD }td d |j D rjf | j d_d_ j
d
krj j j
jd fddj D }tj jf ||}W dQ R X jdk rDd_` d_ n\xZtttjjD ]@\} }
tj
d|
` d_ `d_tj|
| j q\W |S dS dS )ai Compute the metrics.
Usage of positional arguments is not allowed to prevent mistakes.
Args:
predictions (list/array/tensor, optional): Predictions.
references (list/array/tensor, optional): References.
**kwargs (optional): Keyword arguments that will be forwarded to the metrics :meth:`_compute`
method (see details in the docstring).
Return:
dict or None
- Dictionary with the metrics if this metric is run on the main process (``process_id == 0``).
- None if the metric is not run on the main process (``process_id != 0``).
Example:
```py
>>> from datasets import load_metric
>>> metric = load_metric("accuracy")
>>> accuracy = metric.compute(predictions=model_prediction, references=labels)
```
)r r Nc s i | ]}| krd |qS )Nr" )r1 k)
all_kwargsr" r#
s z"Metric.compute..c s g | ]}| kr|qS r" r" )r1 r )r r" r# r s z"Metric.compute..zMetric inputs are missing: z. All required inputs are c s i | ]} | |qS r" r" )r1
input_name)r r" r# r s c s i | ]}|j kr | |qS r" )rI )r1 r )r r r" r# r s c s s | ]}|d k V qd S )Nr" )r1 vr" r" r# r3 s z!Metric.compute..r )r; c s i | ]} j | |qS r" )rt )r1 r )r r" r# r s z Removing ) rI updaterb r< anyvaluesrn r ru r r[ rt Z
set_formatrA rP r r\ _computerq reversedziprx rw loggerrr rc remover' ) r r r r Zmissing_kwargsZmissing_inputsinputsZcompute_kwargsoutputr r r" )r r r r# rm sD
zMetric.computec s| fdd|D }|r0t d| dtj ||d| fddjD jjj jdkrrj yjj W n tj k
rv t
fd d
D rt t fd d D d }d
dt
d| dt
| d }nftjddgkrHdj d}dj fdd
jD }||7 }n dj dt| dt| }t |dY nX dS )a Add a batch of predictions and references for the metric's stack.
Args:
predictions (list/array/tensor, optional): Predictions.
references (list/array/tensor, optional): References.
Example:
```py
>>> from datasets import load_metric
>>> metric = load_metric("accuracy")
>>> metric.add_batch(predictions=model_prediction, references=labels)
```
c s g | ]}| j kr|qS r" )rI )r1 r )r r" r# r s z$Metric.add_batch..zBad inputs for metric: z. All required inputs are )r r c s i | ]} | |qS r" r" )r1 intput_name)batchr" r# r s z$Metric.add_batch..Nc 3 s. | ]&}t | t tt j kV qd S )N)r= nextiterr )r1 c)r r" r# r3 s z#Metric.add_batch..c s( g | ] }t | t kr|qS r" )r= )r1 r )r col0r" r# r s r zMismatch in the number of z (z) and r| r r z@Metric inputs don't match the expected format.
Expected format: z,
c 3 s& | ] }d | dt | V qdS )zInput z: N)r? )r1 r )r r" r# r3 s zPPredictions and/or references don't match the expected format.
Expected format: z,
Input predictions: z,
Input references: )rb r< rI rA Z encode_batchrr _init_writerZ write_batchpa ArrowInvalidr r r r= sortedr4 r? )r r r r
bad_inputsZbad_col error_msgerror_msg_inputsr" )r r r r# rn s,
0
zMetric.add_batch)
prediction referencec s fdd|D }|r0t d| dtj ||d| fddjD jjj jdkrrj yjj W nP tj k
r d j d
}d
j
fd d jD }||7 }t |dY nX dS )
a Add one prediction and reference for the metric's stack.
Args:
prediction (list/array/tensor, optional): Predictions.
reference (list/array/tensor, optional): References.
Example:
```py
>>> from datasets import load_metric
>>> metric = load_metric("accuracy")
>>> metric.add(predictions=model_predictions, references=labels)
```
c s g | ]}| j kr|qS r" )rI )r1 r )r r" r# r s z Metric.add..zBad inputs for metric: z. All required inputs are )r r c s i | ]} | |qS r" r" )r1 r )exampler" r# r
s z Metric.add..Nz@Metric inputs don't match the expected format.
Expected format: z,
c 3 s& | ] }d | dt | V qdS )zInput z: N)r? )r1 r )r r" r# r3 s z Metric.add..) rb r< rI rA Zencode_examplerr r writer r r4 )r r r r r r r r" )r r r# ro s
z
Metric.addc
C s, | j dkr| jdkrtjj| j| j d| j d}t|| _y| jj |d W n. t
k
r~ t d| d| j dd Y nX | j rt
j | _t| jj| j| jd | _nJd | _| jd ks| jd kr| j \}}|| _|| _t| jj| j| jd
| _| j dkr(| jdkr | j | jj n| j d S ) Nr r r z -rdv.lock)r$ zXError in _init_writer: another metric instance is already using the local cache file at z.. Please specify an experiment_id (currently: z:) to avoid collision between distributed metric instances.)rI streamrs )rI rd rs )rZ r[ rc rd r4 rh rF r rv r% TimeoutErrorrb rX r ZBufferOutputStreamrq r rA rI rs rr ru r r r r' r )r r$ r ru r r" r" r# r s4
zMetric._init_writerc C s t dS )a Construct the MetricInfo object. See `MetricInfo` for details.
Warning: This function is only called once and the result is cached for all
following .info() calls.
Returns:
info: (MetricInfo) The metrics information
N)NotImplementedError)r r" r" r# r_ B s z Metric._info)download_config
dl_managerc C sN |dkr@|dkr.t }tjj| jd|_d|_t| j|| jd}| j | dS )a Downloads and prepares dataset for reading.
Args:
download_config (:class:`DownloadConfig`, optional): Specific download configuration parameters.
dl_manager (:class:`DownloadManager`, optional): Specific download manager to use.
NZ downloadsF)Z dataset_namer rh )
r
rc rd r4 rh rY Zforce_downloadr rE _download_and_prepare)r r r r" r" r# download_and_prepareM s zMetric.download_and_preparec C s dS )aY Downloads and prepares resources for the metric.
This is the internal implementation to overwrite called when user calls
`download_and_prepare`. It should download all required resources for the metric.
Args:
dl_manager (:class:`DownloadManager`): `DownloadManager` used to download and cache data.
Nr" )r r r" r" r# r d s z Metric._download_and_preparec K s t dS )zEThis method defines the common API for all the metrics in the libraryN)r )r r r r r" r" r# r o s zMetric._computec C s\ t | dr | jd k r | jj t | dr<| jd k r<| jj t | drJ| `t | drX| `d S )Nr rv rr rt )hasattrr r' rv rr rt )r r" r" r# __del__s s
zMetric.__del__) NFNr r NNrU rV )r )r )NN)%r* r+ r, r- r rR rS ra r floatr r{ r} rg r r r r r r r r dictrm rn ro r r r_ r
r r r r r r r r" r" r" r# rT s8 8: ! #"G/!
'
rT )1r- rc rk r typingr r r r r r Znumpyri Zpyarrowr r r Z
arrow_datasetr
Z arrow_readerr Z arrow_writerr Zdownload.download_configr
Zdownload.download_managerr rI r rA r r Znamingr Zutils.filelockr r r Z
utils.loggingr Zutils.py_utilsr r r* r r r? r@ rT r" r" r" r# s.
>