/usr/local/lib/python3.6/site-packages/datasets/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/datasets/__pycache__/fingerprint.cpython-36.pyc (18979B)
3
<%EgW @ s8 U d dl Z d dlZd dlZd dlZd dlZd dlZd dlZd dlmZ d dl m
Z
d dl m Z m
Z
mZmZmZmZ d dlZd dlZd dlZddlmZ ddlmZ ddlm Z m Z m Z m Z dd l m!Z! dd
l"m#Z# dd l$m%Z%m&Z& e rdd l'm(Z( e#e)Z*d
a+da,ed ,da-eej. -G dd dZ/dd Z0dd Z1dd Z2dd Z3e!de4dddZ5e4d d d Z6e7d d d Z8d!d" Z9G d#d$ d$Z:e9ej e e e e d%d& Z;e9ed'd( Z
d7e7d d,d-Z?d.d/ Z@d8e7d0d1d2ZAd9e4eee7 eee7 eee7 e4ee7 d4d5d6ZBdS ): N)wraps)Path)
TYPE_CHECKINGAnyDictListOptionalUnion ) DatasetInfo)"INVALID_WINDOWS_CHARACTERS_IN_PATH)ConcatenationTable
InMemoryTableMemoryMappedTableTable)
deprecated)
get_logger)asdictdumps)DatasetT_TempDirWithCustomCleanupc @ s* e Zd ZdZd ddZdd Zdd ZdS )
r z
A temporary directory with a custom cleanup function.
We need a custom temporary directory cleanup in order to delete the dataset objects that have
cache files in the temporary directory before deleting the dorectory itself.
Nc O s0 t j | _tj| | j| _|| _|| _|| _ d S )N)
tempfilemkdtempnameweakreffinalize_cleanup
_finalizer
_cleanup_func_cleanup_func_args_cleanup_func_kwargs)self cleanup_funcZcleanup_func_argsZcleanup_func_kwargs r# >/usr/local/lib/python3.6/site-packages/datasets/fingerprint.py__init__8 s
z"_TempDirWithCustomCleanup.__init__c C s. | j | j| j tjj| jr*tj| j d S )N) r r r ospathexistsr shutilrmtree)r! r# r# r$ r ? s z"_TempDirWithCustomCleanup._cleanupc C s | j j r| j d S )N)r detachr )r! r# r# r$ cleanupD s
z!_TempDirWithCustomCleanup.cleanup)N)__name__
__module__ __qualname____doc__r% r r, r# r# r# r$ r 1 s
c C s> t dkr dS tdkr tj atdd | jD r:tj| dS )a
This function registers the datasets that have cache files in _TEMP_DIR_FOR_TEMP_CACHE_FILES in order
to properly delete them before deleting the temporary directory.
The temporary directory _TEMP_DIR_FOR_TEMP_CACHE_FILES is used when caching is disabled.
Nc s s& | ] }t tjt |d jkV qdS )filenameN)r _TEMP_DIR_FOR_TEMP_CACHE_FILESr parents).0
cache_filer# r# r$ V s z?maybe_register_dataset_for_temp_dir_deletion..)r2 _DATASETS_WITH_TABLE_IN_TEMP_DIRr WeakSetany cache_filesadd)datasetr# r# r$ ,maybe_register_dataset_for_temp_dir_deletionI s r= c C s t d k rtt S g S )N)r7 listr# r# r# r$ (get_datasets_with_cache_file_in_temp_dir\ s r? c C s da dS )a
When applying transforms on a dataset, the data are stored in cache files.
The caching mechanism allows to reload an existing cache file if it's already been computed.
Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated
after each transform.
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
More precisely, if the caching is disabled:
- cache files are always recreated
- cache files are written to a temporary directory that is deleted when session closes
- cache files are named using a random hash instead of the dataset fingerprint
- use :func:`datasets.Dataset.save_to_disk` to save a transformed dataset or it will be deleted when session closes
- caching doesn't affect :func:`datasets.load_dataset`. If you want to regenerate a dataset from scratch you should use
the ``download_mode`` parameter in :func:`datasets.load_dataset`.
TN)_CACHING_ENABLEDr# r# r# r$ enable_caching` s rA c C s da dS )a
When applying transforms on a dataset, the data are stored in cache files.
The caching mechanism allows to reload an existing cache file if it's already been computed.
Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated
after each transform.
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
More precisely, if the caching is disabled:
- cache files are always recreated
- cache files are written to a temporary directory that is deleted when session closes
- cache files are named using a random hash instead of the dataset fingerprint
- use :func:`datasets.Dataset.save_to_disk` to save a transformed dataset or it will be deleted when session closes
- caching doesn't affect :func:`datasets.load_dataset`. If you want to regenerate a dataset from scratch you should use
the ``download_mode`` parameter in :func:`datasets.load_dataset`.
FN)r@ r# r# r# r$ disable_cachingu s rB zUse datasets.enable_caching() or datasets.disable_caching() instead. This function will be removed in a future version of datasets.)booleanc C s t | adS )a
When applying transforms on a dataset, the data are stored in cache files.
The caching mechanism allows to reload an existing cache file if it's already been computed.
Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated
after each transform.
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
More precisely, if the caching is disabled:
- cache files are always recreated
- cache files are written to a temporary directory that is deleted when session closes
- cache files are named using a random hash instead of the dataset fingerprint
- use :func:`datasets.Dataset.save_to_disk` to save a transformed dataset or it will be deleted when session closes
- caching doesn't affect :func:`datasets.load_dataset`. If you want to regenerate a dataset from scratch you should use
the ``download_mode`` parameter in :func:`datasets.load_dataset`.
N)boolr@ )rC r# r# r$ set_caching_enabled s rE )returnc C s t tS )a
When applying transforms on a dataset, the data are stored in cache files.
The caching mechanism allows to reload an existing cache file if it's already been computed.
Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated
after each transform.
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
More precisely, if the caching is disabled:
- cache files are always recreated
- cache files are written to a temporary directory that is deleted when session closes
- cache files are named using a random hash instead of the dataset fingerprint
- use :func:`datasets.Dataset.save_to_disk` to save a transformed dataset or it will be deleted when session closes
- caching doesn't affect :func:`datasets.load_dataset`. If you want to regenerate a dataset from scratch you should use
the ``download_mode`` parameter in :func:`datasets.load_dataset`.
)rD r@ r# r# r# r$ is_caching_enabled s rG c C s t dkrdd } t| da t jS )z7Return a directory that is deleted when session closes.Nc S s xt D ] } | j qW d S )N)r? __del__)Zdsetr# r# r$ r" s z9get_temporary_cache_files_directory..cleanup_func)r" )r2 r r )r" r# r# r$ #get_temporary_cache_files_directory s
rI c s fdd}|S )Nc s x D ]}| t j|< qW | S )N)Hasherdispatch)funct)typesr# r$ proxy s
zhashregister..proxyr# )rN rO r# )rN r$ hashregister s rP c @ s e Zd ZU dZi Zedd Zeee e
e f e dddZ ee
e dddZee
e dd d
Ze
d dd d
Ze dddZd S )rJ z-Hasher that accepts python objects as inputs.c C s t j | _d S )N)xxhashxxh64m)r! r# r# r$ r% s zHasher.__init__)valuerF c C s< t |tr|gn|}tj }x|D ]}|j| q"W |j S )N)
isinstancebytesrQ rR update hexdigest)clsrT rS xr# r# r$
hash_bytes s
zHasher.hash_bytesc C s | j t|S )N)r[ r )rY rT r# r# r$ hash_default s zHasher.hash_defaultc C s0 t || jkr"| jt | | |S | j|S d S )N)typerK r\ )rY rT r# r# r$ hash s z Hasher.hashNc C sB dt | d}| j|}| jj|jd | jj|jd d S )Nz==utf8zutf-8)r] r^ rS rW encode)r! rT Zheader_for_updateZvalue_for_updater# r# r$ rW s
z
Hasher.update)rF c C s
| j j S )N)rS rX )r! r# r# r$ rX s zHasher.hexdigest)r- r. r/ r0 rK r r% classmethodr rV r strr[ r r\ r^ rW rX r# r# r# r$ rJ s
rJ c s< fdd dj fddtjD jjdS )Nc s: t | tjr" jdd | jD S j| j jdS d S )Nc s s | ]}|j jd V qdS )zutf-8N) to_stringr` )r4 cr# r# r$ r6 s z9_hash_pa_table.._hash_pa_array..zutf-8)rU paZ ChunkedArrayr[ chunksrc r` )rT )hasherr# r$ _hash_pa_array s z&_hash_pa_table.._hash_pa_array-c 3 s" | ]}|d | V qdS )ri Nr# )r4 col)rh rT r# r$ r6 s z!_hash_pa_table..zutf-8)joinsortedZ column_namesr[ r` )rg rT r# )rh rg rT r$ _hash_pa_table s rm c C s | j tjt|ddjdS )NT) sort_keyszutf-8)r[ jsonr r r` )rg rT r# r# r$ _hash_dataset_info
s rp c C sn | j }t }x2t|D ]&}|dkr$q|j| |j|| qW x$| jD ]}|jtjj|d qHW |j S )N _fingerprintr1 ) __dict__rJ rl rW r: r& r' getmtimerX )r< staterg keyr5 r# r# r$ generate_fingerprint s
rv @ c C s t j| d| d dS )N0 rZ )random getrandbits)nbitsr# r# r$ generate_random_fingerprint% s r} c C sF t }|j| y|j| W n^ trftjddsRtjd| d dtd< qxtjd| d ntjd| d t S xt |D ]}|j| y|j|| W q trtjddstjd| d || d
| d dtd< n"tjd| d || d
| d n"tjd| d || d
| d t S qW |j
S ) NZ(update_fingerprint_transform_hash_failedFz
Transform a couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.Tz= couldn't be hashed properly, a random hash was used instead.zn couldn't be hashed properly, a random hash was used instead. This doesn't affect caching since it's disabled.z Parameter 'z'=z of the transform ) rJ rW r@ fingerprint_warningsgetloggerwarninginfor} rl rX ) fingerprint transformZtransform_argsrg ru r# r# r$ update_fingerprint) s<
r )r c C s~ t | t s| r"td| dx*tD ]"}|| kr(tdt d| dq(W t| |krztd| d| dt| d d
S ) z
Make sure the fingerprint is a non-empty string that is not longer that max_length=64 by default,
so that the fingerprint can be used to name cache files without issues.
zInvalid fingerprint 'z#': it should be a non-empty string.z5Invalid fingerprint. Bad characters from black list 'z ' found in 'z6'. They could create issues when creating cache files.z&Invalid fingerprint. Maximum lenth is z but 'z
' has length z2.It could create issues when creating cache files.N)rU rb
ValueErrorr len)r
max_lengthZ invalid_charr# r# r$ validate_fingerprint[ s
r F)inplace
use_kwargs
ignore_kwargsfingerprint_namesrandomized_functionversionc s dk r&t t r&tdt dk rLt t rLtdt r\ r\td dk rh ndg fdd}|S )a
Wrapper for dataset transforms to update the dataset fingerprint using ``update_fingerprint``
Args:
inplace (:obj:`bool`): If inplace is True, the fingerprint of the dataset is updated inplace.
Otherwise, a parameter "new_fingerprint" is passed to the wrapped method that should take care of
setting the fingerprint of the returned Dataset.
use_kwargs (:obj:`List[str]`, optional): optional white list of argument names to take into account
to update the fingerprint to the wrapped method that should take care of
setting the fingerprint of the returned Dataset. By default all the arguments are used.
ignore_kwargs (:obj:`List[str]`, optional): optional black list of argument names to take into account
to update the fingerprint. Note that ignore_kwargs prevails on use_kwargs.
fingerprint_names (:obj:`List[str]`, optional, defaults to ["new_fingerprint"]):
If the dataset transforms is not inplace and returns a DatasetDict, then it can require
several fingerprints (one per dataset in the DatasetDict). By specifying fingerprint_names,
one fingerprint named after each element of fingerprint_names is going to be passed.
randomized_function (:obj:`bool`, defaults to False): If the dataset transform is random and has
optional parameters "seed" and "generator", then you can set randomized_function to True.
This way, even if users set "seed" and "generator" to None, then the fingerprint is
going to be randomly generated depending on numpy's current state. In this case, the
generator is set to np.random.default_rng(np.random.get_state()[1][0]).
version (:obj:`str`, optional): version of the transform. The version is taken into account when
computing the fingerprint. If a datase transform changes (or at least if the output data
that are cached changes), then one should increase the version. If the version stays the
same, then old cached data could be reused that are not compatible with the new transform.
It should be in the format "MAJOR.MINOR.PATCH".
Nz)use_kwargs is supposed to be a list, not z,ignore_kwargs is supposed to be a list, not z5fingerprint_names are only used when inplace is Falsenew_fingerprintc s r&t fddD r&tdrbd jjkrFtd dd jjkrbtd d j d j d k rd
7 t fd d }d
|_|S )Nc 3 s | ]}| j jkV qd S )N)__code__ co_varnames)r4 r )rL r# r$ r6 s z>fingerprint_transform.._fingerprint..zFfunction {func} is missing parameters {fingerprint_names} in signatureseedz'seed' must be in z 's signature generatorz'generator' must be in .@c s |j }| rXdd tjjj D }| d }| dd } |dd }|jt|| n
|jd}r|fdd|j D }rfdd|j D }r|j d d kr|j d
d krt
j j ^}}}}|d k r|| n|d }t
j j
||d
< d d tjjj D }x8|j D ],\} }
| |kr|| |
kr|j| qW rZt|j|} nHxF D ]>} |j | d kr| |d
< t|j||| < n t|| q`W |f| |}
r| |_|
S )Nc S s g | ]}||j kr|jqS r# ) VAR_KEYWORDr )r4 pr# r# r$
s zPfingerprint_transform.._fingerprint..wrapper..r r
r! c s i | ]\}}| kr||qS r# r# )r4 kv)r r# r$
s zPfingerprint_transform.._fingerprint..wrapper..c s i | ]\}}| kr||qS r# r# )r4 r r )r r# r$ r s r r ip c S s" i | ]}|j tjkr|j |jqS r# )defaultinspect_emptyr )r4 r r# r# r$ r s fingerprint_name)copyr signature
parametersvaluesrW zippopitemsr nprz Z get_stateZ default_rngr rq r )argskwargsZkwargs_for_fingerprintparamsr! _r posdefault_valuesZdefault_varname
default_valuer r out)r rL r r r r r r# r$ wrapper sF
z._fingerprint..wrapperr )allr r r r. r/ r Z_decorator_name_)rL r )r r r r r r )rL r r$ rq s <z+fingerprint_transform.._fingerprint)rU r> r r] )r r r r r r rq r# )r r r r r r r$ fingerprint_transformo s $Nr )rw )rw )NNNFN)Cr ro r& rz r) r r functoolsr pathlibr typingr r r r r r Znumpyr Zpyarrowre rQ r r Znamingr tabler
r r r Zutils.deprecation_utilsr Z
utils.loggingr Zutils.py_utilsr r Z
arrow_datasetr r- r r@ r2 r7 r8 r r= r? rA rB rD rE rG rb rI rP rJ rm rp r~ rv r} r r r r# r# r# r$ sb *
2