FM / __pycache_

/usr/local/lib/python3.6/site-packages/datasets/commands/__pycache__/

/usr/local/lib/python3.6/site-packages/datasets/commands/__pycache__

Name	Size	Mode	Actions
convert.cpython-36.pyc	6175	0644	edit dl rm
datasets_cli.cpython-36.pyc	1549	0644	edit dl rm
dummy_data.cpython-36.pyc	15745	0644	edit dl rm
env.cpython-36.pyc	1688	0644	edit dl rm
run_beam.cpython-36.pyc	4589	0644	edit dl rm
test.cpython-36.pyc	5122	0644	edit dl rm
__init__.cpython-36.pyc	778	0644	edit dl rm

Edit: /usr/local/lib/python3.6/site-packages/datasets/commands/__pycache__/test.cpython-36.pyc (5122B)

3 <%Eg�@s�ddlZddlmZddlmZddlmZmZddlm Z ddl Zddlm Z ddlmZddlmZdd lmZmZdd lmZddlmZmZee�Zdd �ZGdd�de�ZdS)�N)�ArgumentParser)�Path)�copyfile�rmtree)� Generator)�DatasetBuilder)�BaseDatasetsCLICommand)�DownloadMode)�dataset_module_factory�import_main_class)�logger)�ERROR� get_loggerc Cs*t|j|j|j|j|j|j|j|j|j � S)N) �TestCommand�dataset�name� cache_dir�data_dir�all_configs� save_infos�ignore_verifications�force_redownload�clear_cache)�args�r�@/usr/local/lib/python3.6/site-packages/datasets/commands/test.py�_test_command_factorysrc @sHeZdZdZeed�dd��Zeeeeeeeeed� dd�Z dd �Z d S)rF)�parsercCs�|jddd�}|jdtddd�|jdtddd�|jd tdd d�|jddd d�|jdddd�|jdddd�|jdddd�|jdddd�|jdtdd�|jtd�dS)N�testzTest dataset implementation.)�helpz--namezDataset processing name)�type�defaultrz--cache_dirz.Cache directory where the datasets are stored.z --data_dirz@Can be used to specify a manual directory to get the files from.z --all_configs� store_truezTest all dataset configurations)�actionrz--save_infoszSave the dataset infos filez--ignore_verificationsz0Run the test without checksums and splits checksz--force_redownloadzForce dataset redownloadz --clear_cachezBRemove downloaded files and cached datasets after each config testrzName of the dataset to download)r r)�func)� add_parser�add_argument�str�set_defaultsr)rZtest_parserrrr�register_subcommand$s.zTestCommand.register_subcommand) rrrrrrrrrc Cs^||_||_||_||_||_||_||_||_| |_| rP|rPt d�t d�|rZd|_dS)NaWhen --clear_cache is used, specifying a cache directory is mandatory. The 'download' folder of the cache directory and the dataset builder cache will be deleted after each configuration test. Please provide a --cache_dir that will be used to test the dataset script.�T)�_dataset�_name� _cache_dir� _data_dir�_all_configs�_save_infos�_ignore_verifications�_force_redownload�_clear_cache�print�exit) �selfrrrrrrrrrrrr�__init__Bs zTestCommand.__init__c s@t�jt��jdk r,�jr,td�td��j�j}�t|��t �j ��jrb�jrbt�j�nd}t tddfd��fdd�}�x�t|��D�]�\}}td|jj�d|d�d|�d ��d |_|j�js�tjntj�jdd�|j��j�r|j��j�r�tjj�j�tjj �}t!|�jd }tjj||�}tjj"|��rXtjj#|�} n:tjj"|��rl|} n&tjj$|��r�|} nd} td|��| dk �r�tjj| tjj �} t%|| �td| ��j&r�tjj$|j'��r�t(j)d|j'��t*|j'�tjj�j'tjj+�}tjj$|�r�t(j)d|��t*|�q�Wtd�dS)NzABoth parameters `config` and `all_configs` can't be used at once.r*)�returnc3s��jrn�jrnx�t�j�D]P\}}d�jkrH�f�j�jd��j��Vq�f|j�j�jd��j��VqWnFd�jkr��f�j�jd��j��Vn�f��j�jd��j��VdS)N�config_name)rr)r9rr)r/�BUILDER_CONFIGS� enumerateZbuilder_kwargsr-r.r)�i�config)�builder_clsr9�moduler6rr�get_buildersks& z%TestCommand.run..get_builderszTesting builder 'z' (�/�)TF)Z download_moderZtry_from_hf_gcsz.pyzDataset Infos file saved at zClearing cache at zTest successful.),� fl_logger�setLevelr r,r/r4r5r+r r�module_pathr:�lenrrr;r=rZ _record_infosZdownload_and_preparer2r ZREUSE_CACHE_IF_EXISTSZFORCE_REDOWNLOADr1Z as_datasetr0�os�path�joinZget_imported_module_dir�datasetsZDATASETDICT_INFOS_FILENAMEr�isfile�dirname�isdirrr3r-r�warningrZDOWNLOADED_DATASETS_DIR)r6rHZ n_buildersr@�jZbuilderZdataset_infos_pathrZ combined_pathZdataset_dirZuser_dataset_infos_path�download_dirr)r>r9r?r6r�runas\ "$ zTestCommand.runN)�__name__� __module__�__qualname__Z__test__�staticmethodrr)r'�boolr7rQrrrrr!sr)rG�argparser�pathlibr�shutilrr�typingrZdatasets.configrJZdatasets.builderrZdatasets.commandsrZ"datasets.download.download_managerr Z datasets.loadr rZdatasets.utils.filelockrrCZdatasets.utils.loggingr rrRrrrrrr�s