Name	Size	Mode	Actions
arrow_dataset.cpython-36.pyc	189850	0644	editdlrm
arrow_reader.cpython-36.pyc	22483	0644	editdlrm
arrow_writer.cpython-36.pyc	22110	0644	editdlrm
builder.cpython-36.pyc	52779	0644	editdlrm
combine.cpython-36.pyc	5508	0644	editdlrm
config.cpython-36.pyc	5181	0644	editdlrm
dataset_dict.cpython-36.pyc	83319	0644	editdlrm
data_files.cpython-36.pyc	30153	0644	editdlrm
fingerprint.cpython-36.pyc	18979	0644	editdlrm
info.cpython-36.pyc	17359	0644	editdlrm
inspect.cpython-36.pyc	18583	0644	editdlrm
iterable_dataset.cpython-36.pyc	62932	0644	editdlrm
keyhash.cpython-36.pyc	3341	0644	editdlrm
load.cpython-36.pyc	62922	0644	editdlrm
metric.cpython-36.pyc	23217	0644	editdlrm
naming.cpython-36.pyc	2571	0644	editdlrm
search.cpython-36.pyc	31339	0644	editdlrm
splits.cpython-36.pyc	22522	0644	editdlrm
streaming.cpython-36.pyc	4119	0644	editdlrm
table.cpython-36.pyc	79612	0644	editdlrm
__init__.cpython-36.pyc	2282	0644	editdlrm

3 <%Egu��@sj
dd
lZ
dd
lZdd
lZddlmZ
ddlmZmZm Z m Z mZmZ
dd
l ZddlmZ
er�ddlmZ
yddlmZ
Wnek r�


YnXydd
lZWnek r�


YnXe
jjd�
d
k Ze
jjd �
d
k Zeje�
ZGd d�de�ZGdd �d e �Z Gdd�de �Z!Gdd�de �Z"Gdd�de �Z#Gdd�d�Z$Gdd�de$�Z%Gdd�de$�Z&Gdd�d�Z'd
S)�N)
�PurePath)� TYPE_CHECKING�Dict�List� NamedTuple�Optional�Union�
)
�logging)
�Dataset)
� Elasticsearch� elasticsearch�faissc
@seZ
dZd
S)�MissingIndexN)�__name__� __module__�__qualname__�rr�9/usr/local/lib/python3.6/site-packages/datasets/search.pyr!s
rc@seZ
dZUeeeed
S)� SearchResultsN)rrrr�float�scores�int�indicesrrrrr%s

rc@s&eZ
dZUeeeeeed
S)�BatchedSearchResultsN)rrrrr�total_scoresr� total_indicesrrrrr*s

rc@seZ
dZUeeed
S)�NearestExamplesResultsN)rrrrrr�dictZexamplesrrrrr/s

rc@s"eZ
dZUeeeeed
S)�BatchedNearestExamplesResultsN)rrrrrrrZtotal_examplesrrrrr4s

rc@sfeZ
dZd
Zdeed�dd�Zdeed�dd�Ze e efd�
d d �Ze e e efdd�dd ��
ZdS)� BaseIndexzBase class for indexing� )�
k�returnc
Cst�
d
S)z� To implement. This method has to return the scores and the indices of the retrieved examples given a certain query. N)
�NotImplementedError)�self�queryr"rrr�search<szBaseIndex.searchcCsFgg}}x0|
D](}|j||�\}}|j
|�

|j
|�

qWt||�S)
aFind the nearest examples indices to the query. Args: queries (`Union[List[str], np.ndarray]`): The queries as a list of strings if `column` is a text index or as a numpy array if `column` is a vector index. k (`int`): The number of examples to retrieve per query. Ouput: total_scores (`List[List[float]`): The retrieval scores of the retrieved examples per query. total_indices (`List[List[int]]`): The indices of the retrieved examples per query. )r'�appendr)r%�queriesr"rrr&rrrrr�search_batchCs




zBaseIndex.search_batch)
�filec
Cst�
d
S)zSerialize the index on diskN)
r$)r%r+rrr�saveUszBaseIndex.save)r+r#c
Cst�
d
S)zDeserialize the index from diskN)
r$)�clsr+rrr�loadYszBaseIndex.loadN)
r!)
r!)rrr�__doc__rrr'rr*r�strrr,�classmethodr.rrrrr 9s

r c@s�eZ
dZd
Zdeeeeedeeeed�dd�Zde e edfeed�d d �Zdeed�d d�Z deed�dd�ZdS)�ElasticSearchIndexa/
Sparse index using Elasticsearch. It is used to index text and run queries based on BM25 similarity. An Elasticsearch server needs to be accessible, and a python client is declared with ``` es_client = Elasticsearch([{'host': 'localhost', 'port': '9200'}]) ``` for example. Nr)�host�port� es_client� es_index_name�es_index_configcCs�tst
d
�
�
|dk r,|
dk s$|dk r,td�
�
|
p2d}
|p:d}ddl}ddlm}
|dk r\|n||
t|�
d�g
�
|_|dk r~|ndtj j tj�j �
|_|dk r�|n,d d ddd d�i
i
d�dddddd�i
i
d�|_dS)Nz}You must install ElasticSearch to use ElasticSearchIndex. To do so you can run `pip install elasticsearch==7.7.1 for example`zBPlease specify either `es_client` or `(host, port)`, but not both.� localhosti�#r
)
r)r3r4Zhuggingface_datasets_r �analyzerZ stop_standard�standardZ _english_)�typez stopwords)Znumber_of_shardsZanalysisZ properties�textZBM25)r;r9Z similarity)�settingsZmappings)�_has_elasticsearch�ImportError� ValueErrorZelasticsearch.helpersr rr0r5�os�path�basename�tempfile�NamedTemporaryFile�namer6r7)r%r3r4r5r6r7r rrrr�__init__is 





"

zElasticSearchIndex.__init__r)� documents�columncs�|j}|j
}|jjj||d
�
t�
�
}tjd|tj�d�}d}��
fdd�}ddl } x4| j j|j||�d�D]\} }|jd �

|| 7}qnW|t�
�
kr�t jd t�
�
|�dt�
�
���

t jd|d �d��

dS)z� Add documents to the index. If the documents are inside a certain column, you can specify it using the `column` argument. )�index�bodyZdocs)�unit�total�disabler
c3sV�dk r0xHt�
�
D]\}}
|
�|d
�V
qWn"x t�
�
D]\}}
|
|d
�V
q:WdS)N)r<�_id)
� enumerate)�
iZexample)rIrHrr�passage_generator�s



z;ElasticSearchIndex.add_documents..passage_generatorN)�clientrJ�actionsr z>Some documents failed to be added to ElasticSearch. Failures: �
/zIndexed �
dz documents)r6r7r5r�create�lenr �tqdm�is_progress_bar_enabledr ZhelpersZstreaming_bulk�update�logger�warning�info)r%rHrI� index_nameZindex_configZnumber_of_docs�progressZ successesrR�es�ok�actionr)rIrHr� add_documents�s$




 






z ElasticSearchIndex.add_documentsr!)r&r#cCsP|jj
|jd
|
dg
dd�i
|d�d�}|dd}tdd �|D�
d d �|D�
�S)am
Find the nearest examples indices to the query. Args: query (`str`): The query as a string. k (`int`): The number of examples to retrieve. Ouput: scores (`List[List[float]`): The retrieval scores of the retrieved examples. indices (`List[List[int]]`): The indices of the retrieved examples. Zmulti_matchr<Zcross_fields)r&�fieldsr;)r&�size)rJrK�hitsc
Ssg|]}
|
d�qS)
Z_scorer)�.0�hitrrr� �sz-ElasticSearchIndex.search..c
Ssg|]}
t|
d�
�qS)
rO)
r)rhrirrrrj�s)r5r'r6r)r%r&r"�responsergrrrr'�s 


zElasticSearchIndex.search)r"r#cs�d
dl}dg
t
|
�
dg
t
|
�
}}|jj|d�
�Z���
�fdd�t|
�
D�
}x8|jj|�
D](}||} |j�} | j|| <| j|| <q^WWdQRXt ||d�S)Nr
)
�max_workersc
s"i|]\}
}|
�j�j
|�
��qSr)�submitr')rhrQr&)�executorr"r%rr� �sz3ElasticSearchIndex.search_batch..)rr) �concurrent.futuresrX�futures�ThreadPoolExecutorrP�as_completed�resultrrr)r%r)r"rl� concurrentrrZfuture_to_index�futurerJ�resultsr)rnr"r%rr*�s








zElasticSearchIndex.search_batch)NNNNN)
N)
r!)r!r!)rrrr/rr0rrrGrrrdrr'rr*rrrrr2_s



, "$r2c@s
eZ
dZd
Zdeeeeefeeeeedd�dd�Z dee jdfeeeeeeed �d d�Z eddeeeeefdd�d d��
Zde jed�dd�Zd e jed�dd�Zeeefd�
dd�Zed!eeefeeeeefdd�dd��
ZdS)"� FaissIndexa Dense index using Faiss. It is used to index vectors. Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. You can find more information about Faiss here: - For index types and the string factory: https://github.com/facebookresearch/faiss/wiki/The-index-factory - For GPU settings: https://github.com/facebookresearch/faiss/wiki/Faiss-on-the-GPU Nzfaiss.Index)�device�string_factory�metric_type�custom_indexcCsX|d
k r|d
k rtd�
�
|
d
k r0|d
k r0td�
�
|
|_
||_||_||_tsTtd�
�
d
S)a$
Create a Dense index using Faiss. You can specify `device` if you want to run it on GPU (`device` must be the GPU index). You can find more information about Faiss here: - For `string factory`: https://github.com/facebookresearch/faiss/wiki/The-index-factory NzFPlease specify either `string_factory` or `custom_index` but not both.zsCannot pass both 'custom_index' and 'device'. Pass 'custom_index' already transferred to the target device instead.a{
You must install Faiss to use FaissIndex. To do so you can run `conda install -c pytorch faiss-cpu` or `conda install -c pytorch faiss-gpu`. A community supported package is also available on pypi: `pip install faiss-cpu` or `pip install faiss-gpu`. Note that pip may not have the latest version of FAISS, and thus, some of the latest features and bug fixes may not be available.)r@ryrzr{�faiss_index� _has_faissr?)r%ryrzr{r|rrrrG�s









zFaissIndex.__init__��r)�vectorsrI� batch_size� train_size� faiss_verbosecCsd
dl}|j
dkr�|dkr&t|
d
�
nt|
d
|�
}|jdk rn|jdkrZ|j||j�}q�|j||j|j�}n$|jdkr�|j|�
}n|j||j�}|j||j�|_
t j dt|j
�
���

|dk �
r8||j
_t |j
d�r�|j
jdk r�||j
j_t |j
d��
r|j
jdk �
r||j
j_t |j
d��
r8|j
jdk �
r8||j
j_|dk �
r�|dk�
rX|
d|�n|
d|�|} t j dt| �
�d��

|j
j| �

n t j d �

t j d t|
�
�d��

xdtjtd
t|
�
|�tj�d�D]@} |dk�
r�|
| | |�n|
| | |�|}|j
j|�

�
q�WdS) z� Add vectors to the index. If the arrays are inside a certain column, you can specify it using the `column` argument. r
NzCreated faiss index of type rJ� quantizer�clustering_indexz"Training the index with the first z vectorszEIgnored the training step of the faiss index as `train_size` is None.zAdding z vectors to the faiss index)
rN)rr}rXrzr{Z index_factoryZ IndexFlat�_faiss_index_to_deviceryr\r^r;�verbose�hasattrrJr�r�Ztrainr rY�rangerZ�add)r%r�rIr�r�r�rrfrJZ train_vecsrQZvecsrrr�add_vectors�s:
$












&

 
&
.
zFaissIndex.add_vectors)rJryr#cCs�|
d
kr|Sdd
l}t
|
t�rJ|
dkr>|j�}|j||
|�}q�|j|�
}n:t
|
ttf�rl|j|t|
�
d�}nt dt |
�
�d�d�
�
|S) z� Sends a faiss index to a device. A device can either be a positive integer (GPU id), a negative integer (all GPUs), or a list of positive integers (select GPUs to use), or `None` for CPU. Nr
r )
ZgpuszThe argument type: z is not expected. zZPlease pass in either nothing, a positive int, a negative int, or a list of positive ints.�����)r� isinstancerZStandardGpuResourcesZindex_cpu_to_gpuZindex_cpu_to_all_gpus�list�tupleZindex_cpu_to_gpus_list� TypeErrorr;)rJryrZ faiss_resrrrr�2
s 
 




z!FaissIndex._faiss_index_to_devicer!)r&r#cCs~t|
j
�
d
kr2t|
j
�
dks*|
j
dd
kr2td�
�
|
jd
d�}|jjsTtj|dd�}|jj ||�\}}t |d|djt�
�S)aw
Find the nearest examples indices to the query. Args: query (`np.array`): The query as a numpy array. k (`int`): The number of examples to retrieve. Ouput: scores (`List[List[float]`): The retrieval scores of the retrieved examples. indices (`List[List[int]]`): The indices of the retrieved examples. r �r
zHShape of query is incorrect, it has to be either a 1D array or 2D (1, N)�
C)
�orderr�) rX�shaper@Zreshape�flags�c_contiguous�np�asarrayr}r'r�astyper)r%r&r"r)rrrrrr'T
s*




zFaissIndex.search)r)r#cCsNt|
j
�
d
krtd�
�
|
jjs,tj|
dd�}
|jj|
|�\}}t ||j t�
�S)a�
Find the nearest examples indices to the queries. Args: queries (`np.array`): The queries as a numpy array. k (`int`): The number of examples to retrieve. Ouput: total_scores (`List[List[float]`): The retrieval scores of the retrieved examples per query. total_indices (`List[List[int]]`): The indices of the retrieved examples per query. r�zShape of query must be 2Dr�)
r�)rXr�r@r�r�r�r�r}r'rr�r)r%r)r"rrrrrr*h
s




zFaissIndex.search_batch)
r+cCsLd
dl}|j
dk r2t|j
tttf�r2|j|j�
}n|j}|j|t |
�
�
dS)z Serialize the FaissIndex on diskr
N) rryr�rr�r�Zindex_gpu_to_cpur}Zwrite_indexr0)r%r+rrJrrrr,z
s 
zFaissIndex.save)r+ryr#cCs4d
dl}||d�
}|j
t|
�
�
}|j||j�|_|S)z$Deserialize the FaissIndex from diskr
N)
ry)rZ read_indexr0r�ryr})r-r+ryrr}rJrrrr.�
s 


zFaissIndex.load)NNNN)NrNN)
N)
r!)
r!)
N)rrrr/rrrrr0rGr��array�boolr��staticmethodr�rr'rr*rr,r1r.rrrrrx�s"


2


.1
&!
rxc@s&eZ
dZd
Zdd�Zdd�Zdd�Zeed�d d �Z ed�
dd �Z eed�
dd�Zee d�dd�Zd7eeeeeeeefeeeeedeeeed� dd�Zd8ejeeeeeefeeeeedeeeed� dd�Zeeeefd�dd�Zd9eeeefeeeeefd �d!d"�Zd:eeeeeeeed#eeeed$�d%d&�Zd;eeeeeeed#eed'�d(d)�Zed�
d*d+�Zdeeeejfee!d-�d3d4�Z"d?eeeeejfee#d0�d5d6�Z$dS)@�IndexableMixinz+Add indexing features to `datasets.Dataset`c

Cs i|_dS)
N)
�_indexes)
r%rrrrG�
s
zIndexableMixin.__init__c


Cst�
dS)
N)
r$)
r%rrr�__len__�
s
zIndexableMixin.__len__c
Cst�
dS)
N)
r$)r%�keyrrr�__getitem__�
s
zIndexableMixin.__getitem__)r_r#cCs |
|jkS)
N)
r�)r%r_rrr�is_index_initialized�
s
z#IndexableMixin.is_index_initialized)
r_cCs|j|
�
st
d
|
�d��
�
dS)NzIndex with index_name 'zk' not initialized yet. Please make sure that you call `add_faiss_index` or `add_elasticsearch_index` first.)r�r)r%r_rrr�_check_index_is_initialized�
s


z*IndexableMixin._check_index_is_initialized)
r#c

Cs t|j
�
S)
zCList the colindex_nameumns/identifiers of all the attached indexes.)r�r�)
r%rrr�list_indexes�
szIndexableMixin.list_indexescCs|j|
�

|j
|
S)
z�List the index_name/identifiers of all the attached indexes. Args: index_name (:obj:`str`): Index name. Returns: :class:`BaseIndex` )r�r�)r%r_rrr� get_index�
s
zIndexableMixin.get_indexN��Fzfaiss.Index) rIr_ryrzr{r|r�r�r�c CsB|d
k r|n|
}t||||d�} | j
||
||| d�
| |j|<d
S)a�Add a dense index using Faiss for fast retrieval. The index is created using the vectors of the specified column. You can specify `device` if you want to run it on GPU (`device` must be the GPU index, see more below). You can find more information about Faiss here: - For `string factory`: https://github.com/facebookresearch/faiss/wiki/The-index-factory Args: column (:obj:`str`): The column of the vectors to add to the index. index_name (Optional :obj:`str`): The index_name/identifier of the index. This is the index_name that is used to call `.get_nearest` or `.search`. By defaul it corresponds to `column`. device (Optional :obj:`Union[int, List[int]]`): If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs. If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU. string_factory (Optional :obj:`str`): This is passed to the index factory of Faiss to create the index. Default index class is IndexFlatIP. metric_type (Optional :obj:`int`): Type of metric. Ex: `faiss.METRIC_INNER_PRODUCT` or `faiss.METRIC_L2`. custom_index (Optional :obj:`faiss.Index`): Custom Faiss index that you already have instantiated and configured for your needs. batch_size (Optional :obj:`int`): Size of the batch to use while adding vectors to the FaissIndex. Default value is 1000. train_size (Optional :obj:`int`): If the index needs a training step, specifies how many vectors will be used to train the index. faiss_verbose (:obj:`bool`, defaults to False): Enable the verbosity of the Faiss index. N)ryrzr{r|)rIr�r�r�)rxr�r�)r%rIr_ryrzr{r|r�r�r�r}rrr�add_faiss_index�
s


zIndexableMixin.add_faiss_index) �external_arraysr_ryrzr{r|r�r�r�c Cs2t||||d
�} | j
|
d||| d�
| |j|<dS)a<Add a dense index using Faiss for fast retrieval. The index is created using the vectors of `external_arrays`. You can specify `device` if you want to run it on GPU (`device` must be the GPU index). You can find more information about Faiss here: - For `string factory`: https://github.com/facebookresearch/faiss/wiki/The-index-factory Args: external_arrays (:obj:`np.array`): If you want to use arrays from outside the lib for the index, you can set `external_arrays`. It will use `external_arrays` to create the Faiss index instead of the arrays in the given `column`. index_name (:obj:`str`): The index_name/identifier of the index. This is the index_name that is used to call `.get_nearest` or `.search`. device (Optional :obj:`Union[int, List[int]]`): If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs. If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU. string_factory (Optional :obj:`str`): This is passed to the index factory of Faiss to create the index. Default index class is IndexFlatIP. metric_type (Optional :obj:`int`): Type of metric. Ex: `faiss.METRIC_INNER_PRODUCT` or `faiss.METRIC_L2`. custom_index (Optional :obj:`faiss.Index`): Custom Faiss index that you already have instantiated and configured for your needs. batch_size (Optional :obj:`int`): Size of the batch to use while adding vectors to the FaissIndex. Default value is 1000. train_size (Optional :obj:`int`): If the index needs a training step, specifies how many vectors will be used to train the index. faiss_verbose (:obj:`bool`, defaults to False): Enable the verbosity of the Faiss index. )ryrzr{r|N)rIr�r�r�)rxr�r�)r%r�r_ryrzr{r|r�r�r�r}rrr�$add_faiss_index_from_external_arrays�
s 

z3IndexableMixin.add_faiss_index_from_external_arrays)r_r+cCsR|j|
�
}t
|t�s.td
|
�dt|�
�d��
�
|j|�

tjd|
�d|���

dS)a
Save a FaissIndex on disk. Args: index_name (:obj:`str`): The index_name/identifier of the index. This is the index_name that is used to call `.get_nearest` or `.search`. file (:obj:`str`): The path to the serialized faiss index on disk. zIndex 'z' is not a FaissIndex but a '�
'zSaved FaissIndex z at N)r�r�rxr@r;r,r\r^)r%r_r+rJrrr�save_faiss_index s 



zIndexableMixin.save_faiss_index)r_r+ryc Csltj
||d
�}|jjt|�
krHtd|
�d|�d|jj�dt|�
�d� �
�
||j|
<tjd|
�d|���

d S) a�Load a FaissIndex from disk. If you want to do additional configurations, you can have access to the faiss index object by doing `.get_index(index_name).faiss_index` to make it fit your needs. Args: index_name (:obj:`str`): The index_name/identifier of the index. This is the index_name that is used to call `.get_nearest` or `.search`. file (:obj:`str`): The path to the serialized faiss index on disk. device (Optional :obj:`Union[int, List[int]]`): If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs. If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU. )
ryz1Index size should match Dataset size, but Index 'z' at z has z elements while the dataset has z examples.zLoaded FaissIndex z from N) rxr.r}ZntotalrXr@r�r\r^)r%r_r+ryrJrrr�load_faiss_indexs


(
zIndexableMixin.load_faiss_indexr)rIr_r3r4r5r6r7c Cs>|d
k r|n|
}t|||||d�}|j
||
d�
||j|<d
S)a3Add a text index using ElasticSearch for fast retrieval. Args: column (:obj:`str`): The column of the documents to add to the index. index_name (Optional :obj:`str`): The index_name/identifier of the index. This is the index name that is used to call `.get_nearest` or `.search`. By defaul it corresponds to `column`. host (Optional :obj:`str`, defaults to localhost): host of where ElasticSearch is running port (Optional :obj:`str`, defaults to 9200): port of where ElasticSearch is running es_client (Optional :obj:`elasticsearch.Elasticsearch`): The elasticsearch client used to create the index if host and port are None. es_index_name (Optional :obj:`str`): The elasticsearch index name used to create the index. es_index_config (Optional :obj:`dict`): The configuration of the elasticsearch index. Default config is: Config:: { "settings": { "number_of_shards": 1, "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}}, }, "mappings": { "properties": { "text": { "type": "text", "analyzer": "standard", "similarity": "BM25" }, } }, } N)r3r4r5r6r7)
rI)r2rdr�) r%rIr_r3r4r5r6r7Zes_indexrrr�add_elasticsearch_index0s -


z&IndexableMixin.add_elasticsearch_index)r_r6r3r4r5r7cCst|||||d
�|j
|
<dS)a�Load an existing text index using ElasticSearch for fast retrieval. Args: index_name (:obj:`str`): The index_name/identifier of the index. This is the index name that is used to call `.get_nearest` or `.search`. es_index_name (:obj:`str`): The name of elasticsearch index to load. host (Optional :obj:`str`, defaults to localhost): host of where ElasticSearch is running port (Optional :obj:`str`, defaults to 9200): port of where ElasticSearch is running es_client (Optional :obj:`elasticsearch.Elasticsearch`): The elasticsearch client used to create the index if host and port are None. es_index_config (Optional :obj:`dict`): The configuration of the elasticsearch index. Default config is:: { "settings": { "number_of_shards": 1, "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}}, }, "mappings": { "properties": { "text": { "type": "text", "analyzer": "standard", "similarity": "BM25" }, } }, } )r3r4r5r6r7N)r2r�)r%r_r6r3r4r5r7rrr�load_elasticsearch_indexds(
z'IndexableMixin.load_elasticsearch_indexcCs|j|
=d
S)z�Drop the index with the specified column. Args: index_name (:obj:`str`): The index_name/identifier of the index. N)
r�)r%r_rrr� drop_index�szIndexableMixin.drop_indexr!)r_r&r"r#cCs|j|
�

|j
|
j||�S)
aBFind the nearest examples indices in the dataset to the query. Args: index_name (:obj:`str`): The name/identifier of the index. query (:obj:`Union[str, np.ndarray]`): The query as a string if `index_name` is a text index or as a numpy array if `index_name` is a vector index. k (:obj:`int`): The number of examples to retrieve. Returns: scores (:obj:`List[List[float]`): The retrieval scores of the retrieved examples. indices (:obj:`List[List[int]]`): The indices of the retrieved examples. )r�r�r')r%r_r&r"rrrr'�s
zIndexableMixin.search)r_r)r"r#cCs|j|
�

|j
|
j||�S)
a�Find the nearest examples indices in the dataset to the query. Args: index_name (:obj:`str`): The index_name/identifier of the index. queries (:obj:`Union[List[str], np.ndarray]`): The queries as a list of strings if `index_name` is a text index or as a numpy array if `index_name` is a vector index. k (:obj:`int`): The number of examples to retrieve per query. Returns: total_scores (:obj:`List[List[float]`): The retrieval scores of the retrieved examples per query. total_indices (:obj:`List[List[int]]`): The indices of the retrieved examples per query. )r�r�r*)r%r_r)r"rrrr*�s
zIndexableMixin.search_batchcCsD|j|
�

|j
|
||�\}}d
d�|D�
}t|dt|�
�||�S)a"Find the nearest examples in the dataset to the query. Args: index_name (:obj:`str`): The index_name/identifier of the index. query (:obj:`Union[str, np.ndarray]`): The query as a string if `index_name` is a text index or as a numpy array if `index_name` is a vector index. k (:obj:`int`): The number of examples to retrieve. Returns: scores (:obj:`List[float]`): The retrieval scores of the retrieved examples. examples (:obj:`dict`): The retrieved examples. c
Ssg|]}
|
dkr|
�qS)
r
r)rhrQrrrrj�sz7IndexableMixin.get_nearest_examples..N)r�r'rrX)r%r_r&r"rrZtop_indicesrrr�get_nearest_examples�s


z#IndexableMixin.get_nearest_examplescsL�j|
�

�j
|
||�\}}d
d�t||�D�
}�f
dd�|D�
}t||�S)a`Find the nearest examples in the dataset to the query. Args: index_name (:obj:`str`): The index_name/identifier of the index. queries (:obj:`Union[List[str], np.ndarray]`): The queries as a list of strings if `index_name` is a text index or as a numpy array if `index_name` is a vector index. k (:obj:`int`): The number of examples to retrieve per query. Returns: total_scores (`List[List[float]`): The retrieval scores of the retrieved examples per query. total_examples (`List[dict]`): The retrieved examples per query. c
Ss*g|]"\}
}|
dtd
d�|D�
�
��qS)Nc
Ssg|]}
|
dkr|
�qS)
r
r)rhrQrrrrj�szHIndexableMixin.get_nearest_examples_batch...)
rX)rhZscores_iZ indices_irrrrj�s
z=IndexableMixin.get_nearest_examples_batch..c
sg|]}
�dd
�|
D�
�qS)c
Ssg|]}
|
dkr|
�qS)
r
r)rhrQrrrrj�szHIndexableMixin.get_nearest_examples_batch...r)rhr)
r%rrrj�s)r�r*�zipr)r%r_r)r"rrZ total_samplesr)
r%r�get_nearest_examples_batch�s


z)IndexableMixin.get_nearest_examples_batch)NNNNNr�NF)NNNNr�NF)
N)NNNNNN)NNNN)
r!)
r!)
r!)
r!)%rrrr/rGr�r�r0r�r�r�rr�r r�rrrr�r�r�r�rr�r�rr�r�r�rr'rr*rr�rr�rrrrr��
sT







D"





B,




40


*% $

r�)(�importlib.util� importlibrArD�pathlibr�typingrrrrrrZnumpyr��utilsr Z arrow_datasetrr rr?r�util� find_specr>r~� get_loggerrr\� Exceptionrrrrrr r2rxr�rrrr�
s:










 &tC