/usr/local/lib/python3.6/site-packages/datasets/formatting/__pycache__/
/usr/local/lib/python3.6/site-packages/datasets/formatting/__pycache__
NameSizeModeActions
formatting.cpython-36.pyc233670644editdlrm
jax_formatter.cpython-36.pyc24800644editdlrm
tf_formatter.cpython-36.pyc28200644editdlrm
torch_formatter.cpython-36.pyc24080644editdlrm
__init__.cpython-36.pyc36780644editdlrm
Edit: /usr/local/lib/python3.6/site-packages/datasets/formatting/__pycache__/formatting.cpython-36.pyc (23367B)
3 <%EgvZ@s,ddlmZddlmZmZmZmZmZmZm Z m Z m Z m Z ddl ZddlZddlZddlmZmZmZmZddlmZddlmZe dZe d Ze d Ze d Ze e!d d dZ"edddZ#ee e$e%e e&efeejdddZ'ee e$e%e e&efejdddZ(ej)e!dddZ*GdddeeeefZ+ee&eefee&efdddZ,Gd d!d!e+ejej)ejfZ-Gd"d#d#e+e.e/e.fZ0Gd$d%d%e+e.ej1e.fZ2Gd&d'd'e+ej3ej4ej3fZ5Gd(d)d)Z6Gd*d+d+Z7Gd,d-d-eeeefZ8Gd.d/d/e8ejej)ejfZ9Gd0d1d1e8e.e/e.fZ:Gd2d3d3e8e.ej1e.fZ;Gd4d5d5e8Ze e$e%e efe$dd;dd?Z@dFee e$e%e e&efe eejdd@dAZAdGee e$e%e e&efe8e e/dCdDdEZBdS)H)partial) AnyCallableDictGenericIterableListMutableMappingOptionalTypeVarUnionN)_ArrayXDExtensionType_is_zero_copy_onlydecode_nested_examplepandas_types_mapper)Table)no_op_if_value_is_nullT RowFormat ColumnFormat BatchFormat)keyreturncCs|jdko|j|jkS)N)stepstopstart)rrH/usr/local/lib/python3.6/site-packages/datasets/formatting/formatting.py_is_range_contiguous$sr )rcCstd|dt|ddS)NzWrong key type: 'z ' of type 'z6'. Expected one of int, slice, range, str or Iterable.) TypeErrortype)rrrr_raise_bad_key_type(sr#)tablerindicesrcsttr4jjdjddjt|SttrNtj jttrt rj dkrt|ddjj j j jdDStt r|jfdd|jD}t|jdjSttrt|fddDStdS)aE Query a pyarrow Table to extract the subtable that correspond to the given key. The :obj:`indices` parameter corresponds to the indices mapping in case we cant to take into account a shuffling or an indices selection for example. The indices table must contain one column named "indices" of type uint64. rrcSsg|] }|jqSr)as_py).0irrr ?sz5_query_table_with_indices_mapping..csg|]}|kr|qSrr)r'column)rrrr)Dscs&g|]}j|djddjqS)rr) fast_slicer*r&)r'r()r%rrr)GsN) isinstanceintr+num_rowsr*r& _query_tablesliceranger%r rrstrdrop column_names to_pylistrr#)r$rr%r)r%rr!_query_table_with_indices_mapping.s     *  r6)r$rrcsttr|j|jdSttr6tj|jttrhtrhjdkrh|jjj jStt r|j j fdd|j DSttrtjtjtdkr|j jddS|j|jStdS)zY Query a pyarrow Table to extract the subtable that correspond to the given key. rrcsg|]}|kr|qSrr)r'r*)rrrr)Zsz _query_table..N)r,r-r+r.r0r1r%r rrr2r$r3r4rnpZfromiterZint64lenZ fast_gatherr#)r$rr)rrr/Ls      r/)pa_arrayrcCs |jdkS)Nr)Z null_count)r9rrr_is_array_with_nullsesr:c@sFeZdZdZejedddZejedddZ eje dddZ d S) BaseArrowExtractorz Arrow extractor are used to extract data from pyarrow tables. It makes it possible to extract rows, columns and batches. These three extractions types have to be implemented. )pa_tablercCstdS)N)NotImplementedError)selfr<rrr extract_rowpszBaseArrowExtractor.extract_rowcCstdS)N)r=)r>r<rrrextract_columnssz!BaseArrowExtractor.extract_columncCstdS)N)r=)r>r<rrr extract_batchvsz BaseArrowExtractor.extract_batchN) __name__ __module__ __qualname____doc__parrr?rr@rrArrrrr;isr;)py_dictrcCsdd|jDS)z:Return the first element of a batch (dict) as a row (dict)cSsi|]\}}|d|qS)rr)r'rarrayrrr |sz_unnest..)items)rGrrr_unnestzsrKc@sHeZdZejejdddZejejdddZejejdddZdS) SimpleArrowExtractor)r<rcCs|S)Nr)r>r<rrrr?sz SimpleArrowExtractor.extract_rowcCs |jdS)Nr)r*)r>r<rrrr@sz#SimpleArrowExtractor.extract_columncCs|S)Nr)r>r<rrrrAsz"SimpleArrowExtractor.extract_batchN) rBrCrDrFrr?Arrayr@rArrrrrLsrLc@sBeZdZejedddZejedddZejedddZ dS) PythonArrowExtractor)r<rcCs t|jS)N)rK to_pydict)r>r<rrrr?sz PythonArrowExtractor.extract_rowcCs|jdjS)Nr)r*r5)r>r<rrrr@sz#PythonArrowExtractor.extract_columncCs|jS)N)rO)r>r<rrrrAsz"PythonArrowExtractor.extract_batchN) rBrCrDrFrdictr?listr@rArrrrrNsrNc@s`eZdZddZejedddZejej dddZ ejeddd Z ej ej d d d Z d S)NumpyArrowExtractorcKs ||_dS)N)np_array_kwargs)r>rSrrr__init__szNumpyArrowExtractor.__init__)r<rcCst|j|S)N)rKrA)r>r<rrrr?szNumpyArrowExtractor.extract_rowcCs|j||jdS)Nr)_arrow_array_to_numpyr4)r>r<rrrr@sz"NumpyArrowExtractor.extract_columncsfddjDS)Ncsi|]}j||qSr)rU)r'col)r<r>rrrIsz5NumpyArrowExtractor.extract_batch..)r4)r>r<r)r<r>rrAsz!NumpyArrowExtractor.extract_batch)r9rcsbt|tjrt|jtrdt|jjdd|jjddkrNfdd|jDqfdd|jDn2t|jot dd|jDfd d|jDnlt|jtrt|jjdd|jjddkr|j d n |j d n$t|jot | |j d j tdkrJtfd dDrJtjfd d i|jdtiStjfd d i|jS)NT)Zunnestrcs"g|]}|jdD]}|qqS))zero_copy_only)to_list_of_numpy)r'chunkrow)rWrrr)sz=NumpyArrowExtractor._arrow_array_to_numpy..cs"g|]}|jdD]}|qqS))rW)to_numpy)r'rYrZ)rWrrr)scss|]}t| VqdS)N)r:)r'rYrrr sz.cs"g|]}|jdD]}|qqS))rW)r[)r'rYrZ)rWrrr)s)rWc3sH|]@}t|tjr,|jtkp>|jdjkp>t|to>tj|VqdS)rN)r,r7ndarraydtypeobjectshapefloatisnan)r'x)rHrrr\scopyFr^)r,rFZ ChunkedArrayr"rrZ storage_dtyper`chunksallrXr[r:tolistr8anyr7rHrSr_)r>r9r)rHrWrrUs.         z)NumpyArrowExtractor._arrow_array_to_numpyN)rBrCrDrTrFrrPr?r7r]r@rArMrUrrrrrRs rRc@sHeZdZejejdddZejejdddZ ejejdddZ dS) PandasArrowExtractor)r<rcCs|jddjtdS)Nr)length) types_mapper)r0 to_pandasr)r>r<rrrr?sz PandasArrowExtractor.extract_rowcCs|jdgjtd|jdS)Nr)rk)selectrlrr4)r>r<rrrr@sz#PandasArrowExtractor.extract_columncCs |jtdS)N)rk)rlr)r>r<rrrrAsz"PandasArrowExtractor.extract_batchN) rBrCrDrFrpd DataFramer?Seriesr@rArrrrrisric@sFeZdZddZeedddZeeedddZeed d d Z d S) PythonFeaturesDecodercCs ||_dS)N)features)r>rrrrrrTszPythonFeaturesDecoder.__init__)rZrcCs|jr|jj|S|S)N)rrZdecode_example)r>rZrrr decode_rowsz PythonFeaturesDecoder.decode_row)r* column_namercCs|jr|jj||S|S)N)rr decode_column)r>r*rtrrrrusz#PythonFeaturesDecoder.decode_column)batchrcCs|jr|jj|S|S)N)rr decode_batch)r>rvrrrrwsz"PythonFeaturesDecoder.decode_batchN) rBrCrDrTrPrsrQr2rurwrrrrrqsrqc@sReZdZddZejejdddZejeejdddZ ejejd d d Z d S) PandasFeaturesDecodercCs ||_dS)N)rr)r>rrrrrrTszPandasFeaturesDecoder.__init__)rZrcs@jrfddjjDni}|r<|j||t|j<|S)Ncs,i|]$\}}jj|rttt||qSr)rr_column_requires_decodingrrr)r'rtfeature)r>rrrIsz4PandasFeaturesDecoder.decode_row..)rrrJ transformrQkeys)r>rZdecoder)r>rrss z PandasFeaturesDecoder.decode_row)r*rtrcCsF|jr0||jkr0|jj|r0ttt|j|nd}|rB|j|}|S)N)rrryrrrr{)r>r*rtr}rrrrus 0 z#PandasFeaturesDecoder.decode_column)rvrcCs |j|S)N)rs)r>rvrrrrwsz"PandasFeaturesDecoder.decode_batchN) rBrCrDrTrnrorsrpr2rurwrrrrrxs rxc@s~eZdZdZeZeZeZ e Z dddZ e jeeeeefdddZe jed d d Ze jed d d Ze jed ddZdS) Formatterz A formatter is an object that extracts and formats data from pyarrow tables. It defines the formatting for rows, columns and batches. NTcCs(||_||_t|j|_t|j|_dS)N)rrdecodedrqpython_features_decoderrxpandas_features_decoder)r>rrrrrrrTs zFormatter.__init__)r< query_typercCs:|dkr|j|S|dkr$|j|S|dkr6|j|SdS)NrZr*rv) format_row format_column format_batch)r>r<rrrr__call__s   zFormatter.__call__)r<rcCstdS)N)r=)r>r<rrrrszFormatter.format_rowcCstdS)N)r=)r>r<rrrr"szFormatter.format_columncCstdS)N)r=)r>r<rrrr%szFormatter.format_batch)NT)rBrCrDrErLsimple_arrow_extractorrNpython_arrow_extractorrRnumpy_arrow_extractorripandas_arrow_extractorrTrFrr2r rrrrrrrrrrrr~s r~c@sHeZdZejejdddZejejdddZejejdddZdS) ArrowFormatter)r<rcCs|jj|S)N)rr?)r>r<rrrr*szArrowFormatter.format_rowcCs|jj|S)N)rr@)r>r<rrrr-szArrowFormatter.format_columncCs|jj|S)N)rrA)r>r<rrrr0szArrowFormatter.format_batchN) rBrCrDrFrrrMrrrrrrr)src@sBeZdZejedddZejedddZejedddZ dS) PythonFormatter)r<rcCs$|jj|}|jr |jj|}|S)N)rr?rrrs)r>r<rZrrrr5s zPythonFormatter.format_rowcCs,|jj|}|jr(|jj||jd}|S)Nr)rr@rrrur4)r>r<r*rrrr;szPythonFormatter.format_columncCs$|jj|}|jr |jj|}|S)N)rrArrrw)r>r<rvrrrrAs zPythonFormatter.format_batchN) rBrCrDrFrrPrrQrrrrrrr4srcsVeZdZd fdd ZejedddZejej ddd Z ejedd d Z Z S) NumpyFormatterNTc stj||d||_dS)N)rrr)superrTrS)r>rrrrS) __class__rrrTIszNumpyFormatter.__init__)r<rcCs*|jf|jj|}|jr&|jj|}|S)N)rrSr?rrrs)r>r<rZrrrrMs zNumpyFormatter.format_rowcCs2|jf|jj|}|jr.|jj||jd}|S)Nr)rrSr@rrrur4)r>r<r*rrrrSszNumpyFormatter.format_columncCs*|jf|jj|}|jr&|jj|}|S)N)rrSrArrrw)r>r<rvrrrrYs zNumpyFormatter.format_batch)NT) rBrCrDrTrFrrPrr7r]rr __classcell__rr)rrrHsrc@sHeZdZejejdddZejejdddZ ejejdddZ dS) PandasFormatter)r<rcCs$|jj|}|jr |jj|}|S)N)rr?rrrs)r>r<rZrrrras zPandasFormatter.format_rowcCs,|jj|}|jr(|jj||jd}|S)Nr)rr@rrrur4)r>r<r*rrrrgszPandasFormatter.format_columncCs$|jj|}|jr |jj|}|S)N)rrArrrw)r>r<rZrrrrms zPandasFormatter.format_batchN) rBrCrDrFrrnrorrprrrrrrr`srcsheZdZdZdeegefdfdd Zejeddd Z eje dd d Z ejedd d Z Z S)CustomFormattera A user-defined custom formatter function defined by a ``transform``. The transform must take as input a batch of data extracted for an arrow table using the python extractor, and return a batch. If the output batch is not a dict, then output_all_columns won't work. If the ouput batch has several fields, then querying a single column won't work since we don't know which field to return. NT)r{c stj||d||_dS)N)rrr)rrTr{)r>r{rrrkwargs)rrrrT~szCustomFormatter.__init__)r<rcCsJ|j|}yt|Stk rD}ztd||WYdd}~XnXdS)NzPCustom formatting function must return a dict to be able to pick a row, but got )rrK Exceptionr!)r>r<formatted_batchexcrrrrs  zCustomFormatter.format_rowcCs|j|}t|dr>t|jdkrLtdt|jdntd|y||jdStk r}ztd||WYdd}~XnXdS)Nr|rzTried to query a column but the custom formatting function returns too many columns. Only one column was expected but got columns .zPCustom formatting function must return a dict to be able to pick a row, but got r)rhasattrr8r|r!rQr4r)r>r<rrrrrrs    zCustomFormatter.format_columncCs*|jj|}|jr |jj|}|j|S)N)rrArrrwr{)r>r<rvrrrrs zCustomFormatter.format_batch)NT)rBrCrDrErrPrTrFrrrrrrrr)rrrts  r)rcolumnsrcCs ||krtd|d|dS)NzColumn z5 not in the dataset. Current columns in the dataset: )KeyError)rrrrr_check_valid_column_keysr)rsizercCst|tr>|dkr||dks&||kr:td|d|dSt|trJnt|trt|dkrtt||dtt||dnHt|t rt|dkrttt||dttt||dnt |dS)Nrz Invalid key: z is out of bounds for size )r) r,r- IndexErrorr0r1r8_check_valid_index_keymaxminrr#)rrrrrrs      rcCs<t|trdSt|trdSt|tttfr0dSt|dS)NrZr*rv)r,r-r2r0r1rr#)rrrrkey_to_query_types  rcCsxt|tttttfst|t|tr4t||jn|dk rB|j n|j }t |||dkrft ||}nt |||d}|S)a1 Query a Table to extract the subtable that correspond to the given key. Args: table (``datasets.table.Table``): The input Table to query from key (``Union[int, slice, range, str, Iterable]``): The key can be of different types: - an integer i: the subtable containing only the i-th row - a slice [i:j:k]: the subtable containing the rows that correspond to this slice - a range(i, j, k): the subtable containing the rows that correspond to this range - a string c: the subtable containing all the rows but only the column c - an iterable l: the subtable that is the concatenation of all the i-th rows for all i in the iterable indices (Optional ``datasets.table.Table``): If not None, it is used to re-map the given key to the table rows. The indices table must contain one column named "indices" of type uint64. This is used in case of shuffling or rows selection. Returns: ``pyarrow.Table``: the result of the query on the input table N)r%) r,r-r0r1r2rr#rr4r.rr/r6)r$rr%rZ pa_subtablerrr query_tables   rF)r$r formatterformat_columnsc st|tr|j}n|}t|}tdd}dkr<|||dS|dkrd|krV|||S|||dSnx|jfdd|jD}|||d} |rt| tr|jfdd|jD} || |d} | j| nt d| | SdS) a Format a Table depending on the key that was used and a Formatter object. Args: table (``datasets.table.Table``): The input Table to format key (``Union[int, slice, range, str, Iterable]``): Depending on the key that was used, the formatter formats the table as either a row, a column or a batch. formatter (``datasets.formatting.formatting.Formatter``): Any subclass of a Formatter such as PythonFormatter, NumpyFormatter, etc. format_columns (:obj:`List[str]`, optional): if not None, it defines the columns that will be formatted using the given formatter. Other columns are discarded (unless ``output_all_columns`` is True) output_all_columns (:obj:`bool`, defaults to False). If True, the formatted output is completed using the columns that are not in the ``format_columns`` list. For these columns, the PythonFormatter is used. Returns: A row, column or batch formatted object defined by the Formatter: - the PythonFormatter returns a dictionary for a row or a batch, and a list for a column. - the NumpyFormatter returns a dictionary for a row or a batch, and a np.array for a column. - the PandasFormatter returns a pd.DataFrame for a row or a batch, and a pd.Series for a column. - the TorchFormatter returns a dictionary for a row or a batch, and a torch.Tensor for a column. - the TFFormatter returns a dictionary for a row or a batch, and a tf.Tensor for a column. N)rr)rr*c3s|]}|kr|VqdS)Nr)r'rV)rrrr\szformat_table..c3s|]}|kr|VqdS)Nr)r'rV)rrrr\ sz\Custom formatting function must return a dict to work with output_all_columns=True, but got ) r,rr$rrr3r4r updater!) r$rrrZoutput_all_columnsr<rZpython_formatterZpa_table_to_formatZformatted_outputZpa_table_with_remaining_columnsZremaining_columns_dictr)rr format_tables,         r)N)NF)C functoolsrtypingrrrrrrr r r r Znumpyr7ZpandasrnZpyarrowrFZfeatures.featuresrrrrr$rZutils.py_utilsrrrrrr1boolr r#r-r0r2r6r/rMr:r;rKrLrPrQrNr]rRrorprirqrxr~rrrrrrrrrrrrrrsL 0  "$  8  # 1 (