/usr/local/lib/python3.6/site-packages/datasets/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/datasets/__pycache__/table.cpython-36.pyc (79612B)
3
<%EgA @ s d dl Z d dlZd dlZd dlZd dlmZ d dlmZ d dlm Z m
Z
m Z m Z m
Z
mZmZ d dlZd dlZddlmZ ddlmZ e rddlmZmZ eeZ d d
Z e ej d d d
Z ej!ej dddZ"e ej d ddZ#ej e e$dddZ%e&dddZ'e e$ e$e$dddZ(G d d d Z)G d d d e)Z G d d! d!e Z*G d"d# d#e*Z+e
e e,e&f Z-G d$d% d%e*Z.ed&e*e e* e e e* Z/G d'd( d(e Z0dLe e e$e d)d*d+Z1e e e d,d-d.Z2d/d0 Z3e3dMej4ej5d2d3d4Z6e3dNej4d5d6d7d8Z7e3ej4d5d6d9d:Z8ej d;d<d=d>Z9ej ej:d?d@dAZ;ej dBdCdDZ
ej e
ej4gdf dIdJdKZ?dS )O N)partial)groupby)
TYPE_CHECKINGCallableListOptionalTupleTypeVarUnion )config)
get_logger)Features FeatureTypec s fdd}|S )Nc sB j | j d k r| j nd | _ | j jdd| _ t dr> j| _| S )N z
pyarrow.TableTable__annotations__)__doc__replacehasattrr )fn)arrow_table_method 8/usr/local/lib/python3.6/site-packages/datasets/table.pywrapper s
z1inject_arrow_table_documentation..wrapperr )r r r )r r inject_arrow_table_documentation s r )filenamereturnc C s" t j| }t jj|}|j }|S )N)paZ input_streamipc open_streamread_all)r Zin_memory_stream
opened_streampa_tabler r r _in_memory_arrow_table_from_file" s
r$ )bufferr c C s" t j| }t jj|}|j }|S )N)r Z BufferReaderr r r! )r% streamr" tabler r r "_in_memory_arrow_table_from_buffer) s
r( c C s" t j| }t jj|}|j }|S )N)r Z
memory_mapr r r! )r Zmemory_mapped_streamr" r# r r r $_memory_mapped_arrow_table_from_file0 s
r) )r' r r c C s` t |dL}tj|| jd}| j }x|D ]}|j| q*W |j tdd |D S Q R X d S )Nwb)sinkschemac s s | ] }|j V qd S )N)nbytes).0batchr r r > s z'_write_table_to_file..)openr ZRecordBatchStreamWriterr,
to_batchesZ write_batchclosesum)r' r r+ writerbatchesr/ r r r _write_table_to_file7 s
r7 )memoc C sL | j }|j|}||t| < x*| jj D ] \}}t||tj|| q(W |S )z!deepcopy a regular class instance) __class____new__id__dict__itemssetattrcopydeepcopy)xr8 clsresultkvr r r _deepcopyA s
rF )arrrA r c C s dt | d }}x||k r| | | ko6| | k n r||| || | | | | | }| | | ko| |d k n r|S | | |k r|d | }}q|| }}qW td| dt | r| d nd ddS ) ak
Return the position i of a sorted array so that arr[i] <= x < arr[i+1]
Args:
arr (:obj:`List[int]`): non-empty sorted list of integers
x (:obj:`int`): query
Returns:
`int`: the position i so that arr[i] <= x < arr[i+1]
Raises:
`IndexError`: if the array is empty or if the query is outside the array values
r r zInvalid query 'z ' for size none.N)len
IndexError)rG rA ijrD r r r _interpolation_searchK s *($ rO c @ sN e Zd ZejdddZeee e j
f ejdddZ d ejd d
d Z dS )
IndexedTableMixin)r' c C sD |j | _dd |j D | _tjdgdd | jD tjd| _d S )Nc S s g | ]}t |d kr|qS )r )rK )r. Z recordbatchr r r
h s z.IndexedTableMixin.__init__..r c S s g | ] }t |qS r )rK )r. br r r rQ i s )Zdtype)r, _schemar2 _batchesnpZcumsumZint64_offsets)selfr' r r r __init__f s zIndexedTableMixin.__init__)indicesr c sL t |stdtj j|ddd }tjj fddt||D j dS )a'
Create a pa.Table by gathering the records at the records at the specified indices. Should be faster
than pa.concat_tables(table.fast_slice(int(i) % table.num_rows, 1) for i in indices) since NumPy can compute
the binary searches in parallel, highly optimized C
zIndices must be non-emptyright)Zsider c s, g | ]$\}} j | j| j| d qS )r )rT slicerV )r. Z batch_idxrM )rW r r rQ v s z1IndexedTableMixin.fast_gather..)r, )
rK
ValueErrorrU Z searchsortedrV r r from_batchesziprS )rW rY Z
batch_indicesr )rW r fast_gatherk s
z IndexedTableMixin.fast_gatherr N)r c C s
|dk rt dn0|| jd ks0|dk rB|dkrBtjjg | jdS t| j|}|dksh|| | jd kr| j|d }|d j|| j| |d< ndt| j|| d }| j||d }|d jd|| | j| |d < |d j|| j| |d< tjj|| jdS )
aO
Slice the Table using interpolation search.
The behavior is the same as :obj:`pyarrow.Table.slice` but it's significantly faster.
Interpolation search is used to find the start and end indexes of the batches we want to keep.
The batches to keep are then concatenated to form the sliced Table.
r zOffset must be non-negativer N)r, rJ rJ rJ rJ ) rL rV r r r] rS rO rT r[ )rW offsetlengthrM r6 rN r r r
fast_slice| s
" z IndexedTableMixin.fast_slice)r N)
__name__
__module__ __qualname__r r rX r
r intrU Zndarrayr_ rb r r r r rP e s rP c s~ e Zd ZdZejd fdd ZedddZdd Z d
d Z
d d
Z dd Z dd Z
dd Zdd Zdd Zdd Zdd Zd d Zed d Zed d! Zed"d# Zed$d% Zed&d' Zed(d) Zed*d+ Zd,d- Z d.d/ Z d0d1 Z d2d3 Z d4d5 Z d6d7 Z!d8d9 Z"d:d; Z#dd? Z%d@dA Z&dBdC Z'dDdE Z(dFdG Z)dHdI Z*dJdK Z+dLdM Z,e-e. d dNdOdPZ/ Z0S )Qr a
Wraps a pyarrow Table by using composition.
This is the base class for InMemoryTable, MemoryMappedTable and ConcatenationTable.
It implements all the basic attributes/methods of the pyarrow Table class except
the Table transforms: slice, filter, flatten, combine_chunks, cast, add_column,
append_column, remove_column, set_column, rename_columns and drop.
The implementation of these methods differs for the subclasses.
)r' c s t j| || _d S )N)superrX r' )rW r' )r9 r r rX s zTable.__init__)r8 c C s. | j |t| j < t| j|t| j< t| |S )N)r' r; listrT rF )rW r8 r r r __deepcopy__ s zTable.__deepcopy__c C sf | j jtjkrX| j }tjdddd,}|j}tjd| t ||d d|iS Q R X n
d| j iS d S ) Nr* Fz.arrow)deletesuffixzQAttempting to pickle a table bigger than 4GiB. Writing it on the disk instead at )r' r pathr' )
r' r- r Z MAX_TABLE_NBYTES_FOR_PICKLINGtempfileNamedTemporaryFilenameloggerdebugr7 )rW r' Ztmp_filer r r r __getstate__ s zTable.__getstate__c C s\ d|krD|d }t jd| t|}t jd| tj| n|d }tj| | d S )Nrl z(Unpickling a big table from the disk at z!Removing temporary table file at r' )rp rq r$ osremover rX )rW stater r' r r r __setstate__ s zTable.__setstate__c O s | j j||S )a
Perform validation checks. An exception is raised if validation fails.
By default only cheap validation checks are run. Pass `full=True`
for thorough validation checks (potentially O(n)).
Args:
full (:obj:`bool`, defaults to :obj:`False`):
If True, run expensive checks, otherwise cheap checks only.
Raises:
pa.lib.ArrowInvalid: if validation fails
)r' validate)rW argskwargsr r r rw s zTable.validatec O s. t dd |D }dd |D }| jj||S )ab
Check if contents of two tables are equal.
Args:
other (:class:`datasets.table.Table`):
Table to compare against.
check_metadata (:obj:`bool`, defaults to :obj:`False`):
Whether schema metadata equality should be checked as well.
Returns:
:obj:`bool`
c s s" | ]}t |tr|jn|V qd S )N)
isinstancer r' )r. argr r r r0 s z Table.equals..c S s& i | ] \}}t |tr |jn||qS r )rz r r' )r. rD rE r r r
s z Table.equals..)tupler' equals)rW rx ry r r r r~ s
z Table.equalsc O s | j j||S )ar
Convert Table to list of (contiguous) RecordBatch objects.
Args:
max_chunksize (:obj:`int`, defaults to `None`):
Maximum size for RecordBatch chunks. Individual chunks may be
smaller depending on the chunk layout of individual columns.
Returns:
:obj:`List[pyarrow.RecordBatch]`:
)r' r2 )rW rx ry r r r r2 s zTable.to_batchesc O s | j j||S )zg
Convert the Table to a dict or OrderedDict.
Returns:
:obj:`dict`
)r' to_pydict)rW rx ry r r r r s zTable.to_pydictc O s | j j||S )a)
Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
Args:
memory_pool (:obj:`MemoryPool`, defaults to :obj:`None`):
Arrow MemoryPool to use for allocations. Uses the default memory
pool is not passed.
strings_to_categorical (:obj:`bool`, defaults to :obj:`False`):
Encode string (UTF8) and binary types to pandas.Categorical.
categories (:obj:`list`, defaults to :obj:`empty`):
List of fields that should be returned as pandas.Categorical. Only
applies to table-like data structures.
zero_copy_only (:obj:`bool`, defaults to :obj:`False`):
Raise an ArrowException if this function call would require copying
the underlying data.
integer_object_nulls (:obj:`bool`, defaults to :obj:`False`):
Cast integers with nulls to objects
date_as_object (:obj:`bool`, defaults to :obj:`True`):
Cast dates to objects. If False, convert to datetime64[ns] dtype.
timestamp_as_object (:obj:`bool`, defaults to :obj:`False`):
Cast non-nanosecond timestamps (np.datetime64) to objects. This is
useful if you have timestamps that don't fit in the normal date
range of nanosecond timestamps (1678 CE-2262 CE).
If False, all timestamps are converted to datetime64[ns] dtype.
use_threads (:obj:`bool`, defaults to :obj:`True`):
Whether to parallelize the conversion using multiple threads.
deduplicate_objects (:obj:`bool`, defaults to :obj:`False`):
Do not create multiple copies Python objects when created, to save
on memory use. Conversion will be slower.
ignore_metadata (:obj:`bool`, defaults to :obj:`False`):
If True, do not use the 'pandas' metadata to reconstruct the
DataFrame index, if present
safe (:obj:`bool`, defaults to :obj:`True`):
For certain data types, a cast is needed in order to store the
data in a pandas DataFrame or Series (e.g. timestamps are always
stored as nanoseconds in pandas). This option controls whether it
is a safe cast or not.
split_blocks (:obj:`bool`, defaults to :obj:`False`):
If True, generate one internal "block" for each column when
creating a pandas.DataFrame from a RecordBatch or Table. While this
can temporarily reduce memory note that various pandas operations
can trigger "consolidation" which may balloon memory use.
self_destruct (:obj:`bool`, defaults to :obj:`False`):
EXPERIMENTAL: If True, attempt to deallocate the originating Arrow
memory while converting the Arrow object to pandas. If you use the
object after calling to_pandas with this option it will crash your
program.
types_mapper (:obj:`function`, defaults to :obj:`None`):
A function mapping a pyarrow DataType to a pandas ExtensionDtype.
This can be used to override the default pandas type for conversion
of built-in pyarrow types or in absence of pandas_metadata in the
Table schema. The function receives a pyarrow DataType and is
expected to return a pandas ExtensionDtype or ``None`` if the
default conversion should be used for that type. If you have
a dictionary mapping, you can pass ``dict.get`` as function.
Returns:
:obj:`pandas.Series` or :obj:`pandas.DataFrame`: :obj:`pandas.Series` or :obj:`pandas.DataFrame` depending on type of object
)r' to_pandas)rW rx ry r r r r s
r c @ s e Zd ZdZdS )
TableBlockz
TableBlock is the allowed class inside a ConcanetationTable.
Only MemoryMappedTable and InMemoryTable are TableBlock.
This is because we don't want a ConcanetationTable made out of other ConcanetationTables.
N)rc rd re r r r r r r m s r c @ s e Zd ZdZeedddZeejdddZ edd Z
ed
d Z ed d
Z edd Z
d*ddZdd Zdd Zdd Zdd Zd d Zd d Zd d! Zd"d# Zd$d% Zd&d' Zd(d) ZdS )+
InMemoryTableaW
The table is said in-memory when it is loaded into the user's RAM.
Pickling it does copy all the data using memory.
Its implementation is simple and uses the underlying pyarrow Table methods directly.
This is different from the MemoryMapped table, for which pickling doesn't copy all the
data in memory. For a MemoryMapped, unpickling instead reloads the table from the disk.
InMemoryTable must be used when data fit in memory, while MemoryMapped are reserved for
data bigger than memory or when you want the memory footprint of your application to
stay low.
)r c C s t |}| |S )N)r$ )rB r r' r r r from_file s zInMemoryTable.from_file)r% c C s t |}| |S )N)r( )rB r% r' r r r from_buffer s zInMemoryTable.from_bufferc O s | t jj||S )an
Convert pandas.DataFrame to an Arrow Table.
The column types in the resulting Arrow Table are inferred from the
dtypes of the pandas.Series in the DataFrame. In the case of non-object
Series, the NumPy dtype is translated to its Arrow equivalent. In the
case of `object`, we need to guess the datatype by looking at the
Python objects in this Series.
Be aware that Series of the `object` dtype don't carry enough
information to always lead to a meaningful Arrow type. In the case that
we cannot infer a type, e.g. because the DataFrame is of length 0 or
the Series only contains None/nan objects, the type is set to
null. This behavior can be avoided by constructing an explicit schema
and passing it to this function.
Args:
df (:obj:`pandas.DataFrame`):
schema (:obj:`pyarrow.Schema`, optional):
The expected schema of the Arrow Table. This can be used to
indicate the type of columns if we cannot infer it automatically.
If passed, the output will have exactly this schema. Columns
specified in the schema that are not found in the DataFrame columns
or its index will raise an error. Additional columns or index
levels in the DataFrame which are not specified in the schema will
be ignored.
preserve_index (:obj:`bool`, optional):
Whether to store the index as an additional column in the resulting
``Table``. The default of None will store the index as a column,
except for RangeIndex which is stored as metadata only. Use
``preserve_index=True`` to force it to be stored as a column.
nthreads (:obj:`int`, defaults to :obj:`None` (may use up to system CPU count threads))
If greater than 1, convert columns to Arrow in parallel using
indicated number of threads
columns (:obj:`List[str]`, optional):
List of column to be converted. If None, use all columns.
safe (:obj:`bool`, defaults to :obj:`True`):
Check for overflows or other unsafe conversions
Returns:
:class:`datasets.table.Table`:
Examples:
```python
>>> import pandas as pd
>>> import pyarrow as pa
>>> df = pd.DataFrame({
... 'int': [1, 2],
... 'str': ['a', 'b']
... })
>>> pa.Table.from_pandas(df)
```
)r r from_pandas)rB rx ry r r r r s 8zInMemoryTable.from_pandasc O s | t jj||S )a
Construct a Table from Arrow arrays
Args:
arrays (:obj:`List[Union[pyarrow.Array, pyarrow.ChunkedArray]]`):
Equal-length arrays that should form the table.
names (:obj:`List[str]`, optional):
Names for the table columns. If not passed, schema must be passed
schema (:obj:`Schema`, defaults to :obj:`None`):
Schema for the created table. If not passed, names must be passed
metadata (:obj:`Union[dict, Mapping]`, default None):
Optional metadata for the schema (if inferred).
Returns:
:class:`datasets.table.Table`:
)r r from_arrays)rB rx ry r r r r s zInMemoryTable.from_arraysc O s | t jj||S )a
Construct a Table from Arrow arrays or columns
Args:
mapping (:obj:`Union[dict, Mapping]`):
A mapping of strings to Arrays or Python lists.
schema (:obj:`Schema`, defaults to :obj:`None`):
If not passed, will be inferred from the Mapping values
metadata (:obj:`Union[dict, Mapping]`, default None):
Optional metadata for the schema (if inferred).
Returns:
:class:`datasets.table.Table`:
)r r from_pydict)rB rx ry r r r r s zInMemoryTable.from_pydictc O s | t jj||S )a
Construct a Table from a sequence or iterator of Arrow RecordBatches.
Args:
batches (:obj:`Union[Sequence[pyarrow.RecordBatch], Iterator[pyarrow.RecordBatch]]`):
Sequence of RecordBatch to be converted, all schemas must be equal.
schema (:obj:`Schema`, defaults to :obj:`None`):
If not passed, will be inferred from the first RecordBatch.
Returns:
:class:`datasets.table.Table`:
)r r r] )rB rx ry r r r r] s zInMemoryTable.from_batchesr Nc C s t | j||dS )a
Compute zero-copy slice of this Table
Args:
offset (:obj:`int`, defaults to :obj:`0`):
Offset from start of table to slice
length (:obj:`int`, defaults to :obj:`None`):
Length of slice (default is until end of table starting from
offset)
Returns:
:class:`datasets.table.Table`:
)r` ra )r rb )rW r` ra r r r r[ s zInMemoryTable.slicec O s t | jj||S )zY
Select records from a Table. See pyarrow.compute.filter for full usage.
)r r' r )rW rx ry r r r r s zInMemoryTable.filterc O s t t| jf||S )a
Flatten this Table. Each column with a struct type is flattened
into one column per struct field. Other columns are left unchanged.
Args:
memory_pool (:obj:`MemoryPool`, defaults to :obj:`None`):
For memory allocations, if required, otherwise use default pool
Returns:
:class:`datasets.table.Table`:
)r
table_flattenr' )rW rx ry r r r r s zInMemoryTable.flattenc O s t | jj||S )a
Make a new table by combining the chunks this table has.
All the underlying chunks in the ChunkedArray of each column are
concatenated into zero or one chunk.
Args:
memory_pool (:obj:`MemoryPool`, defaults to :obj:`None`):
For memory allocations, if required, otherwise use default pool
Returns:
:class:`datasets.table.Table`:
)r r' r )rW rx ry r r r r % s z InMemoryTable.combine_chunksc O s t t| jf||S )aq
Cast table values to another schema
Args:
target_schema (:obj:`Schema`):
Schema to cast to, the names and order of fields must match
safe (:obj:`bool`, defaults to :obj:`True`):
Check for overflows or other unsafe conversions
Returns:
:class:`datasets.table.Table`:
)r
table_castr' )rW rx ry r r r r 5 s
zInMemoryTable.castc O s t | jj||S )aa
EXPERIMENTAL: Create shallow copy of table by replacing schema
key-value metadata with the indicated new metadata (which may be None,
which deletes any existing metadata
Args:
metadata (:obj:`dict`, defaults to :obj:`None`):
Returns:
:class:`datasets.table.Table`: shallow_copy
)r r' r )rW rx ry r r r r D s z%InMemoryTable.replace_schema_metadatac O s t | jj||S )a]
Add column to Table at position.
A new table is returned with the column added, the original table
object is left unchanged.
Args:
i (:obj:`int`):
Index to place the column at.
field_ (:obj:`Union[str, pyarrow.Field]`):
If a string is passed then the type is deduced from the column
data.
column (:obj:`Union[pyarrow.Array, List[pyarrow.Array]]`):
Column data.
Returns:
:class:`datasets.table.Table`: New table with the passed column added.
)r r' r )rW rx ry r r r r R s zInMemoryTable.add_columnc O s t | jj||S )a
Append column at end of columns.
Args:
field_ (:obj:`Union[str, pyarrow.Field]`):
If a string is passed then the type is deduced from the column
data.
column (:obj:`Union[pyarrow.Array, List[pyarrow.Array]]`):
Column data.
Returns:
:class:`datasets.table.Table`:
New table with the passed column added.
)r r' r )rW rx ry r r r r g s zInMemoryTable.append_columnc O s t | jj||S )a
Create new Table with the indicated column removed.
Args:
i (:obj:`int`):
Index of column to remove.
Returns:
:class:`datasets.table.Table`:
New table without the column.
)r r' r )rW rx ry r r r r x s zInMemoryTable.remove_columnc O s t | jj||S )a
Replace column in Table at position.
Args:
i (:obj:`int`):
Index to place the column at.
field_ (:obj:`Union[str, pyarrow.Field]`):
If a string is passed then the type is deduced from the column
data.
column (:obj:`Union[pyarrow.Array, List[pyarrow.Array]]`):
Column data.
Returns:
:class:`datasets.table.Table`:
New table with the passed column set.
)r r' r )rW rx ry r r r r s zInMemoryTable.set_columnc O s t | jj||S )zJ
Create new table with columns renamed to provided names.
)r r' r )rW rx ry r r r r s z InMemoryTable.rename_columnsc O s t | jj||S )a
Drop one or more columns and return a new table.
Args:
columns (:obj:`List[str]`):
List of field names referencing existing columns.
Raises:
KeyError : if any of the passed columns name are not existing.
Returns:
:class:`datasets.table.Table`:
New table without the columns.
)r r' r )rW rx ry r r r r s zInMemoryTable.drop)r N)rc rd re r classmethodstrr r Bufferr r r r r] r[ r r r r r r r r r r r r r r r r w s*
:
r c s e Zd ZdZd,ejeeee d fdd
Z
e d-edddZ d d
Z
d d Zed.ejeee ejd
ddZe ee dddZd/ddZdd Zdd Zdd Zd d Zd d Zd d! Zd"d# Zd$d% Zd&d' Zd(d) Z d*d+ Z Z S )0MemoryMappedTableaN
The table is said memory mapped when it doesn't use the user's RAM but loads the data
from the disk instead.
Pickling it doesn't copy the data into memory.
Instead, only the path to the memory mapped arrow file is pickled, as well as the list
of transforms to "replay" when reloading the table from the disk.
Its implementation requires to store an history of all the transforms that were applied
to the underlying pyarrow Table, so that they can be "replayed" when reloading the Table
from the disk.
This is different from the InMemoryTable table, for which pickling does copy all the
data in memory.
InMemoryTable must be used when data fit in memory, while MemoryMapped are reserved for
data bigger than memory or when you want the memory footprint of your application to
stay low.
N)r' rl replaysc s( t j| || _|d k r |ng | _d S )N)rg rX rl r )rW r' rl r )r9 r r rX s zMemoryMappedTable.__init__)r c C s t |}| j||}| |||S )N)r) _apply_replays)rB r r r' r r r r s zMemoryMappedTable.from_filec C s | j | jdS )N)rl r )rl r )rW r r r rr s z MemoryMappedTable.__getstate__c C s: |d }|d }t |}| j||}tj| |||d d S )Nrl r )rl r )r) r r rX )rW ru rl r r' r r r rv s
z MemoryMappedTable.__setstate__)r' r r c C sd |d k r`xV|D ]N\}}}|dkr2t | f||} q|dkrLt| f||} qt| |||} qW | S )Nr r )r r getattr)r' r ro rx ry r r r r s z MemoryMappedTable._apply_replays)replayr c C s t j| j}|j| |S )N)r? r@ r append)rW r r r r r _append_replay s
z MemoryMappedTable._append_replayr c C s0 d||fi f}| j |}t| j||d| j|S )a
Compute zero-copy slice of this Table
Args:
offset (:obj:`int`, defaults to :obj:`0`):
Offset from start of table to slice
length (:obj:`int`, defaults to :obj:`None`):
Length of slice (default is until end of table starting from
offset)
Returns:
:class:`datasets.table.Table`:
r[ )r` ra )r r rb rl )rW r` ra r r r r r r[ s
zMemoryMappedTable.slicec O s8 dt j|t j|f}| j|}t| jj||| j|S )zY
Select records from a Table. See pyarrow.compute.filter for full usage.
r )r? r@ r r r' r rl )rW rx ry r r r r r r s
zMemoryMappedTable.filterc O s< dt j|t j|f}| j|}tt| jf||| j|S )a
Flatten this Table. Each column with a struct type is flattened
into one column per struct field. Other columns are left unchanged.
Args:
memory_pool (:obj:`MemoryPool`, defaults to :obj:`None`):
For memory allocations, if required, otherwise use default pool
Returns:
:class:`datasets.table.Table`:
r )r? r@ r r r r' rl )rW rx ry r r r r r r s
zMemoryMappedTable.flattenc O s8 dt j|t j|f}| j|}t| jj||| j|S )a
Make a new table by combining the chunks this table has.
All the underlying chunks in the ChunkedArray of each column are
concatenated into zero or one chunk.
Args:
memory_pool (:obj:`MemoryPool`, defaults to :obj:`None`):
For memory allocations, if required, otherwise use default pool
Returns:
:class:`datasets.table.Table`:
r )r? r@ r r r' r rl )rW rx ry r r r r r r s
z MemoryMappedTable.combine_chunksc O s< dt j|t j|f}| j|}tt| jf||| j|S )aq
Cast table values to another schema
Args:
target_schema (:obj:`Schema`):
Schema to cast to, the names and order of fields must match
safe (:obj:`bool`, defaults to :obj:`True`):
Check for overflows or other unsafe conversions
Returns:
:class:`datasets.table.Table`:
r )r? r@ r r r r' rl )rW rx ry r r r r r r - s
zMemoryMappedTable.castc O s8 dt j|t j|f}| j|}t| jj||| j|S )aa
EXPERIMENTAL: Create shallow copy of table by replacing schema
key-value metadata with the indicated new metadata (which may be None,
which deletes any existing metadata
Args:
metadata (:obj:`dict`, defaults to :obj:`None`):
Returns:
:class:`datasets.table.Table`: shallow_copy
r )r? r@ r r r' r rl )rW rx ry r r r r r r > s
z)MemoryMappedTable.replace_schema_metadatac O s8 dt j|t j|f}| j|}t| jj||| j|S )a]
Add column to Table at position.
A new table is returned with the column added, the original table
object is left unchanged.
Args:
i (:obj:`int`):
Index to place the column at.
field_ (:obj:`Union[str, pyarrow.Field]`):
If a string is passed then the type is deduced from the column
data.
column (:obj:`Union[pyarrow.Array, List[pyarrow.Array]]`):
Column data.
Returns:
:class:`datasets.table.Table`: New table with the passed column added.
r )r? r@ r r r' r rl )rW rx ry r r r r r r N s
z MemoryMappedTable.add_columnc O s8 dt j|t j|f}| j|}t| jj||| j|S )a
Append column at end of columns.
Args:
field_ (:obj:`Union[str, pyarrow.Field]`):
If a string is passed then the type is deduced from the column
data.
column (:obj:`Union[pyarrow.Array, List[pyarrow.Array]]`):
Column data.
Returns:
:class:`datasets.table.Table`:
New table with the passed column added.
r )r? r@ r r r' r rl )rW rx ry r r r r r r e s
z MemoryMappedTable.append_columnc O s8 dt j|t j|f}| j|}t| jj||| j|S )a
Create new Table with the indicated column removed.
Args:
i (:obj:`int`):
Index of column to remove.
Returns:
:class:`datasets.table.Table`:
New table without the column.
r )r? r@ r r r' r rl )rW rx ry r r r r r r x s
z MemoryMappedTable.remove_columnc O s8 dt j|t j|f}| j|}t| jj||| j|S )a
Replace column in Table at position.
Args:
i (:obj:`int`):
Index to place the column at.
field_ (:obj:`Union[str, pyarrow.Field]`):
If a string is passed then the type is deduced from the column
data.
column (:obj:`Union[pyarrow.Array, List[pyarrow.Array]]`):
Column data.
Returns:
:class:`datasets.table.Table`:
New table with the passed column set.
r )r? r@ r r r' r rl )rW rx ry r r r r r r s
z MemoryMappedTable.set_columnc O s8 dt j|t j|f}| j|}t| jj||| j|S )zJ
Create new table with columns renamed to provided names.
r )r? r@ r r r' r rl )rW rx ry r r r r r r s
z MemoryMappedTable.rename_columnsc O s8 dt j|t j|f}| j|}t| jj||| j|S )a
Drop one or more columns and return a new table.
Args:
columns (:obj:`List[str]`):
List of field names referencing existing columns.
Raises:
KeyError : if any of the passed columns name are not existing.
Returns:
:class:`datasets.table.Table`:
New table without the columns.
r )r? r@ r r r' r rl )rW rx ry r r r r r r s
zMemoryMappedTable.drop)N)N)N)r N) rc rd re r r r r r r ReplayrX r r rr rv staticmethodr r r[ r r r r r r r r r r r r r r )r9 r r s* ""
r TableBlockContainerc s` e Zd ZdZejeee d fdd Zdd Z dd Z
e d4ee eejf e
ejd
d d Zeeee ejd
ddZed5eee
ed
ddZeeed
ddZeed d
ddZed6ee ejef e
d dddZedd Zd7d d Zd d Zd d! Zd"d# Z d$d% Z d&d' Z d(d) Z d*d+ Z d,d- Z!d.d/ Z"d0d1 Z#d2d3 Z$ Z%S )8ConcatenationTableao
The table comes from the concatenation of several tables called blocks.
It enables concatenation on both axis 0 (append rows) and axis 1 (append columns).
The underlying tables are called "blocks" and can be either InMemoryTable
or MemoryMappedTable objects.
This allows to combine tables that come from memory or that are memory mapped.
When a ConcatenationTable is pickled, then each block is pickled:
- the InMemoryTable objects are pickled by copying all the data in memory;
- the MemoryMappedTable objects are pickled without copying the data into memory.
Instead, only the path to the memory mapped arrow file is pickled, as well as the list
of transforms to "replays" when reloading the table from the disk.
Its implementation requires to store each block separately.
The ``blocks`` attributes stores a list of list of blocks.
The first axis concatenates the tables along the axis 0 (it appends rows),
while the second axis concatenates tables along the axis 1 (it appends columns).
If some columns are missing when concatenating on axis 0, they are filled with null values.
This is done using `pyarrow.concat_tables(tables, promote=True)`.
You can access the fully combined table by accessing the ConcatenationTable.table attribute,
and the blocks by accessing the ConcatenationTable.blocks attribute.
)r' blocksc sL t j| || _x4|D ],}x&|D ] }t|ts"td| dq"W qW d S )Nz_The blocks of a ConcatenationTable must be InMemoryTable or MemoryMappedTable objects, but got rI )rg rX r rz r TypeError)rW r' r subtablessubtable)r9 r r rX s
zConcatenationTable.__init__c C s
d| j iS )Nr )r )rW r r r rr s z ConcatenationTable.__getstate__c C s& |d }| j |}tj| ||d d S )Nr )r )*_concat_blocks_horizontally_and_verticallyr rX )rW ru r r' r r r rv s
z ConcatenationTable.__setstate__r )r axisr c C s dd | D }|dkr$t j|ddS |dkr|xJt|D ]>\}}|dkrL|}q6x&t|j|jD ]\}}|j||}q\W q6W |S tdd S )Nc S s g | ]}t |d r|jn|qS )r' )r r' )r. r' r r r rQ s z5ConcatenationTable._concat_blocks..r T)Zpromoter z 'axis' must be either 0 or 1)r
concat_tables enumerater^ r r r r\ )r r Z pa_tablesrM r' r# ro colr r r _concat_blocks s z!ConcatenationTable._concat_blocks)r r c C sF g }x2t |D ]&\}}|s q| j|dd}|j| qW | j|ddS )Nr )r r )r r r )rB r Z pa_tables_to_concat_verticallyrM tablesZ"pa_table_horizontally_concatenatedr r r r s z=ConcatenationTable._concat_blocks_horizontally_and_verticallyNc s |d k rTg }xt |dd dD ]0\}}|rBt jt||dg}|t|7 }q W n< fdd|D }tdd |D r jd d |D d
d}|S ) Nc S s
t | tS )N)rz r )rA r r r s z2ConcatenationTable._merge_blocks..)key)r c s g | ]} j |d dqS )r )r )
_merge_blocks)r. row_block)rB r r rQ s z4ConcatenationTable._merge_blocks..c s s | ]}t |d kV qdS )r N)rK )r. r r r r r0 s z3ConcatenationTable._merge_blocks..c S s g | ]}|D ]}|q qS r r )r. r blockr r r rQ s r )r r r rh allr )rB r r Z
merged_blocksZ is_in_memoryZ block_groupr )rB r r
s z ConcatenationTable._merge_blocksc C s8 t |tr|S t |d tr*| j|ddS | j|S d S )Nr )r )rz r r )rB r r r r _consolidate_blocks s
z&ConcatenationTable._consolidate_blocksc C st | j |}t|tr(|}| |j|ggS t|d tr\| j|dd}dd |D }| ||S | j|}| ||S d S )Nr )r c S s g | ]
}|gqS r r )r. tr r r rQ + s z2ConcatenationTable.from_blocks..)r rz r r' r r )rB r r' r r r from_blocks# s
z ConcatenationTable.from_blocks)r r r c sp dd }dd d t t t td fdd
}||d }x*|d d
D ]}||}||||d }qHW | j|S )
a Create ConcatenationTable from list of tables.
Args:
tables (list of :class:`Table` or list of :obj:`pyarrow.Table`): List of tables.
axis: (``{0, 1}``, defaults to :obj:`0`, meaning over rows):
Axis to concatenate over, where ``0`` means over rows (vertically) and ``1`` means over columns
(horizontally).
*New in version 1.6.0*
c S s: t | tjrt| ggS t | tr.tj| jS | ggS d S )N)rz r r r r r? r@ r )r' r r r to_blocks> s
z1ConcatenationTable.from_tables..to_blocksc sH g }d}x,|D ]$}|d j }|j||f ||7 }qW fdd|D S )Nr c s" g | ] fd dd D qS )c s" g | ]}|j d d dqS )r r )r` ra )r[ )r. r )splitr r rQ N s zRConcatenationTable.from_tables.._split_like...r r )r. )blocks_to_split)r r rQ N s zGConcatenationTable.from_tables.._split_like..)r r )r Z blocks_likesplitsr` Z block_rowra r )r r _split_likeF s
z3ConcatenationTable.from_tables.._split_liker )r r c s |dkr| j | nr|dkrt| dkr@t|dkr@ | |} n"t|dkrbt| dkrb || }x"t|D ]\}}| | j | qlW | S )Nr r )extendrK r )rC r r rM Z
row_blocks)r r r _extend_blocksQ s
z6ConcatenationTable.from_tables.._extend_blocksr N)r )r )r r rf r )rB r r r r r r' Z table_blocksr )r r from_tables1 s
z ConcatenationTable.from_tablesc c s6 d}x,| j D ]"}t|d }||fV ||7 }q W d S )Nr )r rK )rW r` r ra r r r _slicesd s
zConcatenationTable._slicesc s | j j d} dk r n| j g }x| jD ]}t|d } dkrNP q2|kr`| q2| kr|jfdd|D | d q2|j fdd|D d\ q2W t||S )a
Compute zero-copy slice of this Table
Args:
offset (:obj:`int`, defaults to :obj:`0`):
Offset from start of table to slice
length (:obj:`int`, defaults to :obj:`None`):
Length of slice (default is until end of table starting from
offset)
Returns:
:class:`datasets.table.Table`:
)ra Nr c s g | ]}|j qS r )r[ )r. r )r` r r rQ s z,ConcatenationTable.slice..c s g | ]}|j qS r )r[ )r. r )ra r` r r rQ s )r r )r' r[ r r rK r r )rW r` ra r' r r Zn_rowsr )ra r` r r[ l s
zConcatenationTable.slicec sj | j j|f }g }xFt| j| jD ]4\\}}}|j|||j fdd|D q(W t||S )zY
Select records from a Table. See pyarrow.compute.filter for full usage.
c s g | ]}|j f qS r )r )r. r )rx ry submaskr r rQ s z-ConcatenationTable.filter..)r' r r^ r r r[ r r ) rW maskrx ry r' r r` ra r r )rx ry r r r s zConcatenationTable.filterc sJ t | jf }g }x(| jD ] }|j fdd|D q W t||S )a
Flatten this Table. Each column with a struct type is flattened
into one column per struct field. Other columns are left unchanged.
Args:
memory_pool (:obj:`MemoryPool`, defaults to :obj:`None`):
For memory allocations, if required, otherwise use default pool
Returns:
:class:`datasets.table.Table`:
c s g | ]}|j qS r )r )r. r )rx ry r r rQ s z.ConcatenationTable.flatten..)r r' r r r )rW rx ry r' r r r )rx ry r r s
zConcatenationTable.flattenc sF | j j }g }x(| jD ] }|j fdd|D qW t||S )a
Make a new table by combining the chunks this table has.
All the underlying chunks in the ChunkedArray of each column are
concatenated into zero or one chunk.
Args:
memory_pool (:obj:`MemoryPool`, defaults to :obj:`None`):
For memory allocations, if required, otherwise use default pool
Returns:
:class:`datasets.table.Table`:
c s g | ]}|j qS r )r )r. r )rx ry r r rQ s z5ConcatenationTable.combine_chunks..)r' r r r r )rW rx ry r' r r r )rx ry r r s
z!ConcatenationTable.combine_chunksc s ddl m} t| j|f||}|j|g }x| jD ]}g }t|} xz|D ]r}
g } x4|
jD ]* | j| j t
fddt | D q\W |fdd| D } | j }
|j|
j
|
f|| qLW |j| q6W t||S )aq
Cast table values to another schema
Args:
target_schema (:obj:`Schema`):
Schema to cast to, the names and order of fields must match
safe (:obj:`bool`, defaults to :obj:`True`):
Check for overflows or other unsafe conversions
Returns:
:class:`datasets.table.Table`:
r )r c 3 s | ]\}}|j kr|V qd S )N)ro )r. rM r )ro r r r0 s z*ConcatenationTable.cast..c s i | ]} |j |j qS r )ro )r. subfield)target_featuresr r r| s z+ConcatenationTable.cast..)featuresr r r' from_arrow_schemar rh r r popnextr arrow_schemar r )rW Z
target_schemarx ry r r' r r Z
new_tablesfieldsr Z subfieldsZ subfeaturesZ subschemar )ro r r r s
* zConcatenationTable.castc sH | j j }g }x(| jD ] }|j fdd|D qW t|| jS )aa
EXPERIMENTAL: Create shallow copy of table by replacing schema
key-value metadata with the indicated new metadata (which may be None,
which deletes any existing metadata
Args:
metadata (:obj:`dict`, defaults to :obj:`None`):
Returns:
:class:`datasets.table.Table`: shallow_copy
c s g | ]}|j qS r )r )r. r )rx ry r r rQ s z>ConcatenationTable.replace_schema_metadata..)r' r r r r )rW rx ry r' r r r )rx ry r r s
z*ConcatenationTable.replace_schema_metadatac O s
t dS )a]
Add column to Table at position.
A new table is returned with the column added, the original table
object is left unchanged.
Args:
i (:obj:`int`):
Index to place the column at.
field_ (:obj:`Union[str, pyarrow.Field]`):
If a string is passed then the type is deduced from the column
data.
column (:obj:`Union[pyarrow.Array, List[pyarrow.Array]]`):
Column data.
Returns:
:class:`datasets.table.Table`: New table with the passed column added.
N)r )rW rx ry r r r r s z ConcatenationTable.add_columnc O s
t dS )a
Append column at end of columns.
Args:
field_ (:obj:`Union[str, pyarrow.Field]`):
If a string is passed then the type is deduced from the column
data.
column (:obj:`Union[pyarrow.Array, List[pyarrow.Array]]`):
Column data.
Returns:
:class:`datasets.table.Table`:
New table with the passed column added.
N)r )rW rx ry r r r r s z ConcatenationTable.append_columnc sZ | j j|f }| j j| g }x*| jD ] }|j fdd|D q,W t||S )a
Create new Table with the indicated column removed.
Args:
i (:obj:`int`):
Index of column to remove.
Returns:
:class:`datasets.table.Table`:
New table without the column.
c s4 g | ],}|j kr,|j|j jf n|qS r )r r index)r. r )rx ry ro r r rQ % s z4ConcatenationTable.remove_column..)r' r r r r r )rW rM rx ry r' r r r )rx ry ro r r s z ConcatenationTable.remove_columnc O s
t dS )a
Replace column in Table at position.
Args:
i (:obj:`int`):
Index to place the column at.
field_ (:obj:`Union[str, pyarrow.Field]`):
If a string is passed then the type is deduced from the column
data.
column (:obj:`Union[pyarrow.Array, List[pyarrow.Array]]`):
Column data.
Returns:
:class:`datasets.table.Table`:
New table with the passed column set.
N)r )rW rx ry r r r r + s z ConcatenationTable.set_columnc s` | j jf }tt| j jg }x*| jD ] }|j fdd|D q2W t||S )zJ
Create new table with columns renamed to provided names.
c s. g | ]&}|j fd d|jD f qS )c s g | ] } | qS r r )r. ro )namesr r rQ G s z@ConcatenationTable.rename_columns...)r r )r. r )rx ry r r r rQ G s z5ConcatenationTable.rename_columns..)r' r r r^ r r r r )rW r rx ry r' r r r )rx ry r r r > s z!ConcatenationTable.rename_columnsc sF | j j}g }x*| jD ] }|j fdd|D qW t||S )a
Drop one or more columns and return a new table.
Args:
columns (:obj:`List[str]`):
List of field names referencing existing columns.
Raises:
KeyError : if any of the passed columns name are not existing.
Returns:
:class:`datasets.table.Table`:
New table without the columns.
c s, g | ]$ j fd dD fqS )c s g | ]}| j kr|qS r )r )r. c)r r r rQ ] s z6ConcatenationTable.drop...)r )r. )rx r ry )r r rQ ] s z+ConcatenationTable.drop..)r' r r r r )rW r rx ry r' r r r )rx r ry r r K s
zConcatenationTable.drop)r )N)r )r N)&rc rd re r r r r r rX rr rv r r
rf r r r r r r r r r r r r[ r r r r r r r r r r r r r r )r9 r r s:
&
$2
r )r r r c C s* t | } t| dkr | d S tj| |dS )aY
Concatenate tables.
Args:
tables (list of :class:`Table`): List of tables to be concatenated.
axis (``{0, 1}``, defaults to :obj:`0`, meaning over rows):
Axis to concatenate over, where ``0`` means over rows (vertically) and ``1`` means over columns
(horizontally).
*New in version 1.6.0*
Returns:
:class:`datasets.table.Table`:
If the number of input tables is > 1, then the returned table is a :obj:`datasets.table.ConcatenationTable`.
Otherwise if there's only one table, it is returned as is.
r r )r )rh rK r r )r r r r r r a s r )r' r c C sV t | tr t | tjr*tj fdd| jD S | f S d S )Nc s g | ]}|f qS r r )r. chunk)rx funcry r r rQ s z=_wrap_for_chunked_arrays..wrapper..)rz r ChunkedArrayZ
chunked_arraychunks)arrayrx ry )r )rx ry r r s z)_wrap_for_chunked_arrays..wrapperr )r r r )r r _wrap_for_chunked_arrays s r T)r pa_typec s t t|d ttjr jt|tjr4|jS j|krBS tj j
jrtj j
|rt dd |D t dd jD kr fdd|D }tj j
|t|j dS ntj jjrRtj j|r|jt tjkrNtjj
j|j|jS nFtj j|r~jdkr4tjd| d tjj
j j|jS n,tj jjrtj j|rtjj
j|j|jS tj j|r~tj t td
tj }jdkrtjd| d tjj
| j|jS n| r>tj j |r>tj j js"tj j!jr>t"d j d | d
| tj j|rttj jj rtt"d j d | j#|S t"dj d| dS )a- Improved version of pa.Array.cast
It supports casting pa.StructArray objects to re-order the fields.
It also let you control certain aspects of the casting, e.g. whether
to disable numbers (floats or ints) to strings.
Args:
array (pa.Array): PyArrow array to cast
pa_type (pa.DataType): target PyArrow type
allow_number_to_str (bool, defaults to :obj:`True`): Whether to allow casting numbers to strings.
Defaults to True.
Raises:
pa.ArrowInvalidError: if the arrow data casting fails
TypeError: if the target type is not supported according, e.g.
- if a field is missing
= if casting from numbers to strings and allow_number_to_str is False
Returns:
:obj:`List[pyarrow.Array]`: the casted array
)allow_number_to_strc s s | ] }|j V qd S )N)ro )r. r r r r r0 s z array_cast..c s g | ]} j |j|jqS r )r ro type)r. r )_cr r r rQ s z array_cast..)r r r zBNone values are converted to empty lists when converting array to z. More info: https://github.com/huggingface/datasets/issues/3676. This will raise an error in a future major version of `datasets`r z Couldn't cast array of type z to z% since allow_number_to_str is set to z Couldn't cast array of type
z
to
N)$r
array_castrz r ExtensionArraystorageZ
ExtensionTypeZ
wrap_arrayr types is_structr StructArrayr rh is_nullis_listis_fixed_size_listZ list_sizerK valuesFixedSizeListArrayZ
value_type
null_countwarningswarn ListArrayoffsetsr r int32 is_stringZ is_floating
is_integerr r )r r r arrays offsets_arrr )r r r r sV
&
r r )r featurec sL ddl m m} tt|dttjr.jt drBj
S tj j j
rt rtjtr fddjj D ttrtdd j
D tkr؇fd d
j D }tjj|tj d S ntj jj
rttrLjd }|j
jj
kr S jd kr:tjd
d tjjj |S nt rj dkrj t t jkrtj jjjj S nZjj}|j
jj
krS jd krtjd
d tjjj jjS ntj j j
rttrLjd kr0tjd
d tjjj jd S t rj dkrj t t jkrtj jjjj S nRtj!t"t d tj# }jd krtjd
d tjj|jjS tj jj
rt$||dS t ttt%fs2t$ |dS t&dj
d dS )a9 Cast an array to the arrow type that corresponds to the requested feature type.
For custom features like Audio or Image, it takes into account the "cast_storage" methods
they defined to enable casting from other arrow types.
Args:
array (pa.Array): the PyArrow array to cast
feature (FeatureType): the target feature type
allow_number_to_str (bool, defaults to :obj:`True`): Whether to allow casting numbers to strings.
Defaults to True.
Raises:
pa.ArrowInvalidError: if the arrow data casting fails
TypeError: if the target type is not supported according, e.g.
- if a field is missing
= if casting from numbers to strings and allow_number_to_str is False
Returns:
array (:obj:`pyarrow.Array`): the casted array
r )Sequenceget_nested_type)r cast_storagec s i | ]\}} |j d |qS ))ra )ra )r. ro
subfeature)r r r r r| s z)cast_array_to_feature..c s s | ] }|j V qd S )N)ro )r. r r r r r0
s z(cast_array_to_feature..c s g | ]\}} j ||qS r )r )r. ro r )r r r r rQ s z)cast_array_to_feature..)r r r zBNone values are converted to empty lists when converting array to z. More info: https://github.com/huggingface/datasets/issues/3676. This will raise an error in a future major version of `datasets`z Couldn't cast array of type
z
to
NrJ rJ )'features.featuresr r r cast_array_to_featurerz r r r r r r r r r r r= r r r rh r r r r r r r r ra rK r r r r r r r} r )r r r r r
casted_valuesr r )r r r r r r sj
& r c s ddl m tttjr"jtdr6jS tj j
j rt rttj t
rt fddj j D tt
rfddj D }tjj|tj dS ntj jj rttrjd krtjd
d tjjjjd S t rjdkrVjt tjkrtj jjj jS nZjj }|j jj krxS jd krtjd
d tjjjjj S ntj j j rttrjd krtjd
d tjjjjd S t rjdkrVjt tjkrtj jjj jS nRtj t td tj }jd krtjd
d tjj|jj S t t
tt!fsS t"d j d
dS )a& Embed data into an arrays's storage.
For custom features like Audio or Image, it takes into account the "embed_storage" methods
they defined to enable embedding external data (e.g. an image file) into an other arrow types.
Args:
array (pa.Array): the PyArrow array in which to embed data
feature (FeatureType): array features
Raises:
TypeError: if the target type is not supported according, e.g.
- if a field is missing
Returns:
array (:obj:`pyarrow.Array`): the casted array
r )r
embed_storagec s i | ]\}} |j d |qS ))ra )ra )r. ro r )r r r r r| ^ s z'embed_array_storage..c s g | ]\}} j ||qS r )r )r. ro r )_er r r rQ b s z'embed_array_storage..)r r r zKNone values are converted to empty lists when embedding array storage with z. More info: https://github.com/huggingface/datasets/issues/3676. This will raise an error in a future major version of `datasets`z Couldn't embed array of type
z
with
NrJ rJ )#r r embed_array_storagerz r r r r r r r r r r r= r r rh r r r r r r r r ra rK r r r r r r} r )r r r r
r r )r r r r r r
A s`
r
r )r' r c sR t jt |kr*td j d| d fdd|j D }tjj||jdS )a Cast a table to the arrow schema that corresponds to the requested features.
Args:
table (:obj:`pyarrow.Table`): PyArrow table to cast
features (Features): target features.
Returns:
table (:obj:`pyarrow.Table`): the casted table
zCouldn't cast
z
to
z!
because column names don't matchc s g | ]\}}t | |qS r )r )r. ro r )r' r r rQ s z*cast_table_to_features..)r, ) sortedr r\ r, r= r r r r )r' r r r )r' r cast_table_to_features s
r )r' r, c sf ddl m} |j|}t jt|kr@td j d| d fdd|j D }tj j
||dS ) a Cast a table to the arrow schema. Different from `cast_table_to_features`, this method can preserve nullability.
Args:
table (pa.Table): PyArrow table to cast
features (Features): target features.
Returns:
pa.Table: the casted table
r )r zCouldn't cast
z
to
z!
because column names don't matchc s g | ]\}}t | |qS r )r )r. ro r )r' r r rQ s z(cast_table_to_schema..)r, ) r r r r r r\ r, r= r r r )r' r, r r r r )r' r cast_table_to_schema s
r )r' c sF ddl m}m |jj} fdd|j D }tjj||j dS )zEmbed external data into a table's storage.
Args:
table (:obj:`pyarrow.Table`): PyArrow table in which to embed data
Returns:
table (:obj:`pyarrow.Table`): the table with embedded data
r )r require_storage_embedc s. g | ]&\}} |r"t | |n| qS r )r
)r. ro r )r r' r r rQ s z'embed_table_storage..)r, )
r r r r r, r= r r r r )r' r r r r )r r' r embed_table_storage s
r c C s6 | j |krt| |S | j j|jkr.| j|jS | S dS )a; Improved version of pa.Table.cast.
It supports casting to feature types stored in the schema metadata.
Args:
table (:obj:`pyarrow.Table`): PyArrow table to cast
schema (:obj:`pyarrow.Schema`): target PyArrow schema.
Returns:
table (:obj:`pyarrow.Table`): the casted table
N)r, r metadatar )r' r, r r r r s
r c s ddl m} |j| j}tdd |j D rg }g }x| jD ] | j j}| j }tj j
j rt |d s~|j
|kr|j|j
|j fdd j D q>|j| |j j q>W tjj||d}n| j
}|j
d d
|fd d |jD |jjjS )
a Improved version of pa.Table.flatten.
It behaves as pa.Table.flatten in a sense it does 1-step flatten of the columns with a struct type into one column per struct field,
but updates the metadata and skips decodable features unless the `decode` attribute of these features is set to False.
Args:
table (Table): PyArrow table to flatten
Returns:
Table: the flattened table
r )r c s s$ | ] }t |d o|j |kV qdS )r N)r r )r. r r r r r0 s z table_flatten..r c s g | ]} j d |j qS )rI )ro )r. r )r r r rQ s z!table_flatten..)r )Z max_depthc s i | ]} | |qS r r )r. Z column_name)
flat_featuresr r r| s z!table_flatten..)r r r r, anyr r ro r r r r r r r r r r r r r r )r' r r Z flat_arraysZflat_column_namesr r Z
flat_tabler )r r r r s*
r )r' functionc sT ddl m}m |j| j} fddx"|j D ]\}}| | | q6W dS )zVisit all arrays in a table and apply a function to them.
Args:
table (:obj:`pyarrow.Table`): PyArrow table to visit
function (Callable[[pa.Array], None]): function to apply to each array
r )r r c s t | tjr(x| jD ]}| qW nt | tjr:| j} | tjj| jrt d rt rt j
t r fdd j
j D xh j D ]\}}| j
|| qW nBtjj| jrt tr| j d nt r| j j
d S )Nr c s i | ]\}} |j d |qS ))ra )ra )r. ro r )r r r r r| s z1table_visitor.._visit..r )rz r r r r r r r r r r r r= r r rh r )r r r ro r )r _visitr )r r r s"
z table_visitor.._visitN)r r r r r, r= )r' r r r ro r r )r r r r
table_visitor s
r )r )T)T)@r? rs rm r functoolsr itertoolsr typingr r r r r r r
ZnumpyrU Zpyarrowr r r Z
utils.loggingr
r r r rc rp r r r r$ r r( r) rf r7 r rF rO rP r r r} r r r r r r r ArrayZDataTyper r r
r ZSchemar r r r r r r r r sj $
/ \
=
$ MZP)