/usr/local/lib64/python3.6/site-packages/pyarrow/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/pyarrow/__pycache__/parquet.cpython-36.pyc (66609B)
3
8%Eg
d/d0 Z?d1d2 Z@d3hZAG d4d5 d5ZBdcd6d7ZCd8ZDd9ZEG d:d; d;ZFddd>d?ZGd@dA ZHG dBdC dCZIdDZJdedGdHZKeJjLdIdJjMeEdKfdLe/eK_NdfdMdNZOeJjLdOdJjMeEdPfdQe/eO_NdgdTdUZPdVjLe7eP_NdWdX ZQdhdYdZZRdid[d\ZSdjd]d^ZTdkd_d`ZUdS )l ) defaultdict)futures)partialreduceN)
Collection)
ParquetReader
Statistics FileMetaDataRowGroupMetaDataColumnChunkMetaData
ParquetSchema ColumnSchema)LocalFileSystem
FileSystem _resolve_filesystem_and_path_ensure_filesystem)
filesystem)guid
_is_path_like_stringify_pathhdfsc C s, t | } tjj| }|jtkr$|jS | S d S )N)r urllibparseurlparsescheme_URI_STRIP_SCHEMESpath)r Z
parsed_uri r ;/usr/local/lib64/python3.6/site-packages/pyarrow/parquet.py
_parse_uri0 s
r c C s2 | d krt j|| S t j| } t|}| |fS d S )N)legacyfsresolve_filesystem_and_pathr r )Zpassed_filesystemr Z parsed_pathr r r _get_filesystem_and_path; s
r" c C sT t | tr>xD| D ](}t |tr(td}nd}||krdS qW nt | trPd| kS dS )Nr T F)
isinstancebyteschrstr)valbyteZ
compare_tor r r _check_contains_nullD s
r* Tc C s | dk rt | dks&tdd | D r.tdt| d d trF| g} |rxL| D ]D}x>|D ]6\}}}t|trtdd |D st|rZtdqZW qPW | S )z+
Check if filters are well-formed.
Nr c s s | ]}t |d kV qdS )r N)len).0fr r r W s z!_check_filters..zMalformed filtersc s s | ]}t |V qd S )N)r* )r, vr r r r. c s zBNull-terminated binary strings are not supported as filter values.) r+ any
ValueErrorr$ r' listallr* NotImplementedError)filterscheck_null_strings conjunctioncolopr( r r r _check_filtersR s
r: az Predicates are expressed in disjunctive normal form (DNF), like
``[[('x', '=', 0), ...], ...]``. DNF allows arbitrary boolean logical
combinations of single column predicates. The innermost tuples each
describe a single column predicate. The list of inner predicates is
interpreted as a conjunction (AND), forming a more selective and
multiple column predicate. Finally, the most outer list combines these
filters as a disjunction (OR).
Predicates may also be passed as List[Tuple]. This form is interpreted
as a single conjunction. To express OR in predicates, one must
use the (preferred) List[List[Tuple]] notation.
Each tuple has format: (``key``, ``op``, ``value``) and compares the
``key`` with the ``value``.
The supported ``op`` are: ``=`` or ``==``, ``!=``, ``<``, ``>``, ``<=``,
``>=``, ``in`` and ``not in``. If the ``op`` is ``in`` or ``not in``, the
``value`` must be a collection such as a ``list``, a ``set`` or a
``tuple``.
Examples:
.. code-block:: python
('x', '=', 0)
('y', 'in', ['a', 'b', 'c'])
('z', 'not in', {'a','b'})
c st ddl jt| jr| S t| dd} fdd g }x0| D ](} fdd|D }|jttj| qz<=z>=inznot inz,"{0}" is not a valid operator in predicates.)fieldisinr1 format)r8 r9 r( r? )dsr r convert_single_predicate s(
z8_filters_to_expression..convert_single_predicatec s g | ]\}}} |||qS r r )r, r8 r9 r( )rC r r
s z*_filters_to_expression..)
pyarrow.datasetdatasetr$
Expressionr: appendr operatorand_or_)r5 Zdisjunction_membersr7 Zconjunction_membersr )rC rB r _filters_to_expression s
rL c @ s e Zd ZdZd ddZdd Zed d
Zed d Zed
d Z edd Z
d ddZ d!ddZ d"ddZ
d#ddZd$dd Zd%d d ZdS )& ParquetFilea
Reader interface for a single Parquet file.
Parameters
----------
source : str, pathlib.Path, pyarrow.NativeFile, or file-like object
Readable source. For passing bytes or buffer-like file containing a
Parquet file, use pyarrow.BufferReader.
metadata : FileMetaData, default None
Use existing metadata object, rather than reading from file.
common_metadata : FileMetaData, default None
Will be used in reads for pandas schema metadata if not found in the
main file's metadata, no other uses at the moment.
memory_map : bool, default False
If the source is a file path, use a memory map to read file, which can
improve performance in some environments.
buffer_size : int, default 0
If positive, perform read buffering when deserializing individual
column chunks. Otherwise IO calls are unbuffered.
pre_buffer : bool, default False
Coalesce and issue file reads in parallel to improve performance on
high-latency filesystems (e.g. S3). If True, Arrow will use a
background I/O thread pool.
read_dictionary : list
List of column names to read directly as DictionaryArray.
coerce_int96_timestamp_unit : str, default None.
Cast timestamps that are stored in INT96 format to a particular
resolution (e.g. 'ms'). Setting to None is equivalent to 'ns'
and therefore INT96 timestamps will be infered as timestamps
in nanoseconds.
NFr c C s6 t | _| jj|||||||d || _| j | _d S )N)Zuse_memory_map buffer_size
pre_bufferread_dictionarymetadatacoerce_int96_timestamp_unit)r readeropencommon_metadata_build_nested_paths_nested_paths_by_prefix) selfsourcerQ rU rP
memory_maprN rO rR r r r __init__ s zParquetFile.__init__c C sv | j j}tt}x`t|D ]T\}}|d }|dd }x6|| j| |sLP dj||d f}|dd }q8W qW |S )Nr .)rS Z column_pathsr r2 enumeraterH join)rX pathsresultir keyrestr r r rV s z ParquetFile._build_nested_pathsc C s | j jS )N)rS rQ )rX r r r rQ s zParquetFile.metadatac C s | j jS )zG
Return the Parquet schema, unconverted to Arrow types
)rQ schema)rX r r r re s zParquetFile.schemac C s | j jS )zj
Return the inferred Arrow schema, converted from the whole Parquet
file's schema
)rS schema_arrow)rX r r r rf
s zParquetFile.schema_arrowc C s | j jS )N)rS num_row_groups)rX r r r rg s zParquetFile.num_row_groupsTc C s | j ||d}| jj|||dS )a
Read a single row group from a Parquet file.
Parameters
----------
i : int
Index of the individual row group that we want to read.
columns : list
If not None, only these columns will be read from the row group. A
column name may be a prefix of a nested field, e.g. 'a' will select
'a.b', 'a.c', and 'a.d.e'.
use_threads : bool, default True
Perform multi-threaded column reads.
use_pandas_metadata : bool, default False
If True and file has custom pandas schema metadata, ensure that
index columns are also loaded.
Returns
-------
pyarrow.table.Table
Content of the row group as a table (of columns)
)use_pandas_metadata)column_indices use_threads)_get_column_indicesrS read_row_group)rX rb columnsrj rh ri r r r rl s
zParquetFile.read_row_groupc C s | j ||d}| jj|||dS )a
Read a multiple row groups from a Parquet file.
Parameters
----------
row_groups : list
Only these row groups will be read from the file.
columns : list
If not None, only these columns will be read from the row group. A
column name may be a prefix of a nested field, e.g. 'a' will select
'a.b', 'a.c', and 'a.d.e'.
use_threads : bool, default True
Perform multi-threaded column reads.
use_pandas_metadata : bool, default False
If True and file has custom pandas schema metadata, ensure that
index columns are also loaded.
Returns
-------
pyarrow.table.Table
Content of the row groups as a table (of columns).
)rh )ri rj )rk rS read_row_groups)rX
row_groupsrm rj rh ri r r r rn 6 s
zParquetFile.read_row_groups c C s< |dkrt d| jj}| j||d}| jj||||d}|S )a
Read streaming batches from a Parquet file
Parameters
----------
batch_size : int, default 64K
Maximum number of records to yield per batch. Batches may be
smaller if there aren't enough rows in the file.
row_groups : list
Only these row groups will be read from the file.
columns : list
If not None, only these columns will be read from the file. A
column name may be a prefix of a nested field, e.g. 'a' will select
'a.b', 'a.c', and 'a.d.e'.
use_threads : boolean, default True
Perform multi-threaded column reads.
use_pandas_metadata : boolean, default False
If True and file has custom pandas schema metadata, ensure that
index columns are also loaded.
Returns
-------
iterator of pyarrow.RecordBatch
Contents of each batch as a record batch
Nr )rh )ro ri rj )rangerQ rg rk rS iter_batches)rX
batch_sizero rm rj rh ri Zbatchesr r r rr T s
zParquetFile.iter_batchesc C s | j ||d}| jj||dS )a
Read a Table from Parquet format,
Parameters
----------
columns : list
If not None, only these columns will be read from the file. A
column name may be a prefix of a nested field, e.g. 'a' will select
'a.b', 'a.c', and 'a.d.e'.
use_threads : bool, default True
Perform multi-threaded column reads.
use_pandas_metadata : bool, default False
If True and file has custom pandas schema metadata, ensure that
index columns are also loaded.
Returns
-------
pyarrow.table.Table
Content of the file as a table (of columns).
)rh )ri rj )rk rS Zread_all)rX rm rj rh ri r r r readz s
zParquetFile.readc C s | j |}| jj||dS )a
Read contents of file for the given columns and batch size.
Notes
-----
This function's primary purpose is benchmarking.
The scan is executed on a single thread.
Parameters
----------
columns : list of integers, default None
Select columns to read, if None scan all columns.
batch_size : int, default 64K
Number of rows to read at a time internally.
Returns
-------
num_rows : number of rows in file
)rs )rk rS
scan_contents)rX rm rs ri r r r ru s
zParquetFile.scan_contentsc s |d kr d S g }x&|D ] }| j kr|j j | qW |r jj} jd k rV jjnd }|rpd|krpt|}n|rd|krt|}ng }|d k r|r| fdd|D 7 }|S )Ns pandasc s" g | ]}t |ts jj|qS r )r$ dictrS Zcolumn_name_idx)r, descr)rX r r rD s z3ParquetFile._get_column_indices..)rW extendrQ rU _get_pandas_index_columns)rX Z column_namesrh indicesnameZfile_keyvaluesZcommon_keyvalues
index_columnsr )rX r rk s&
z ParquetFile._get_column_indices)NNNFr FN)NTF)NTF)rp NNTF)NTF)Nrp )F)__name__
__module__ __qualname____doc__r[ rV propertyrQ re rf rg rl rn rr rt ru rk r r r r rM s$
%
rM z [ ,;{}()
=]c C s t jd| S )N_)_SPARK_DISALLOWED_CHARSsub)r{ r r r _sanitized_spark_field_name s r c C s d|kr|g }d}xR| D ]J}|j }t|}||krVd}tj||j|j|j}|j| q|j| qW tj|| jd}||fS | dfS d S )NsparkFT)rQ ) r{ r par? typeZnullablerQ rH re ) re flavorZsanitized_fieldsschema_changedr? r{ Zsanitized_nameZsanitized_field
new_schemar r r _sanitize_schema s
r c s8 d|kr0 fddt jD }tjj||dS S d S )Nr c s g | ] } | qS r r )r, rb )tabler r rD s z#_sanitize_table..)re )rq Z num_columnsr Table from_arrays)r r r Z column_datar )r r _sanitize_table s r a version : {"1.0", "2.4", "2.6"}, default "1.0"
Determine which Parquet logical types are available for use, whether the
reduced set from the Parquet 1.x.x format or the expanded logical types
added in later format versions.
Files written with version='2.4' or '2.6' may not be readable in all
Parquet implementations, so version='1.0' is likely the choice that
maximizes file compatibility.
UINT32 and some logical types are only available with version '2.4'.
Nanosecond timestamps are only available with version '2.6'.
Other features such as compression algorithms or the new serialized
data page format must be enabled separately (see 'compression' and
'data_page_version').
use_dictionary : bool or list
Specify if we should use dictionary encoding in general or only for
some columns.
use_deprecated_int96_timestamps : bool, default None
Write timestamps to INT96 Parquet format. Defaults to False unless enabled
by flavor argument. This take priority over the coerce_timestamps option.
coerce_timestamps : str, default None
Cast timestamps to a particular resolution. If omitted, defaults are chosen
depending on `version`. By default, for ``version='1.0'`` (the default)
and ``version='2.4'``, nanoseconds are cast to microseconds ('us'), while
for other `version` values, they are written natively without loss
of resolution. Seconds are always cast to milliseconds ('ms') by default,
as Parquet does not have any temporal type with seconds resolution.
If the casting results in loss of data, it will raise an exception
unless ``allow_truncated_timestamps=True`` is given.
Valid values: {None, 'ms', 'us'}
data_page_size : int, default None
Set a target threshold for the approximate encoded size of data
pages within a column chunk (in bytes). If None, use the default data page
size of 1MByte.
allow_truncated_timestamps : bool, default False
Allow loss of data when coercing timestamps to a particular
resolution. E.g. if microsecond or nanosecond data is lost when coercing to
'ms', do not raise an exception. Passing ``allow_truncated_timestamp=True``
will NOT result in the truncation exception being ignored unless
``coerce_timestamps`` is not None.
compression : str or dict
Specify the compression codec, either on a general basis or per-column.
Valid values: {'NONE', 'SNAPPY', 'GZIP', 'BROTLI', 'LZ4', 'ZSTD'}.
write_statistics : bool or list
Specify if we should write statistics in general (default is True) or only
for some columns.
flavor : {'spark'}, default None
Sanitize schema or set other compatibility options to work with
various target systems.
filesystem : FileSystem, default None
If nothing passed, will be inferred from `where` if path-like, else
`where` is already a file-like object so no filesystem is needed.
compression_level : int or dict, default None
Specify the compression level for a codec, either on a general basis or
per-column. If None is passed, arrow selects the compression level for
the compression codec in use. The compression level has a different
meaning for each codec, so you have to read the documentation of the
codec you are using.
An exception is thrown if the compression codec does not allow specifying
a compression level.
use_byte_stream_split : bool or list, default False
Specify if the byte_stream_split encoding should be used in general or
only for some columns. If both dictionary and byte_stream_stream are
enabled, then dictionary is preferred.
The byte_stream_split encoding is valid only for floating-point data types
and should be combined with a compression codec.
data_page_version : {"1.0", "2.0"}, default "1.0"
The serialized Parquet data page format version to write, defaults to
1.0. This does not impact the file schema logical types and Arrow to
Parquet type casting behavior; for that use the "version" option.
use_compliant_nested_type : bool, default False
Whether to write compliant Parquet nested type (lists) as defined
`here `_, defaults to ``False``.
For ``use_compliant_nested_type=True``, this will write into a list
with 3-level structure where the middle level, named ``list``,
is a repeated group with a single field named ``element``::
group (LIST) {
repeated group list {
element;
}
}
For ``use_compliant_nested_type=False``, this will also write into a list
with 3-level structure, where the name of the single field of the middle
level ``list`` is taken from the element name for nested columns in Arrow,
which defaults to ``item``::
group (LIST) {
repeated group list {
item;
}
}
c @ sJ e Zd ZdjeZdddZd d
Zd d Zd
d Z dddZ
dd Z dS )
ParquetWritera
Class for incrementally building a Parquet file for Arrow tables.
Parameters
----------
where : path or file-like object
schema : arrow Schema
{}
writer_engine_version : unused
**options : dict
If options contains a key `metadata_collector` then the
corresponding value is assumed to be a list (or any object with
`.append` method) that will be filled with the file metadata instance
of the written file.
N1.0TsnappyFc
K s | d kr"|d k r d|kr d} nd} || _ |d k rBt||\}| _nd| _|| _|| _d | _t||dd\}}|d k rt|tj r|j
|d }| _q|j |d d }| _n|}|j dd | _
d}tj||f||||| |
| ||
|d
|| _d| _d S )
Nr TF)Zallow_legacy_filesystemwb) compressionmetadata_collectorZV2)
versionr use_dictionarywrite_statistics use_deprecated_int96_timestampscompression_leveluse_byte_stream_splitwriter_engine_versiondata_page_versionuse_compliant_nested_type)r r r re where file_handler r$ r r rT Zopen_output_streampop_metadata_collector_parquetr writeris_open)rX r re r r r r r r r r r r r r optionsr ZsinkZengine_versionr r r r[ d sF
zParquetWriter.__init__c C s t | ddr| j d S )Nr F)getattrclose)rX r r r __del__ s zParquetWriter.__del__c C s | S )Nr )rX r r r __enter__ s zParquetWriter.__enter__c O s | j dS )NF)r )rX argskwargsr r r __exit__ s zParquetWriter.__exit__c C s^ | j rt|| j| j}| js t|jj| jddsJdj|j| j}t|| j j
||d d S )NF)check_metadatazTTable schema does not match schema used to create file:
table:
{!s} vs.
file:
{!s})row_group_size) r r re r r AssertionErrorequalsrA r1 r write_table)rX r r msgr r r r s
zParquetWriter.write_tablec C sH | j r0| jj d| _ | jd k r0| jj| jj | jd k rD| jj d S )NF)r r r r rH rQ r )rX r r r r s
zParquetWriter.close) NNr Tr TNNFNr F)N) r} r~ r rA _parquet_writer_arg_docsr r[ r r r r r r r r r r R s$
7
r c C s t j| d jdd S )Ns pandasutf8r| )jsonloadsdecode) keyvaluesr r r ry s ry c @ s e Zd ZdZeedddddfddZeeedddddfddZd d
Z d d Z
d
d Z dd Z dd Z
dd ZdddZdS )ParquetDatasetPiecea
DEPRECATED: A single chunk of a potentially larger Parquet dataset to read.
The arguments will indicate to read either a single row group or all row
groups, and whether to add partition keys to the resulting pyarrow.Table.
.. deprecated:: 5.0
Directly constructing a ``ParquetDatasetPiece`` is deprecated, as well
as accessing the pieces of a ``ParquetDataset`` object. Specify
``use_legacy_dataset=False`` when constructing the ``ParquetDataset``
and use the ``ParquetDataset.fragments`` attribute instead.
Parameters
----------
path : str or pathlib.Path
Path to file in the file system where this piece is located.
open_file_func : callable
Function to use for obtaining file handle to dataset piece.
partition_keys : list of tuples
Two-element tuples of ``(column name, ordinal index)``.
row_group : int, default None
Row group to load. By default, reads all row groups.
file_options : dict
Options
rb)modeNc C s& t jdtdd | j||||| d S )Nz^ParquetDatasetPiece is deprecated as of pyarrow 5.0.0 and will be removed in a future version. )
stacklevel)warningswarnDeprecationWarning_init)rX r open_file_func file_options row_grouppartition_keysr r r r[ s
z ParquetDatasetPiece.__init__c C s t jt }|j| |||| |S )N)r __new__r )r r r r r rX r r r _create s
zParquetDatasetPiece._createc C s. t || _|| _|| _|p g | _|p&i | _d S )N)r r r r r r )rX r r r r r r r r r s
zParquetDatasetPiece._initc C s2 t |tsdS | j|jko0| j|jko0| j|jkS )NF)r$ r r r r )rX otherr r r __eq__ s
zParquetDatasetPiece.__eq__c C s dj t| j| j| j| jS )Nz-{}({!r}, row_group={!r}, partition_keys={!r}))rA r r} r r r )rX r r r __repr__ s z ParquetDatasetPiece.__repr__c C s^ d}t | jdkr6djdd | jD }|dj|7 }|| j7 }| jd k rZ|dj| j7 }|S )N r z, c s s | ]\}}d j ||V qdS )z{}={}N)rA )r, r{ indexr r r r. s z.ParquetDatasetPiece.__str__..zpartition[{}] z | row_group={})r+ r r_ rA r r )rX ra Z
partition_strr r r __str__ s
zParquetDatasetPiece.__str__c C s | j }|jS )zn
Return the file's metadata.
Returns
-------
metadata : FileMetaData
)rT rQ )rX r- r r r get_metadata# s z ParquetDatasetPiece.get_metadatac C s( | j | j}t|ts$t|f| j}|S )z1
Return instance of ParquetFile.
)r r r$ rM r )rX rS r r r rT . s
zParquetDatasetPiece.openTFc C s | j dk r| j }n(|dk r,t|f| j}nt| jf| j}t|||d}| jdk rf|j| jf|}n |jf |}t | j
dkr|dkrt dxTt | j
D ]F\} \}
} t
jt || dd} |j| j}
tjj| |
}|j|
|}qW |S )a
Read this piece as a pyarrow.Table.
Parameters
----------
columns : list of column names, default None
use_threads : bool, default True
Perform multi-threaded column reads.
partitions : ParquetPartitions, default None
file : file-like object
Passed to ParquetFile.
use_pandas_metadata : bool
If pandas metadata should be used or not.
Returns
-------
table : pyarrow.Table
N)rm rj rh r zMust pass partition setsi4)Zdtype)r rT rM r r rv r rl rt r+ r r1 r^ npfulllevels
dictionaryr ZDictionaryArrayr Z
append_column)rX rm rj
partitionsfilerh rS r r rb r{ r rz r Zarrr r r rt 7 s(
zParquetDatasetPiece.read)NTNNF)r} r~ r r r rT r[ staticmethodr r r r r r rt r r r r r s
r c @ s: e Zd ZdZd ddZdd Zedd Zed d
ZdS ) PartitionSeta
A data structure for cataloguing the observed Parquet partitions at a
particular level. So if we have
/foo=a/bar=0
/foo=a/bar=1
/foo=a/bar=2
/foo=b/bar=0
/foo=b/bar=1
/foo=b/bar=2
Then we have two partition sets, one for foo, another for bar. As we visit
levels of the partition hierarchy, a PartitionSet tracks the distinct
values and assigns categorical codes to use when reading the pieces
Parameters
----------
name : str
Name of the partition set. Under which key to collect all values.
keys : list
All possible values that have been collected for that partition set.
Nc C s0 || _ |p g | _dd t| jD | _d | _d S )Nc S s i | ]\}}||qS r r )r, rb kr r r
s z)PartitionSet.__init__..)r{ keysr^ key_indices _dictionary)rX r{ r r r r r[ s
zPartitionSet.__init__c C s< || j kr| j | S t| j }| jj| || j |< |S dS )z
Get the index of the partition value if it is known, otherwise assign
one
Parameters
----------
key : The value for which we want to known the index.
N)r r+ r rH )rX rc r r r r get_index s
zPartitionSet.get_indexc C sp | j d k r| j S t| jdkr&tdy dd | jD }tj|}W n tk
rd tj| j}Y nX || _ |S )Nr zNo known partition keysc S s g | ] }t |qS r )int)r, xr r r rD s z+PartitionSet.dictionary..)r r+ r r1 libarray)rX Z integer_keysr r r r r s
zPartitionSet.dictionaryc C s t | jt| jkS )N)r2 r sorted)rX r r r is_sorted s zPartitionSet.is_sorted)N) r} r~ r r r[ r r r r r r r r r y s
r c @ sD e Zd Zdd Zdd Zdd Zdd Zd d
Zd d Zd
d Z dS )ParquetPartitionsc C s g | _ t | _d S )N)r setpartition_names)rX r r r r[ s zParquetPartitions.__init__c C s
t | jS )N)r+ r )rX r r r __len__ s zParquetPartitions.__len__c C s
| j | S )N)r )rX rb r r r __getitem__ s z ParquetPartitions.__getitem__c C s* t |tstd| j|jko(| j|jkS )Nz0`other` must be an instance of ParquetPartitions)r$ r TypeErrorr r )rX r r r r r s
zParquetPartitions.equalsc C s$ y
| j |S tk
r tS X d S )N)r r NotImplemented)rX r r r r r s
zParquetPartitions.__eq__c C sV |t | jkrF|| jkr&tdj|t|}| jj| | jj| | j| j|S )aT
Record a partition value at a particular level, returning the distinct
code for that value at that level.
Example:
partitions.get_index(1, 'foo', 'a') returns 0
partitions.get_index(1, 'foo', 'b') returns 1
partitions.get_index(1, 'foo', 'c') returns 2
partitions.get_index(1, 'foo', 'a') returns 0
Parameters
----------
level : int
The nesting level of the partition we are observing
name : str
The partition name
key : str or int
The partition value
z1{} was the name of the partition in another level) r+ r r r1 rA r rH addr )rX levelr{ rc Zpart_setr r r r s
zParquetPartitions.get_indexc C s` |\}}|\}}}||kr dS t |} |dkrt|tsDtd| j|sPtdtdd |D dkrptd |t tt|} n t|t rt|trtd
|| | j
| j | j }
|d ks|d kr|
|kS |d
kr|
|kS |dk r|
|k S |dkr|
|kS |dkr|
|kS |dkr*|
|kS |dkr<|
|kS |dkrN|
|kS td|d d S )NTr> not inz '%s' object is not a collectionz+Cannot use empty collection as filter valuec S s h | ] }t |qS r )r )r, itemr r r s z=ParquetPartitions.filter_accepts_partition..r\ z8All elements of the collection '%s' must be of same typez-Op '%s' not supported with a collection valuer; z==z!=r< r= z<=z>=z+'%s' is not a valid operator in predicates.> r r> )
r r$ r r r} r1 r+ nextiterr' r r Zas_py) rX part_keyfilterr Zp_columnZ
p_value_indexZf_columnr9 Zf_valueZf_typeZp_valuer r r filter_accepts_partition sL
z*ParquetPartitions.filter_accepts_partitionN)
r} r~ r r[ r r r r r r r r r r r s r c @ s> e Zd ZdddZdd Zd d
Zd d Zd
d Zdd ZdS )ParquetManifestN/hiver\ c C s t ||\}}|| _|| _|| _t|| _|| _t | _g | _ || _
t j |d| _
d | _d | _| jd| jg | j jdd d | jd kr| j| _| j
j d S )N) max_workersr c S s | j S )N)r )piecer r r > s z*ParquetManifest.__init__..)rc )r" r r pathsepr dirpathpartition_schemer r pieces_metadata_nthreadsr ThreadPoolExecutor _thread_poolcommon_metadata_path
metadata_path _visit_levelsortshutdown)rX r r r r r metadata_nthreadsr r r r[ ) s$
zParquetManifest.__init__c s j }t|j \}}}g }xX|D ]P} jj | f}
| jdrJ|
_q$| jdr\|
_q$j| rjq$q$|j |
q$W fdd|D } |j
| j
t |dkrt | dkrt dj
n(t | dkrj|| | n j|| d S )NZ_common_metadata _metadatac s$ g | ] }t |sjj |fqS r )_is_private_directoryr r_ )r, r ) base_pathrX r r rD X s z0ParquetManifest._visit_level..r z,Found files in an intermediate directory: {})r r walkr r_ endswithr r _should_silently_excluderH r r+ r1 rA _visit_directories _push_pieces) rX r r part_keysfsr directoriesfilesZfiltered_filesr full_pathZfiltered_directoriesr )r rX r r F s,
z ParquetManifest._visit_levelc C s0 |j dp.|j dp.|jdp.|jdp.|tkS )Nz.crcz _$folder$r] r )r
startswithEXCLUDED_PARQUET_PATHS)rX file_namer r r r g s
z(ParquetManifest._should_silently_excludec
C s g }x|D ]~}t || j\}}t|\}} | jj||| }
|||
fg } || jk rv| jj| j|d || } |j | q
| j|d || q
W |rt
j | d S )Nr\ ) _path_splitr _parse_hive_partitionr r r r submitr rH r wait)
rX r r r Z futures_listr headtailr{ rc r Z
dir_part_keysfuturer r r r n s
z"ParquetManifest._visit_directoriesc C s& | j dkrt|S tdj| j d S )Nr zpartition schema: {})r r r4 rA )rX dirnamer r r _parse_partition s
z ParquetManifest._parse_partitionc s j j fdd|D d S )Nc s g | ]}t j| jd qS ))r r )r r r )r, r )r rX r r rD s z0ParquetManifest._push_pieces..)r rx )rX r r r )r rX r r s z ParquetManifest._push_pieces)NNr r r\ ) r} r~ r r[ r r r r" r r r r r r ' s
!r c C s" d| krt dj| | jddS )Nr; z3Directory name did not appear to be a partition: {}r\ )r1 rA split)valuer r r r s r c C s, t jj| \}}|jds$|jdo*d|kS )Nr r] r; )osr r# r )r r r r r r r s r c C s: | j |d }| d | | |d }}|j|}||fS )Nr\ )rfindrstrip)r seprb r r r r r r s
r Z_SUCCESSc @ s e Zd ZdZdS )_ParquetDatasetMetadatar rZ rP rU rN N)r rZ rP rU rN )r} r~ r __slots__r r r r r) s r) c C sF | j d k r*t| j tj r*| j j|dd}t||| j| j| j| j dS )Nr )r )rQ rZ rP rU rN )
r r$ r r rT rM rZ rP rU rN )rF r metar r r _open_dataset_file s
r, z['{}' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version.{}a read_dictionary : list, default None
List of names or column paths (for nested types) to read directly
as DictionaryArray. Only supported for BYTE_ARRAY storage. To read
a flat column as dictionary-encoded pass the column name. For
nested types, you must pass the full column "path", which could be
something like level1.level2.list.item. Refer to the Parquet
file's schema to obtain the paths.
memory_map : bool, default False
If the source is a file path, use a memory map to read file, which can
improve performance in some environments.
buffer_size : int, default 0
If positive, perform read buffering when deserializing individual
column chunks. Otherwise IO calls are unbuffered.
partitioning : Partitioning or str or list of str, default "hive"
The partitioning scheme for a partitioned dataset. The default of "hive"
assumes directory names with key=value pairs like "/year=2009/month=11".
In addition, a scheme like "/2009/11" is also supported, in which case
you need to specify the field names or a full schema. See the
``pyarrow.dataset.partitioning()`` function for more details.c @ s e Zd ZdjeeZd(dd Zd)d
d Zd d
Z dd Z
dd Z d*ddZ dd Z
dd Zdd Zedd Zed d Zed d Zed d! Zed"d# Zeejd$Zed%d& Zeejd'ZdS )+ParquetDataseta
Encapsulates details of reading a complete Parquet dataset possibly
consisting of multiple files and partitions in subdirectories.
Parameters
----------
path_or_paths : str or List[str]
A directory name, single file name, or list of file names.
filesystem : FileSystem, default None
If nothing passed, paths assumed to be found in the local on-disk
filesystem.
metadata : pyarrow.parquet.FileMetaData
Use metadata obtained elsewhere to validate file schemas.
schema : pyarrow.parquet.Schema
Use schema obtained elsewhere to validate file schemas. Alternative to
metadata parameter.
split_row_groups : bool, default False
Divide files into pieces for each row group in the file.
validate_schema : bool, default True
Check that individual file schemas are all the same / compatible.
filters : List[Tuple] or List[List[Tuple]] or None (default)
Rows which do not match the filter predicate will be removed from scanned
data. Partition keys embedded in a nested directory structure will be
exploited to avoid loading files at all if they contain no matching rows.
If `use_legacy_dataset` is True, filters can only reference partition
keys and only a hive-style directory structure is supported. When
setting `use_legacy_dataset` to False, also within-file level filtering
and different partitioning schemes are supported.
{1}
metadata_nthreads : int, default 1
How many threads to allow the thread pool which is used to read the
dataset metadata. Increasing this is helpful to read partitioned
datasets.
{0}
use_legacy_dataset : bool, default True
Set to False to enable the new code path (experimental, using the
new Arrow Dataset API). Among other things, this allows to pass
`filters` for all columns and not only the partition keys, enables
different partitioning schemes, etc.
pre_buffer : bool, default True
Coalesce and issue file reads in parallel to improve performance on
high-latency filesystems (e.g. S3). If True, Arrow will use a
background I/O thread pool. This option is only supported for
use_legacy_dataset=False. If using a filesystem layer that itself
performs readahead (e.g. fsspec's S3FS), disable readahead for best
results.
coerce_int96_timestamp_unit : str, default None.
Cast timestamps that are stored in INT96 format to a particular resolution
(e.g. 'ms'). Setting to None is equivalent to 'ns' and therefore INT96
timestamps will be infered as timestamps in nanoseconds.
NFTr\ r r c C sR |
d kr t |trd}
nd}
|
sDt|||| | |
| |||||||dS tj| }|S )NFT)
r r5 partitioningrP rZ rN rO rR re rQ split_row_groupsvalidate_schemar )r$ r _ParquetDatasetV2objectr )cls
path_or_pathsr re rQ r/ r0 r5 r rP rZ rN r. use_legacy_datasetrO rR rX r r r r s(
zParquetDataset.__new__c C sr | dkrt dt | _|}t|tr.|d }t||\| j_}t|tr\dd |D | _n
t|| _| | j_ |
| j_
| | j_ t || j
|tt| jd\| _| _| _| _| jd k r| j
j| j}t||
d| j_W d Q R X nd | j_|d ko| jd k r$| j
j| j}t||
d| _W d Q R X n|| _|| _|| _|rDtd|d k r`t|}| j | |rn| j d S ) Nr zVOnly "hive" for hive-like partitioning is supported when using use_legacy_dataset=Truer c S s g | ] }t |qS r )r )r, r r r r rD @ s z+ParquetDataset.__init__..)r r )rZ z$split_row_groups not yet implemented) r1 r) r
r$ r2 r" r r` r rP rZ rN _make_manifest_fsr r, _pieces _partitionsr r rT
read_metadatarU rQ re r/ r4 r: _filtervalidate_schemas)rX r4 r re rQ r/ r0 r5 r rP rZ rN r. r5 rO rR Za_pathr r- r r r r[ / sH
zParquetDataset.__init__c C sz t |tstd| jj|jjkr&dS x$dD ] }t| |t||kr,dS q,W x(dD ] }t| j|t|j|krRdS qRW dS )Nz-`other` must be an instance of ParquetDatasetFr` r8 r9 r r rU rQ re r/ rZ rN T) r` r8 r9 r r rU rQ re r/ )rZ rN )r$ r- r r7 __class__r r
)rX r propr r r r m s
zParquetDataset.equalsc C s$ y
| j |S tk
r tS X d S )N)r r r )rX r r r r r s
zParquetDataset.__eq__c C s | j d kr>| jd kr>| jd k r*| jj| _qR| jd j j| _n| jd krR| j j| _| jj }| jd k rx2| jjD ]&}|j|dkrp|j|}|j |}qpW x@| jD ]6}|j }|jj }|j
|ddst dj |||qW d S )Nr r\ F)r z-Schema in {!s} was different.
{!s}
vs
{!s})
rQ re rU r8 r to_arrow_schemar9 r get_field_indexremover r1 rA )rX Zdataset_schemaZpartition_nameZ field_idxr Z
file_metadataZ file_schemar r r r< s&
z ParquetDataset.validate_schemasc
C sr g }x,| j D ]"}|j||| j|d}|j| q W tj|}|rn| j }|jjpRi } |rnd| krn|j d|i}|S )a
Read multiple Parquet files as a single pyarrow.Table.
Parameters
----------
columns : List[str]
Names of columns to read from the file.
use_threads : bool, default True
Perform multi-threaded column reads
use_pandas_metadata : bool, default False
Passed through to each dataset piece.
Returns
-------
pyarrow.Table
Content of the file as a table (of columns).
)rm rj r rh s pandas)
r8 rt r9 rH r Z
concat_tables_get_common_pandas_metadatare rQ replace_schema_metadata)
rX rm rj rh Ztablesr r Zall_datarU Zcurrent_metadatar r r rt s
zParquetDataset.readc K s | j f ddi|S )a
Read dataset including pandas metadata, if any. Other arguments passed
through to ParquetDataset.read, see docstring for further details.
Parameters
----------
**kwargs : optional
All additional options to pass to the reader.
Returns
-------
pyarrow.Table
Content of the file as a table (of columns).
rh T)rt )rX r r r r read_pandas s zParquetDataset.read_pandasc C s" | j d krd S | j j}|jdd S )Ns pandas)rU rQ get)rX r r r r rC s
z*ParquetDataset._get_common_pandas_metadatac s< | j j fddfddfdd| jD | _d S )Nc s t fddt| jD S )Nc 3 s | ]\}} ||V qd S )Nr )r, r r )accepts_filterr r r r. s zEParquetDataset._filter..one_filter_accepts..)r3 r^ r )r r )rG )r r one_filter_accepts s z2ParquetDataset._filter..one_filter_acceptsc s t fddD S )Nc 3 s& | ] }t fd d|D V qdS )c 3 s | ]} |V qd S )Nr )r, r- )rH r r r r. s zOParquetDataset._filter..all_filters_accept...N)r3 )r, r7 )rH r r r r. s zEParquetDataset._filter..all_filters_accept..)r0 )r )r5 rH )r r all_filters_accept s z2ParquetDataset._filter..all_filters_acceptc s g | ]} |r|qS r r )r, p)rI r r rD s z*ParquetDataset._filter..)r9 r r8 )rX r5 r )rG rI r5 rH r r; s zParquetDataset._filterc C s t jtjddtdd | jS )NzParquetDataset.piecesz{ Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.r )r )r r _DEPR_MSGrA r r8 )rX r r r r s
zParquetDataset.piecesc C s t jtjddtdd | jS )NzParquetDataset.partitionsz~ Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.partitioning' attribute instead.r )r )r r rK rA r r9 )rX r r r r s
zParquetDataset.partitionsc C s t jtjddtdd | jjS )NzParquetDataset.memory_mapr r )r )r r rK rA r r
rZ )rX r r r rZ s
zParquetDataset.memory_mapc C s t jtjddtdd | jjS )Nz ParquetDataset.read_dictionaryr r )r )r r rK rA r r
rP )rX r r r rP s
z ParquetDataset.read_dictionaryc C s t jtjddtdd | jjS )NzParquetDataset.buffer_sizer r )r )r r rK rA r r
rN )rX r r r rN s
zParquetDataset.buffer_sizez _metadata.fsc C s t jtjddtdd | jjS )NzParquetDataset.fsz| Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.filesystem' attribute instead.r )r )r r rK rA r r
r )rX r r r r s
zParquetDataset.fsz_metadata.common_metadata)NNNNFTNr\ NFr r NTN)NNNFTNr\ NFr r TTN)NTF)r} r~ r rA _read_docstring_common_DNF_filter_docr r r[ r r r< rt rE rC r; r r r rZ rP rN rI
attrgetterr7 r rU r r r r r- s< 5
9
(
r- r r\ c C s d }d }d }t | tr*t| dkr*| d } t| rp|j| rpt| ||t|dd|d}|j}|j}|j } |j
}nft | ts| g} t| dkrt dg } x<| D ]4}
|j |
st
dj|
tj|
|d} | j| qW | |||fS ) Nr\ r r r )r r r r z Must pass at least one file pathzPassed non-file path: {})r )r$ r2 r+ r isdirr r r r r r r1 isfileOSErrorrA r r rH ) r4 r r r r r r r manifestr r r r r r r6 - s6
r6 c C s t | tpt | tjS )N)r$ r r )r r r r _is_local_file_systemT s rS c @ st e Zd ZdZdddZedd Zdd
d Zd d
Zedd Z edd Z
edd Z edd Z edd Z
dS )r1 zC
ParquetDataset shim using the Dataset API under the hood.
Nr FTc K s dd l j} x2dD ]*\}
}|
| kr| |
|k rtd
j|
qW | |
d }|rZ|jd|d |d k rn|j|d
|| _|o~t|| _|d k rt||d}n|d kr|rt |d}t
|dr|d k rt | rt dt
| d }t|trt|d krl|d }nht|rht|}|d krTytj|\}}W n tk
rR t |d}Y nX |j|jrl|}n|}|d k rd| _|jdd | jf |}|j||}| j|g|j||j d| _ d S d| _| jf |}|dkr| j j dd}| j|||||d| _ d S ) Nr re rQ r/ Fr0 Tr r\ z;Keyword '{0}' is not yet supported with the new Dataset API)rO rR )Zuse_buffered_streamrN )Zdictionary_columns)Zuse_mmap
__fspath__zQPath-like objects with __fspath__ must only be used with local file systems, not )Z!enable_parallel_column_conversion)re rA r r )Zinfer_dictionary)r rA r. ignore_prefixesre NrQ Nr/ Fr0 Tr r\ )rV rW rX rY rZ ) rE rF r1 rA update_filtersrL _filter_expressionr r hasattrrS r r r$ r2 r+ r r r Zfrom_uriZ
get_file_infois_file"_enable_parallel_column_conversionParquetFileFormatZ
make_fragmentZFileSystemDatasetZphysical_schemar _datasetZHivePartitioningZdiscover)rX r4 r r5 r. rP rN rZ rU rO rR r rB keyworddefaultZ read_optionsZ single_fileparquet_formatfragmentr r r r[ _ s|
z_ParquetDatasetV2.__init__c C s | j jS )N)rb re )rX r r r re s z_ParquetDatasetV2.schemac C s | j j}|dk rN|rN|rNd|krNdd t|D }t|tt|t| }| jr\|r\d}| jj|| j|d}|r|rd|kr|j jpi }|j d|d i |j
|}|S )a
Read (multiple) Parquet files as a single pyarrow.Table.
Parameters
----------
columns : List[str]
Names of columns to read from the dataset. The partition fields
are not automatically included (in contrast to when setting
``use_legacy_dataset=True``).
use_threads : bool, default True
Perform multi-threaded column reads.
use_pandas_metadata : bool, default False
If True and file has custom pandas schema metadata, ensure that
index columns are also loaded.
Returns
-------
pyarrow.Table
Content of the file as a table (of columns).
Ns pandasc S s g | ]}t |ts|qS r )r$ rv )r, r8 r r r rD s z*_ParquetDatasetV2.read..F)rm r rj ) re rQ ry r2 r r` rb Zto_tabler] r[ rD )rX rm rj rh rQ r| r Z new_metadatar r r rt s"
z_ParquetDatasetV2.readc K s | j f ddi|S )z
Read dataset including pandas metadata, if any. Other arguments passed
through to ParquetDataset.read, see docstring for further details.
rh T)rt )rX r r r r rE s z _ParquetDatasetV2.read_pandasc C s& t jtjddtdd t| jj S )NzParquetDataset.piecesz' Use the '.fragments' attribute insteadr )r )r r rK rA r r2 rb
get_fragments)rX r r r r s
z_ParquetDatasetV2.piecesc C s t | jj S )N)r2 rb rg )rX r r r fragments s z_ParquetDatasetV2.fragmentsc C s | j jS )N)rb r )rX r r r r s z_ParquetDatasetV2.filesc C s | j jS )N)rb r )rX r r r r s z _ParquetDatasetV2.filesystemc C s | j jS )zH
The partitioning of the Dataset source, if discovered.
)rb r. )rX r r r r. s z _ParquetDatasetV2.partitioning) NNr NNFNTN)NTF)r} r~ r r r[ r re rt rE r rh r r r. r r r r r1 Z s
a
8 r1 a
{0}
Parameters
----------
source : str, pyarrow.NativeFile, or file-like object
If a string passed, can be a single file name or directory name. For
file-like objects, only read a single file. Use pyarrow.BufferReader to
read a file contained in a bytes or buffer-like object.
columns : list
If not None, only these columns will be read from the file. A column
name may be a prefix of a nested field, e.g. 'a' will select 'a.b',
'a.c', and 'a.d.e'. If empty, no columns will be read. Note
that the table will still have the correct num_rows set despite having
no columns.
use_threads : bool, default True
Perform multi-threaded column reads.
metadata : FileMetaData
If separately computed
{1}
use_legacy_dataset : bool, default False
By default, `read_table` uses the new Arrow Datasets API since
pyarrow 1.0.0. Among other things, this allows to pass `filters`
for all columns and not only the partition keys, enables
different partitioning schemes, etc.
Set to True to use the legacy behaviour.
ignore_prefixes : list, optional
Files matching any of these prefixes will be ignored by the
discovery process if use_legacy_dataset=False.
This is matched to the basename of a path.
By default this is ['.', '_'].
Note that discovery happens only if a directory is passed as source.
filesystem : FileSystem, default None
If nothing passed, paths assumed to be found in the local on-disk
filesystem.
filters : List[Tuple] or List[List[Tuple]] or None (default)
Rows which do not match the filter predicate will be removed from scanned
data. Partition keys embedded in a nested directory structure will be
exploited to avoid loading files at all if they contain no matching rows.
If `use_legacy_dataset` is True, filters can only reference partition
keys and only a hive-style directory structure is supported. When
setting `use_legacy_dataset` to False, also within-file level filtering
and different partitioning schemes are supported.
{3}
pre_buffer : bool, default True
Coalesce and issue file reads in parallel to improve performance on
high-latency filesystems (e.g. S3). If True, Arrow will use a
background I/O thread pool. This option is only supported for
use_legacy_dataset=False. If using a filesystem layer that itself
performs readahead (e.g. fsspec's S3FS), disable readahead for best
results.
coerce_int96_timestamp_unit : str, default None.
Cast timestamps that are stored in INT96 format to a particular
resolution (e.g. 'ms'). Setting to None is equivalent to 'ns'
and therefore INT96 timestamps will be infered as timestamps
in nanoseconds.
Returns
-------
{2}
Fr c C s | s|d k rt dy t| ||
||| || |
|d
}W nj tk
r |d k rTt d|
dkrdt dt| |\}}|d k r|j|} t| |||| |
|d}Y nX |j|||dS | d k rt dt| rt| |||| |||
|d }nt| |||| |d
}|j|||dS ) NzThe 'metadata' keyword is no longer supported with the new datasets-based implementation. Specify 'use_legacy_dataset=True' to temporarily recover the old behaviour.) r r. rZ rP rN r5 rU rO rR zWthe 'filters' keyword is not supported when the pyarrow.dataset module is not availabler z\the 'partitioning' keyword is not supported when the pyarrow.dataset module is not available)rQ rP rZ rN rO rR )rm rj rh zMThe 'ignore_prefixes' keyword is only supported when use_legacy_dataset=False)rQ rZ rP rN r r5 r. rR )rQ rP rZ rN rR ) r1 r1 ImportErrorr Zopen_input_filerM rt r r- )rY rm rj rQ rh rZ rP r r5 rN r. r5 rU rO rR rF r pfr r r
read_tableb sh
rk z}Read a Table from Parquet format
Note: starting with pyarrow 1.0, the default for `use_legacy_dataset` is
switched to False.
zuse_pandas_metadata : bool, default False
If True and file has custom pandas schema metadata, ensure that
index columns are also loaded.z=pyarrow.Table
Content of the file as a table (of columns)c K s t | f|dd|S )NT)rm rh )rk )rY rm r r r r rE s rE zcRead a Table from Parquet format, also reading DataFrame
index values if known in the file metadataz4**kwargs : additional options for :func:`read_table`zgpyarrow.Table
Content of the file as a Table of Columns, including DataFrame
indexes as columns1.0r c K s |j d|}|}yPt|| jf| || ||||
| |||
|||d|}|j| |d W d Q R X W nH tk
r t|rytjt| W n tj k
r Y nX Y nX d S )N
chunk_size)r r r r r coerce_timestampsdata_page_sizeallow_truncated_timestampsr r r r r r )r )
r r re r Exceptionr r% rB r error)r r r r r r r r ro rq rp r r r r r r r Z use_int96r r r r r s8
r z
Write a Table to Parquet format.
Parameters
----------
table : pyarrow.Table
where : string or pyarrow.NativeFile
row_group_size : int
The number of rows per rowgroup
{}
**kwargs : optional
Additional options for ParquetWriter
c C sJ | j rF| j| rFy| j| W n" tk
rD | j|s@tY nX d S )N)Z _isfilestoreexistsmkdirrQ r )r r r r r _mkdir_if_not_exists s
rv c s |dkr t |trd}nd}|sddlj}|jdd}|jdd} d}
|jddd} dk rjfd d
} |dk rt|
jd |j } | jf |}
|dk rt |}d}|r| j
|j }|j |d d
}|j
| ||| |
||| | d dS tj||\}}t|| |jdd|dk rt|dkr| j fdd|D } j|dd} jj|}t|dkrntd| j }x,| j jD ] }||kr~|j|j|}q~W x|j|D ]\}}t |ts|f}djdd t||D }t j j ||dd}t|dj||g |r||}n
t d }dj||g}dj||g} |j | d } t!|| fdi| W dQ R X dk rd j"| qW nn|r|d}n
t d }dj||g} |j | d } t!| | fdi| W dQ R X dk rd j"| dS ) a' Wrapper around parquet.write_table for writing a Table to
Parquet format by partitions.
For each combination of partition columns and values,
a subdirectories are created in the following
manner:
root_dir/
group1=value1
group2=value1
.parquet
group2=value2
.parquet
group1=valueN
group2=value1
.parquet
group2=valueN
.parquet
Parameters
----------
table : pyarrow.Table
root_path : str, pathlib.Path
The root directory of the dataset
filesystem : FileSystem, default None
If nothing passed, paths assumed to be found in the local on-disk
filesystem
partition_cols : list,
Column names by which to partition the dataset
Columns are partitioned in the order they are given
partition_filename_cb : callable,
A callback function that takes the partition key(s) as an argument
and allow you to override the partition filename. If nothing is
passed, the filename will consist of a uuid.
use_legacy_dataset : bool
Default is True unless a ``pyarrow.fs`` filesystem is passed.
Set to False to enable the new code path (experimental, using the
new Arrow Dataset API). This is more efficient when using partition
columns, but does not (yet) support `partition_filename_cb` and
`metadata_collector` keywords.
**kwargs : dict,
Additional kwargs for write_table function. See docstring for
`write_table` or `ParquetWriter` for more information.
Using `metadata_collector` in kwargs allows one to collect the
file metadata instances of dataset pieces. The file paths in the
ColumnChunkMetaData will be set relative to `root_path`.
NFTr re rj zGThe '{}' argument is not supported with the new dataset implementation.r c s j | j d S )N)rH rQ )Z written_file)r r r file_visitor[ s z&write_to_dataset..file_visitorpartition_filename_cbr )r )r rA r re r. rj rw c s g | ] } | qS r r )r, r8 )dfr r rD | s z$write_to_dataset..rm )Zaxisz.No data left to save outside partition columnsr c S s g | ]\}}d j ||dqS )z{colname}={value})Zcolnamer$ )rA )r, r{ r( r r r rD s )re safez.parquetr r\ r? r? )#r$ r rE rF r r1 rA ra Zmake_write_optionsr selectre r. Z
write_datasetr r! rv r+ Z to_pandasZdroprm namesrB rA groupbytupler_ zipr r Z from_pandasr rT r Z
set_file_path) r root_pathZpartition_colsrx r r5 r rB re rj r rw re Z
write_optionsr. Z part_schemar r Zdata_dfZ data_colsZ subschemar8 r ZsubgroupsubdirZsubtableoutfileZ
relative_pathr r- r )ry r r write_to_dataset s 1
r c K sL t || f|}|j |dk rHt|}x|D ]}|j| q,W |j| dS )aT
Write metadata-only Parquet file from schema. This can be used with
`write_to_dataset` to generate `_common_metadata` and `_metadata` sidecar
files.
Parameters
----------
schema : pyarrow.Schema
where : string or pyarrow.NativeFile
metadata_collector : list
where to collect metadata information.
**kwargs : dict,
Additional kwargs for ParquetWriter class. See docstring for
`ParquetWriter` for more information.
Examples
--------
Write a dataset and collect metadata information.
>>> metadata_collector = []
>>> write_to_dataset(
... table, root_path,
... metadata_collector=metadata_collector, **writer_kwargs)
Write the `_common_metadata` parquet file without row groups statistics.
>>> write_metadata(
... table.schema, root_path / '_common_metadata', **writer_kwargs)
Write the `_metadata` parquet file with row groups statistics.
>>> write_metadata(
... table.schema, root_path / '_metadata',
... metadata_collector=metadata_collector, **writer_kwargs)
N)r r r: Zappend_row_groupsZwrite_metadata_file)re r r r r rQ mr r r write_metadata s %
r c C s t | |djS )a#
Read FileMetadata from footer of a single Parquet file.
Parameters
----------
where : str (filepath) or file-like object
memory_map : bool, default False
Create memory map when the source is a file path.
Returns
-------
metadata : FileMetadata
)rZ )rM rQ )r rZ r r r r: s r: c C s t | |djj S )a#
Read effective Arrow schema from Parquet file metadata.
Parameters
----------
where : str (filepath) or file-like object
memory_map : bool, default False
Create memory map when the source is a file path.
Returns
-------
schema : pyarrow.Schema
)rZ )rM re r@ )r rZ r r r read_schema s r )r )T)N)r r\ N)NTNFFNNNr r FNTN)N)Nrm Tr TNNFNNNNFrm F)NNNN)N)F)F)V collectionsr
concurrentr functoolsr r r collections.abcr Znumpyr r% rerI urllib.parser r Zpyarrowr Z pyarrow.libr Zpyarrow._parquetr r r r r
r r r
Z
pyarrow.fsr r r r r r Z pyarrow.utilr r r r r r" r* r: rM rL rM compiler r r r r r ry r r r r r r r r r) r, rK rL r- r6 rS r1 Z_read_table_docstringrk rA r_ r rE r rv r r r: r r r r r s
$
66
ew
'Fhk
X
&
K
,
1