/usr/local/lib/python3.6/site-packages/fsspec/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/fsspec/__pycache__/parquet.cpython-36.pyc (11181B)
3
8%EgJL @ s d dl Z d dlZd dlZddlmZ ddlmZ dd d Zd d
dZd ddZ dd Z
dd Z dd Z G dd dZ
G dd dZdS ) N ) url_to_fs)merge_offset_rangesrbFauto @B@B c K s |dkr t | |pi dd }|dk r>t|dkr>|j| |dS t|}t| g|||||| |
| d }
|
rrtt|
n| }| jdi j }|j|f|d||
j |i |dd | S )
a
Return a file-like object for a single Parquet file.
The specified parquet `engine` will be used to parse the
footer metadata, and determine the required byte ranges
from the file. The target path will then be opened with
the "parts" (`KnownPartsOfAFile`) caching strategy.
Note that this method is intended for usage with remote
file systems, and is unlikely to improve parquet-read
performance on local file systems.
Parameters
----------
path: str
Target file path.
mode: str, optional
Mode option to be passed through to `fs.open`. Default is "rb".
metadata: Any, optional
Parquet metadata object. Object type must be supported
by the backend parquet engine. For now, only the "fastparquet"
engine supports an explicit `ParquetFile` metadata object.
If a metadata object is supplied, the remote footer metadata
will not need to be transferred into local memory.
fs: AbstractFileSystem, optional
Filesystem object to use for opening the file. If nothing is
specified, an `AbstractFileSystem` object will be inferred.
engine : str, default "auto"
Parquet engine to use for metadata parsing. Allowed options
include "fastparquet", "pyarrow", and "auto". The specified
engine must be installed in the current environment. If
"auto" is specified, and both engines are installed,
"fastparquet" will take precedence over "pyarrow".
columns: list, optional
List of all column names that may be read from the file.
row_groups : list, optional
List of all row-groups that may be read from the file. This
may be a list of row-group indices (integers), or it may be
a list of `RowGroup` metadata objects (if the "fastparquet"
engine is used).
storage_options : dict, optional
Used to generate an `AbstractFileSystem` object if `fs` was
not specified.
strict : bool, optional
Whether the resulting `KnownPartsOfAFile` cache should
fetch reads that go beyond a known byte-range boundary.
If `False` (the default), any read that ends outside a
known part will be zero padded. Note that using
`strict=True` may be useful for debugging.
max_gap : int, optional
Neighboring byte ranges will only be merged when their
inter-range gap is <= `max_gap`. Default is 64KB.
max_block : int, optional
Neighboring byte ranges will only be merged when the size of
the aggregated range is <= `max_block`. Default is 256MB.
footer_sample_size : int, optional
Number of bytes to read from the end of the path to look
for the footer metadata. If the sampled bytes do not contain
the footer, a second read request will be required, and
performance will suffer. Default is 1MB.
**kwargs :
Optional key-word arguments to pass to `fs.open`
N)storage_optionsr )mode)metadatacolumns
row_groupsenginemax_gap max_blockfooter_sample_size
cache_optionsparts)datastrict)r Z
cache_typer )
r lenopen _set_engine_get_parquet_byte_rangesnextiterpopcopyget)pathr fsr r
r r
r r r r r kwargsr fnoptions r% 8/usr/local/lib/python3.6/site-packages/fsspec/parquet.pyopen_parquet_file s2 Q
r' c C s t |trt|}|dk r0t|||||||dS |j| } i }
g } g } g }
d}|dkr|dkrxbt| D ]V\}}i |
|< xDtd| | |D ]0}| j| | j| |
jt|| | | qW qhW d}ng }g }x>t| D ]2\}}|j| | t d| | | }|j| qW |j
| ||}|j }d}xbt| D ]V\}}t j
|| d
d d}| | |d }||| k r6|||< t ||d }q6W |rtjd | d
x
= .)r footer footer_start)r r sorti)
isinstancestrr &_get_parquet_byte_ranges_from_metadatasizes enumeraterangeappendminmax
cat_rangesr int
from_byteswarningswarn_parquet_byte_rangesr r _transfer_ranges_add_header_magic) pathsr! r r
r r r r r Z
file_sizesresult
data_paths data_starts data_endsZadd_header_magicir bZ
footer_startsZ footer_endsZ sample_sizeZfooter_samplesZmissing_footer_startsZ large_footerZ footer_sizeZreal_footer_startblockZpath_data_startsZpath_data_endsr% r% r& r s
"r c C sb |j ||| d\}}} t||| ||dd\}}} dd tt|D }
t||
||| t|
|
S )zSimplified version of `_get_parquet_byte_ranges` for
the case that an engine-specific `metadata` object is
provided, and the remote footer metadata does not need to
be transferred before calculating the required byte ranges.
)r r F)r r r. c S s i | ]
}i |qS r% r% ).0r# r% r% r&
C s z:_get_parquet_byte_ranges_from_metadata..)r> r listsetr? r@ ) r r! r r
r r r rC rD rE rB r% r% r& r2 " s r2 c
C sD |||f}x4t || j| f D ] \}}}} | || ||f< q W d S )N)zipr9 )
r! blocksrA ZstartsZendsrangesr startstopr r% r% r& r? L s
"r? c C sj xdt t| j D ]P\}}d}x2| | j D ]"}|d dkr,|d dkr,d}P q,W |rd| | d< qW d S )NTr r r) Fs PAR1)r r) )r4 rK keys)r rF r Z add_magickr% r% r& r@ S s r@ c C s | dkrd}n2t | ts"tdn | d kr:t| dn| g}xB|D ]:}y |dkrZt S |dkrht S W qF tk
r~ Y qFX qFW td| dd S )
Nr fastparquetpyarrowzMFailed to set parquet engine! Please pass 'fastparquet', 'pyarrow', or 'auto'z) engine not supported by `fsspec.parquet`zLThe following parquet engines are not installed in your python environment: zS.Please install 'fastparquert' or 'pyarrow' to utilize the `fsspec.parquet` module.)rT rU )rT rU )r0 r1
ValueErrorFastparquetEngine
PyarrowEngine ImportError)Z
engine_strZ try_enginesr r% r% r& r _ s$
r c @ s& e Zd Zdd Zdd ZdddZdS ) rW c C s dd l }|| _d S )Nr )rT fp)selfrZ r% r% r& __init__ s zFastparquetEngine.__init__c C s
|j |S )N)Zrow_group_filename)r[ row_grouppfr% r% r& _row_group_filename s z%FastparquetEngine._row_group_filenameNc C sx |}g g g }}} |d kr.| j jtj|}|d kr:d nt|}
|
d k rxt|drxdd |jjdg D } |
t| O }
|rt|d t rd } n
|} |j
}xt |D ]\}
}| d ks|
| kr| j ||}x|j
D ]}|jjd }|
d ks||
kr|jj}|d kr |jj}|jj}|d ks&||k r|j| |j| | jt|| |pR|| qW qW |rp||| fS || fS )Npandas_metadatac S s g | ]}t |ts|qS r% )r0 dict)rI indr% r% r&
s z:FastparquetEngine._parquet_byte_ranges..
index_columnsr )rZ ParquetFileioBytesIOrL hasattrr` r r0 r: r r4 r_ r
Z meta_datapath_in_schemadictionary_page_offsetdata_page_offsettotal_compressed_sizer6 r7 )r[ r
r r r, r- r^ rC rD rE
column_setmd_indexZrow_group_indicesrr] r# columnname file_offset0 num_bytesr% r% r& r> sB
z&FastparquetEngine._parquet_byte_ranges)NNNN)__name__
__module__ __qualname__r\ r_ r> r% r% r% r& rW s rW c @ s& e Zd Zdd Zdd ZdddZdS ) rX c C s dd l j}|| _d S )Nr )Zpyarrow.parquetZparquetpq)r[ rw r% r% r& r\ s
zPyarrowEngine.__init__c C s t d S )N)NotImplementedError)r[ r] r r% r% r& r_ s z!PyarrowEngine._row_group_filenameNc C sf |d k rt dg g }}| jjtj|j}|d kr:d nt|} | d k r|jj }
|
jd k ofd|
jk} | rdd t j
|
jd j dj dg D } | t| O } xt
|jD ]}
|d ks|
|kr|j|
}xt
|jD ]}|j|}|j}|jdd }| d ks|| ks|| kr|j}|d kr*|j}|j}||k r|j| |jt|| | qW qW ||fS ) Nz.metadata input not supported for PyarrowEngines pandasc S s g | ]}t |ts|qS r% )r0 ra )rI rb r% r% r& rc s z6PyarrowEngine._parquet_byte_ranges..utf8rd r+ r )rV rw re rf rg r rL schemaZto_arrow_schemajsonloadsdecoder r5 Znum_row_groupsr] Z num_columnsrp ri splitrj rk rl r6 r7 )r[ r
r r r, r- rD rE Zmdrm rz Zhas_pandas_metadatarn ro r] crp rq Z
split_namerr rs r% r% r& r> sB
z"PyarrowEngine._parquet_byte_ranges)NNNN)rt ru rv r\ r_ r> r% r% r% r& rX s rX ) r NNNNNFr r r r )NNNr r r r )NNr r )rf r{ r< corer utilsr r' r r2 r? r@ r rW rX r% r% r% r& s@
u
# $X