/usr/local/lib64/python3.6/site-packages/pandas/io/__pycache__/
/usr/local/lib64/python3.6/site-packages/pandas/io/__pycache__
NameSizeModeActions
api.cpython-36.pyc10960644editdlrm
clipboards.cpython-36.pyc33920644editdlrm
common.cpython-36.pyc140640644editdlrm
date_converters.cpython-36.pyc19470644editdlrm
feather_format.cpython-36.pyc29360644editdlrm
gbq.cpython-36.pyc74590644editdlrm
html.cpython-36.pyc319770644editdlrm
orc.cpython-36.pyc18020644editdlrm
parquet.cpython-36.pyc95000644editdlrm
parsers.cpython-36.pyc882000644editdlrm
pickle.cpython-36.pyc56790644editdlrm
pytables.cpython-36.pyc1308050644editdlrm
spss.cpython-36.pyc13280644editdlrm
sql.cpython-36.pyc522830644editdlrm
stata.cpython-36.pyc975630644editdlrm
__init__.cpython-36.pyc1450644editdlrm
Edit: /usr/local/lib64/python3.6/site-packages/pandas/io/__pycache__/pytables.cpython-36.pyc (130805B)
3 :%Eg@sdZddlZddlmZmZddlZddlZddlZddlm Z m Z m Z m Z m Z mZmZmZddlZddlZddlmZmZddlmZmZddlmZddlmZmZm Z dd l!m"Z"dd l#m$Z$dd l%m&Z&dd l'm(Z(dd l)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4ddl5m6Z6ddl7m8Z8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@mAZAddlBmCZCmDZDmEZEddlFjGjHZIddlJmKZKmLZLddlMmNZNddlOmPZPddlQmRZRmSZSe rddlTmUZUmVZVmWZWdZXdZYddZZddZ[ddZ\eKZ]e^dd d!Z_Gd"d#d#e`ZaGd$d%d%e`ZbGd&d'd'ecZdd(ZeGd)d*d*ecZfd+ZgGd,d-d-ecZhd.Zid/Zjd0d0d1d1d2Zke8dgiZld3Zmd4Znejod58ejpd6d7emejqd8ejpd9denejrd0d1dgd8WdQRXdasd7atd:d;Zudeveeve e^e evewe evewe ee^e eve^ffe ewe eewe evfevevd? d@dAZxdeveve e^e e^e e^dCdDdEZydFdFewdGdHdIZzGdJdKdKZ{GdLdMdMZ|GdNdOdOZ}GdPdQdQe}Z~GdRdSdSe}ZGdTdUdUeZGdVdWdWeZGdXdYdYZGdZd[d[eZGd\d]d]eZGd^d_d_eZGd`dadaeZGdbdcdceZGdddedeeZGdfdgdgeZGdhdidieZGdjdkdkeZGdldmdmeZGdndodoeZGdpdqdqeZde8e^e:e8drdsdtZeeevefdudvdwZdeeje:fe eevefeweeje9fdxdydzZeve:eveve}d{d|d}Zeveveveeje:fd~ddZevdddZejevevejdddZejevevejdddZejevevevdddZevevevdddZevewdddZevevdddZevevdddZedddZGdddZdS)zY High level interface to PyTables for reading and writing pandas data structures to disk N)datetzinfo) TYPE_CHECKINGAnyDictListOptionalTupleTypeUnion)config get_option)libwriters) timezones) ArrayLike FrameOrSeriesLabel)import_optional_dependency) patch_pickle)PerformanceWarning)cache_readonly) ensure_objectis_categorical_dtypeis_complex_dtypeis_datetime64_dtypeis_datetime64tz_dtypeis_extension_array_dtype is_list_likeis_string_dtypeis_timedelta64_dtype)ABCExtensionArray)array_equivalent) DataFrame DatetimeIndexIndex Int64Index MultiIndex PeriodIndexSeriesTimedeltaIndexconcatisna) Categorical DatetimeArray PeriodArray) PyTablesExprmaybe_expression) ensure_index)stringify_path)adjoin pprint_thing)ColFileNodez0.15.2UTF-8cCst|tjr|jd}|S)z* if we have bytes, decode them to unicode zUTF-8) isinstancenpbytes_decode)sr?>/usr/local/lib64/python3.6/site-packages/pandas/io/pytables.py_ensure_decodedFs  rAcCs|dkr t}|S)N)_default_encoding)encodingr?r?r@_ensure_encodingMsrDcCst|trt|}|S)z Ensure that an index / column name is a str (python 3); otherwise they may be np.string dtype. Non-string dtypes are passed through unchanged. https://github.com/pandas-dev/pandas/issues/13492 )r:str)namer?r?r@ _ensure_strUs rG) scope_levelcCs|d}t|ttfr^g}xpsz_ensure_term..)rHN)r:listtuplefilterr1appendTermlen)whererHlevelZwlistwr?r?r@ _ensure_termds   rUc@s eZdZdS)PossibleDataLossErrorN)__name__ __module__ __qualname__r?r?r?r@rV{srVc@s eZdZdS)ClosedFileErrorN)rWrXrYr?r?r?r@rZsrZc@s eZdZdS)IncompatibilityWarningN)rWrXrYr?r?r?r@r[sr[z where criteria is being ignored as this version [%s] is too old (or not-defined), read the file in and write it out to a new file to upgrade (with the copy_to method) c@s eZdZdS)AttributeConflictWarningN)rWrXrYr?r?r?r@r\sr\zu the [%s] attribute of the existing index is [%s] which conflicts with the new [%s], resetting the attribute to None c@s eZdZdS)DuplicateWarningN)rWrXrYr?r?r?r@r]sr]z; duplicate entries in table, taking most recently appended z your performance may suffer as PyTables will pickle object types that it cannot map directly to c-types [inferred_type->%s,key->%s] [items->%s] fixedtable)fr^tr_z; : boolean drop ALL nan rows when appending to a table z~ : format default format writing format, if None, then put will default to 'fixed' and append will default to 'table' zio.hdfZ dropna_tableF)Z validatordefault_formatc Cs>tdkr:ddl}|ay|jjdkaWntk r8YnXtS)Nrstrict) _table_modtablesfileZ_FILE_OPEN_POLICY!_table_file_open_policy_is_strictAttributeError)rer?r?r@_tablessriaTrc) keyvaluemode complevelcomplibrOformatindex min_itemsizedropna data_columnserrorsrCc s|r$ f dd}n f dd}t|}t|trxt||||d}||WdQRXn||dS)z- store this object, close it if we opened it c s|j d S)N)rprqrrnan_reprsrtrurC)rO)store) rtrsrCrurprqrkrrrvrlr?r@rKszto_hdf..c s|jd S)N)rprqrrrvrtrurC)put)rw) rtrCrurprqrkrrrvrlr?r@rKs)rmrnroN)r3r:rEHDFStore) path_or_bufrkrlrmrnrorOrprqrrrvrsrtrurCr`rwr?) rtrsrCrurprqrkrrrvrlr@to_hdfs   r{r)rmrustartstop chunksizec Ks|dkrtd|d|dk r,t|dd}t|trN|jsDtd |} d } nvt|}t|tshtd yt j j |} Wnt tfk rd } YnX| st d |d t|f||d| } d} yz|dkr&| j}t|dkrtd|d}x*|ddD]}t||stdqW|j}| j||||||| | dStt tfk rt|tsy | jWntk rYnXYnXdS)a Read from the store, close it if we opened it. Retrieve pandas object stored in file, optionally based on where criteria. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- path_or_buf : str, path object, pandas.HDFStore or file-like object Any valid string path is acceptable. The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. A local file could be: ``file://localhost/path/to/table.h5``. If you want to pass in a path object, pandas accepts any ``os.PathLike``. Alternatively, pandas accepts an open :class:`pandas.HDFStore` object. By file-like object, we refer to objects with a ``read()`` method, such as a file handler (e.g. via builtin ``open`` function) or ``StringIO``. key : object, optional The group identifier in the store. Can be omitted if the HDF file contains a single pandas object. mode : {'r', 'r+', 'a'}, default 'r' Mode to use when opening the file. Ignored if path_or_buf is a :class:`pandas.HDFStore`. Default is 'r'. errors : str, default 'strict' Specifies how encoding and decoding errors are to be handled. See the errors argument for :func:`open` for a full list of options. where : list, optional A list of Term (or convertible) objects. start : int, optional Row number to start selection. stop : int, optional Row number to stop selection. columns : list, optional A list of columns names to return. iterator : bool, optional Return an iterator object. chunksize : int, optional Number of rows to include in an iteration when using an iterator. **kwargs Additional keyword arguments passed to HDFStore. Returns ------- item : object The selected object. Return type depends on the object stored. See Also -------- DataFrame.to_hdf : Write a HDF file from a DataFrame. HDFStore : Low-level access to HDF files. Examples -------- >>> df = pd.DataFrame([[1, 1.0, 'a']], columns=['x', 'y', 'z']) >>> df.to_hdf('./store.h5', 'data') >>> reread = pd.read_hdf('./store.h5') r|r+rjzmode zG is not allowed while performing a read. Allowed modes are r, r+ and a.NrI)rHz&The HDFStore must be open for reading.Fz5Support for generic buffers has not been implemented.zFile z does not exist)rmruTrz]Dataset(s) incompatible with Pandas data types, not table, or no datasets found in HDF5 file.z?key must be provided when HDF5 file contains multiple datasets.)rRr}r~columnsiteratorr auto_close)r|rrj) ValueErrorrUr:ryis_openIOErrorr3rENotImplementedErrorospathexists TypeErrorFileNotFoundErrorgroupsrQ_is_metadata_of _v_pathnameselectKeyErrorcloserh)rzrkrmrurRr}r~rrrkwargsrwrrrZcandidate_only_groupZgroup_to_checkr?r?r@read_hdfsdS          rr8)group parent_groupreturncCsJ|j|jkrdS|}x0|jdkrD|j}||kr<|jdkrZ%d|eeeeeed?d@dAZ&d}e dBdCdDZ'd~ee(eeee)ee*eeffeeeee dGdHdIZ+deddJdKZ,dee(eeee)ee*eeffee eeeedLdMdNZ-de*dOdPdQZ.deeeeedRdSdTZ/dUdVZ0ddXdYZ1eedZdd[d\Z2ee)ddd_d`Z3de eee dbdcddZ4eddedfZ5dgdhZ6eedidjdkZ7dee(eee)ddmdndoZ8dee(eeee)ee*eeffee dpdqdrZ9dZdsdtduZ:dS)ryaa Dict-like IO interface for storing pandas objects in PyTables. Either Fixed or Table format. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- path : str File path to HDF5 file. mode : {'a', 'w', 'r', 'r+'}, default 'a' ``'r'`` Read-only; no data can be modified. ``'w'`` Write; a new file is created (an existing file with the same name would be deleted). ``'a'`` Append; an existing file is opened for reading and writing, and if the file does not exist it is created. ``'r+'`` It is similar to ``'a'``, but the file must already exist. complevel : int, 0-9, default None Specifies a compression level for data. A value of 0 or None disables compression. complib : {'zlib', 'lzo', 'bzip2', 'blosc'}, default 'zlib' Specifies the compression library to be used. As of v0.20.2 these additional compressors for Blosc are supported (default if no compressor specified: 'blosc:blosclz'): {'blosc:blosclz', 'blosc:lz4', 'blosc:lz4hc', 'blosc:snappy', 'blosc:zlib', 'blosc:zstd'}. Specifying a compression library which is not available issues a ValueError. fletcher32 : bool, default False If applying compression use the fletcher32 checksum. **kwargs These parameters will be passed to the PyTables open_file method. Examples -------- >>> bar = pd.DataFrame(np.random.randn(10, 4)) >>> store = pd.HDFStore('test.h5') >>> store['foo'] = bar # write to HDF5 >>> bar = store['foo'] # retrieve >>> store.close() **Create or load HDF5 file in-memory** When passing the `driver` option to the PyTables open_file method through **kwargs, the HDF5 file is loaded or created in-memory and will only be written when closed: >>> bar = pd.DataFrame(np.random.randn(10, 4)) >>> store = pd.HDFStore('test.h5', driver='H5FD_CORE') >>> store['foo'] = bar >>> store.close() # only now, data is written to disk r7rjNF)rmrn fletcher32cKsd|krtdtd}|dk r@||jjkr@td|jjd|dkrX|dk rX|jj}t||_|dkrnd}||_d|_|r|nd|_ ||_ ||_ d|_ |j fd|i|dS) Nrpz-format is not a defined argument for HDFStorerezcomplib only supports z compression.rjrrm)rrfiltersZ all_complibsZdefault_complibr3_path_mode_handle _complevel_complib _fletcher32_filtersopen)selfrrmrnrorrrer?r?r@__init__ s$  zHDFStore.__init__cCs|jS)N)r)rr?r?r@ __fspath__+szHDFStore.__fspath__cCs|j|jjS)z return the root node )_check_if_openrroot)rr?r?r@r.sz HDFStore.rootcCs|jS)N)r)rr?r?r@filename4szHDFStore.filename)rkcCs |j|S)N)get)rrkr?r?r@ __getitem__8szHDFStore.__getitem__cCs|j||dS)N)rx)rrkrlr?r?r@ __setitem__;szHDFStore.__setitem__cCs |j|S)N)remove)rrkr?r?r@ __delitem__>szHDFStore.__delitem__)rFcCsDy |j|Sttfk r"YnXtdt|jd|ddS)z& allow attribute access to get stores 'z' object has no attribute 'N)rrrZrhtyperW)rrFr?r?r@ __getattr__As  zHDFStore.__getattr__)rkrcCs8|j|}|dk r4|j}||ks0|dd|kr4dSdS)zx check for existence of this key can match the exact pathname or the pathnm w/o the leading '/' NrITF)get_noder)rrknoderFr?r?r@ __contains__Ks  zHDFStore.__contains__)rcCs t|jS)N)rQr)rr?r?r@__len__WszHDFStore.__len__cCst|j}t|d|dS)Nz File path:  )r5rr)rpstrr?r?r@__repr__Zs zHDFStore.__repr__cCs|S)Nr?)rr?r?r@ __enter__^szHDFStore.__enter__cCs |jdS)N)r)rexc_type exc_value tracebackr?r?r@__exit__aszHDFStore.__exit__pandas)includercCs^|dkrdd|jDS|dkrJ|jdk s0tdd|jjddd DStd |d dS) a! Return a list of keys corresponding to objects stored in HDFStore. Parameters ---------- include : str, default 'pandas' When kind equals 'pandas' return pandas objects When kind equals 'native' return native HDF5 Table objects .. versionadded:: 1.1.0 Returns ------- list List of ABSOLUTE path-names (e.g. have the leading '/'). Raises ------ raises ValueError if kind has an illegal value rcSsg|] }|jqSr?)r).0nr?r?r@ {sz!HDFStore.keys..ZnativeNcSsg|] }|jqSr?)r)rrr?r?r@rs/Table) classnamez8`include` should be either 'pandas' or 'native' but is 'r)rrAssertionErrorZ walk_nodesr)rrr?r?r@keysdsz HDFStore.keyscCs t|jS)N)iterr)rr?r?r@__iter__szHDFStore.__iter__ccs"x|jD]}|j|fVq WdS)z' iterate on key->group N)rr)rgr?r?r@itemsszHDFStore.items)rmcKst}|j|krR|jdkr$|dkr$n(|dkrL|jrLtd|jd|jd||_|jr`|j|jr|jdkrtj|j|j|j d |_ y|j |j|jf||_ Wnt k r}z|j}td|jd|d}|WYd d }~XnNtk r}z0|jdkrdt|krt t||WYd d }~XnXd S)a9 Open the file in the specified mode Parameters ---------- mode : {'a', 'w', 'r', 'r+'}, default 'a' See HDFStore docstring or tables.open_file for info about modes **kwargs These parameters will be passed to the PyTables open_file method. rjrTr|r+zRe-opening the file [z ] with mode [z] will delete the current file!r)rzcan not be writtenzOpening z in read-only modeNZFILE_OPEN_POLICYz PyTables [zY] no longer supports opening multiple files even in read-only mode on this HDF5 version [z]. You can accept this and not open the same file multiple times at once, upgrade the HDF5 version, or downgrade to PyTables 3.0.0 which allows files to be opened multiple times at once zUnable to open/create file)rjrT)r|r)rT)rirrrVrrrFiltersrrr open_filerrrEprintrZget_hdf5_version __version__ Exception)rrmrreerrZ hdf_versionr?r?r@rs>    z HDFStore.opencCs|jdk r|jjd|_dS)z0 Close the PyTables file handle N)rr)rr?r?r@rs  zHDFStore.closecCs|jdkrdSt|jjS)zF return a boolean indicating whether the file is open NF)rboolZisopen)rr?r?r@rs zHDFStore.is_open)fsyncc CsF|jdk rB|jj|rBytj|jjWntk r@YnXdS)a Force all buffered modifications to be written to disk. Parameters ---------- fsync : bool (default False) call ``os.fsync()`` on the file handle to force writing to disk. Notes ----- Without ``fsync=True``, flushing may not guarantee that the OS writes to disk. With fsync, the operation will block until the OS claims the file has been written; however, other caching layers may still interfere. N)rflushrrfilenoOSError)rrr?r?r@rs  zHDFStore.flushc Cs>t.|j|}|dkr*td|d|j|SQRXdS)z Retrieve pandas object stored in file. Parameters ---------- key : str Returns ------- object Same type as object stored in file. NzNo object named z in the file)rrr _read_group)rrkrr?r?r@rs  z HDFStore.get)rkrc st|j|} | dkr"td|dt|dd}|j| jfdd} t|| |j|||||d } | jS) a Retrieve pandas object stored in file, optionally based on where criteria. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- key : str Object being retrieved from file. where : list, default None List of Term (or convertible) objects, optional. start : int, default None Row number to start selection. stop : int, default None Row number to stop selection. columns : list, default None A list of columns that if not None, will limit the return columns. iterator : bool, default False Returns an iterator. chunksize : int, default None Number or rows to include in iteration, return an iterator. auto_close : bool, default False Should automatically close the store when finished. Returns ------- object Retrieved object from file. NzNo object named z in the filerI)rHcsj|||dS)N)r}r~rRr)read)_start_stop_where)rr>r?r@funcRszHDFStore.select..func)rRnrowsr}r~rrr)rrrU_create_storer infer_axes TableIteratorr get_result) rrkrRr}r~rrrrrritr?)rr>r@rs&.   zHDFStore.select)rkr}r~cCs8t|dd}|j|}t|ts(td|j|||dS)a return the selection as an Index .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- key : str where : list of Term (or convertible) objects, optional start : integer (defaults to None), row number to start selection stop : integer (defaults to None), row number to stop selection rI)rHz&can only read_coordinates with a table)rRr}r~)rU get_storerr:rrread_coordinates)rrkrRr}r~tblr?r?r@select_as_coordinateses    zHDFStore.select_as_coordinates)rkcolumnr}r~cCs,|j|}t|tstd|j|||dS)a~ return a single column from the table. This is generally only useful to select an indexable .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- key : str column : str The column of interest. start : int or None, default None stop : int or None, default None Raises ------ raises KeyError if the column is not found (or key is not a valid store) raises ValueError if the column can not be extracted individually (it is part of a data block) z!can only read_column with a table)rr}r~)rr:rr read_column)rrkrr}r~rr?r?r@ select_columns#  zHDFStore.select_column)rc  st|dd}t|ttfr.t|dkr.|d}t|trRj||||||| dSt|ttfshtdt|sxtd|dkr|d}fdd |Dj |} d} xzt j | |fgt |D]^\} } | dkrt d | d | jstd | jd | dkr| j} q| j| krtdqWdd D}tdd|Ddfdd}t| ||| ||||| d }|jddS)a Retrieve pandas objects from multiple tables. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- keys : a list of the tables selector : the table to apply the where criteria (defaults to keys[0] if not supplied) columns : the columns I want back start : integer (defaults to None), row number to start selection stop : integer (defaults to None), row number to stop selection iterator : boolean, return an iterator, default False chunksize : nrows to include in iteration, return an iterator auto_close : bool, default False Should automatically close the store when finished. Raises ------ raises KeyError if keys or selector is not found or keys is empty raises TypeError if keys is not a list or tuple raises ValueError if the tables are not ALL THE SAME DIMENSIONS rI)rHr)rkrRrr}r~rrrzkeys must be a list/tuplez keys must have a non-zero lengthNcsg|]}j|qSr?)r)rk)rr?r@rsz/HDFStore.select_as_multiple..zInvalid table []zobject [z>] is not a table, and cannot be used in all select as multiplez,all tables must have exactly the same nrows!cSsg|]}t|tr|qSr?)r:r)rrJr?r?r@rscSsh|]}|jddqS)r)non_index_axes)rrar?r?r@ sz.HDFStore.select_as_multiple..cs*fddD}t|ddjS)Ncsg|]}|jdqS))rRrr}r~)r)rra)rrrrr?r@rsz=HDFStore.select_as_multiple..func..F)axisverify_integrity)r+ _consolidate)rrrobjs)rrtbls)rrrr@r sz)HDFStore.select_as_multiple..func)rRrr}r~rrrT) coordinates)rUr:rLrMrQrErrrr itertoolschainzipris_tablepathnamerrr)rrrRselectorrr}r~rrrr>rrarZ_tblsrrr?)rrrrr@select_as_multiples^+   "    zHDFStore.select_as_multipleTrc)rkrlrnrrrtru track_timescCsF|dkrtdpd}|j|}|j||||||||| | | | | d dS)a Store object in HDFStore. Parameters ---------- key : str value : {Series, DataFrame} format : 'fixed(f)|table(t)', default is 'fixed' Format to use when storing object in HDFStore. Value can be one of: ``'fixed'`` Fixed format. Fast writing/reading. Not-appendable, nor searchable. ``'table'`` Table format. Write as a PyTables Table structure which may perform worse but allow more flexible operations like searching / selecting subsets of the data. append : bool, default False This will force Table format, append the input data to the existing. data_columns : list, default None List of columns to create as data columns, or True to use all columns. See `here `__. encoding : str, default None Provide an encoding for strings. dropna : bool, default False, do not write an ALL nan row to The store settable by the option 'io.hdf.dropna_table'. track_times : bool, default True Parameter is propagated to 'create_table' method of 'PyTables'. If set to False it enables to have the same h5 files (same hashes) independent on creation time. .. versionadded:: 1.1.0 Nzio.hdf.default_formatr^) rprqrOrornrrrvrtrCrur)r _validate_format_write_to_group)rrkrlrprqrOrornrrrvrtrCrurr?r?r@rx's"2  z HDFStore.putcCst|dd}y|j|}Wn~tk r0Ynjtk rDYnVtk r}z:|dk rftd||j|}|dk r|jdddSWYdd}~XnXtj |||r|j jddn|j std|j |||dSdS) a= Remove pandas object partially by specifying the where condition Parameters ---------- key : string Node to remove or delete rows from where : list of Term (or convertible) objects, optional start : integer (defaults to None), row number to start selection stop : integer (defaults to None), row number to stop selection Returns ------- number of rows removed (or None if not a Table) Raises ------ raises KeyError if key is not a valid store rI)rHNz5trying to remove a node with a non-None where clause!T) recursivez7can only remove with where on objects written as tables)rRr}r~) rUrrrrrrZ _f_removecomall_nonerrdelete)rrkrRr}r~r>rrr?r?r@rls,   zHDFStore.remove)rkrlrnrrrsrtrucCsl| dk rtd|dkr td}|dkr4tdp2d}|j|}|j||||||||| | | | ||||ddS)a6 Append to Table in file. Node must already exist and be Table format. Parameters ---------- key : str value : {Series, DataFrame} format : 'table' is the default Format to use when storing object in HDFStore. Value can be one of: ``'table'`` Table format. Write as a PyTables Table structure which may perform worse but allow more flexible operations like searching / selecting subsets of the data. append : bool, default True Append the input data to the existing. data_columns : list of columns, or True, default None List of columns to create as indexed data columns for on-disk queries, or True to use all columns. By default only the axes of the object are indexed. See `here `__. min_itemsize : dict of columns that specify minimum str sizes nan_rep : str to use as str nan representation chunksize : size to chunk the writing expectedrows : expected TOTAL row size of this table encoding : default None, provide an encoding for str dropna : bool, default False Do not write an ALL nan row to the store settable by the option 'io.hdf.dropna_table'. Notes ----- Does *not* check if data being appended overlaps with existing data in the table, so be careful Nz>columns is not a supported keyword in append, try data_columnszio.hdf.dropna_tablezio.hdf.default_formatr_)rpaxesrqrOrornrrrvr expectedrowsrsrtrCru)rr rr)rrkrlrprrqrOrornrrrrvrrrsrtrCrur?r?r@rOs28  zHDFStore.append)dc s|dk rtdt|ts"td||kr2tdtttjttt d}d} g} x<|j D]0\} dkr| dk rtd| } qj| j qjW| dk rڈj |} | j t| } t| j| } | j| || <|dkr||}|r2fdd|jD}t|}x|D]}|j|}qWj||jd d}xt|j D]h\} | |kr^|nd}j|d }|dk rfd d |j Dnd}|j| |f||d |qHWdS)a Append to multiple tables Parameters ---------- d : a dict of table_name to table_columns, None is acceptable as the values of one node (this will get all the remaining columns) value : a pandas object selector : a string that designates the indexable table; all of its columns will be designed as data_columns, unless data_columns is passed, in which case these are used data_columns : list of columns to create as data columns, or True to use all columns dropna : if evaluates to True, drop rows from all tables if any single row in each table has all NaN. Default False. Notes ----- axes parameter is currently not accepted Nztaxes is currently not accepted as a parameter to append_to_multiple; you can create the tables independently insteadzQappend_to_multiple must have a dictionary specified as the way to split the valuez=append_to_multiple requires a selector that is in passed dictrzEsz.HDFStore.append_to_multiple..rr)rcsi|]\}}|kr||qSr?r?)rrkrl)vr?r@ Usz/HDFStore.append_to_multiple..)rtrr)rr:dictrrLsetrangendim _AXES_MAPrrextendr differencer%sorted get_indexertakevaluesnext intersectionlocpopreindexrO)rrrlrrtrrsrrZ remain_keyZ remain_valuesrorderedZorddZidxsZ valid_indexrqrrdcvalfilteredr?)r rlr@append_to_multiplesN &     zHDFStore.append_to_multiple)rkoptlevelkindcCsBt|j|}|dkrdSt|ts.td|j|||ddS)a Create a pytables index on the table. Parameters ---------- key : str columns : None, bool, or listlike[str] Indicate which columns to create an index on. * False : Do not create any indexes. * True : Create indexes on all columns. * None : Create indexes on all columns. * listlike : Create indexes on the given columns. optlevel : int or None, default None Optimization level, if None, pytables defaults to 6. kind : str or None, default None Kind of index, if None, pytables defaults to "medium". Raises ------ TypeError: raises if the node is not a table Nz1cannot create table index on a Fixed format store)rr#r$)rirr:rr create_index)rrkrr#r$r>r?r?r@create_table_index[s  zHDFStore.create_table_indexcCs"t|jdd|jjDS)z Return a list of all the top-level nodes. Each node returned is not a pandas storage object. Returns ------- list List of objects. cSsRg|]J}t|tjj rt|jddsJt|ddsJt|tjjr|jdkr|qS) pandas_typeNr_) r:rdlinkLinkgetattr_v_attrsr_rr)rrr?r?r@rs  z#HDFStore.groups..)rirr walk_groups)rr?r?r@rs zHDFStore.groupsrccst|jx|jj|D]}t|jdddk r4qg}g}xP|jjD]B}t|jdd}|dkr~t|t j j r|j |j qH|j |j qHW|jjd||fVqWdS)au Walk the pytables group hierarchy for pandas objects. This generator will yield the group path, subgroups and pandas object names for each group. Any non-pandas PyTables objects that are not a group will be ignored. The `where` group itself is listed first (preorder), then each of its child groups (following an alphanumerical order) is also traversed, following the same procedure. .. versionadded:: 0.24.0 Parameters ---------- where : str, default "/" Group where to start walking. Yields ------ path : str Full path to a group (without trailing '/'). groups : list Names (strings) of the groups contained in `path`. leaves : list Names (strings) of the pandas objects contained in `path`. r'Nr)rirrr,r*r+Z _v_childrenrr:rdrGrouprOrrrstrip)rrRrrZleaveschildr'r?r?r@walksz HDFStore.walkr8c Cs~|j|jdsd|}|jdk s(ttdk s4ty|jj|j|}Wntjjk r`dSXt |tj sztt ||S)z; return the node with the key or None if it does not exist rN) r startswithrrrdrr exceptionsZNoSuchNodeErrorr:r8r)rrkrr?r?r@rs  zHDFStore.get_node GenericFixedrcCs8|j|}|dkr"td|d|j|}|j|S)z> return the storer object for a key, raise if not in the file NzNo object named z in the file)rrrr)rrkrr>r?r?r@rs   zHDFStore.get_storerrT) propindexesrnrc  Cst|||||d} |dkr&t|j}t|ttfs:|g}x|D]} |j| } | dk r@| | krl|rl| j| |j| } t| trd} |rdd| j D} | j | | | t | dd| j dq@| j | | | j dq@W| S) a Copy the existing store to a new file, updating in place. Parameters ---------- propindexes: bool, default True Restore indexes in copied file. keys : list of keys to include in the copy (defaults to all) overwrite : overwrite (remove and replace) existing nodes in the new store (default is True) mode, complib, complevel, fletcher32 same as in HDFStore.__init__ Returns ------- open file handle of the new store )rmrornrNFcSsg|]}|jr|jqSr?) is_indexedrF)rrjr?r?r@rsz!HDFStore.copy..rt)rqrtrC)rC)ryrLrr:rMrrrrrrOr*rCrx)rrfrmr4rrornr overwriteZ new_storerr>datarqr?r?r@copys2        z HDFStore.copyc Cst|j}t|d|d}|jrt|j}t|rg}g}x|D]}y<|j|}|dk r|jt|j pn||jt|pdWqHt k rYqHt k r}z*|j|t|} |jd| dWYdd}~XqHXqHW|t d||7}n|d7}n|d 7}|S) zg Print detailed information on the store. Returns ------- str z File path: rNzinvalid_HDFStore nodez[invalid_HDFStore node: r EmptyzFile is CLOSED) r5rrrrrrQrrOrrrr4) rroutputZlkeysrrrr>ZdetailZdstrr?r?r@info"s.     ( z HDFStore.infocCs|jst|jddS)Nz file is not open!)rrZr)rr?r?r@rLszHDFStore._check_if_open)rprcCsJyt|j}Wn4tk rD}ztd|d|WYdd}~XnX|S)z validate / deprecate formats z#invalid HDFStore format specified [rN) _FORMAT_MAPlowerrr)rrprr?r?r@rPs $zHDFStore._validate_formatUTF-8)rlrCrurc"s$dk r tttf r tdfdd}ttjdd}ttjdd}|dkr̈dkrttdk svt tddsttj j rd}d }qtd n(td td i} | t }dkr|d 7}d|kr,t td} y | |} Wn.tk r} z|d| WYdd} ~ XnX| |||dS|dkrƈdk r|dkrtdd} | dk r| jdkrrd}n| jdkrd}nB|dkrtdd} | dk r| jdkrd}n| jdkrd}ttttttd}y ||} Wn.tk r} z|d| WYdd} ~ XnX| |||dS)z$ return a suitable class to operate Nz(value must be None, Series, or DataFramec s$td|ddtdS)Nz(cannot properly create the storer for: [z ] [group->z,value->z ,format->)rr)ra)rprrlr?r@errorhsz&HDFStore._create_storer..errorr' table_typer_ frame_table generic_tablezKcannot create a storer if the object is not existing nor a value are passedseriesframeZ_table)rDrE _STORER_MAP)rCru series_tablerqrIappendable_seriesappendable_multiseriesappendable_frameappendable_multiframe)rCrHrIrJrKworm _TABLE_MAP)r:r)r#rrAr*r+rirdrr_rr SeriesFixed FrameFixedrnlevels GenericTableAppendableSeriesTableAppendableMultiSeriesTableAppendableFrameTableAppendableMultiFrameTable WORMTable)rrrprlrCrur@ptttZ _TYPE_MAPrFclsrrqrMr?)rprrlr@rZsj                     zHDFStore._create_storer)rkrlrnrrrurcCs~|j|}|jdk st|dk r:| r:|jj|ddd}t|ddrV|dksR|rVdS|dkr|jd}d}xX|D]P}t|sqr|}|jds|d7}||7}|j|}|dkr|jj||}|}qrW|j |||||d}|r|j p|j o|dko|j rt d|j s |j n|j |j r8|r8t d |j|||||| | | | | |||d t|trz|rz|j|d dS) NT)remptyr_r)rCrur^zCan only append to Tablesz0Compression not supported on Fixed format stores) objrrOrornrrrrrrsrvrtr)r)rrr remove_noder*splitrQendswithZ create_grouprr is_existsrset_object_infowriter:rr%)rrkrlrprrqrOrornrrrrrrsrvrtrCrurrpathsrpnew_pathr>r?r?r@rs\      zHDFStore._write_to_group)rcCs|j|}|j|jS)N)rrr)rrr>r?r?r@rs zHDFStore._read_group)rjNNF)r)rj)F)NNNNFNF)NNN)NN)NNNNNFNF) NTFNNNNNNrcT)NNN)NNTTNNNNNNNNNNrc)NNF)NNN)r)r3r)rTTNNNFT)NNr?rc)r3r)NTFNNNNNNFNNNrcT);rWrXrY__doc__rrrErintrrrrrpropertyrrrrrrrrrrrrrrr iteritemsrrrrrrrrrrr rrxrrOr"r&rr0rrr8r<rrrrrr?r?r?r@rys @    "JE&t<7=@H]# / 3*  [2Eryc@sfeZdZUdZeeee d dee de eee dddZ d d Z d d Zde d ddZdS)raa Define the iteration interface on a table Parameters ---------- store : HDFStore s : the referred storer func : the function to execute the query where : the where of the query nrows : the rows to iterate on start : the passed start value (default is None) stop : the passed stop value (default is None) iterator : bool, default False Whether to use the default iterator. chunksize : the passed chunking value (default is 100000) auto_close : bool, default False Whether to automatically close the store at the end of iteration. r3rNF)rwr>rrrc Cs||_||_||_||_|jjrN|dkr,d}|dkr8d}|dkrD|}t||}||_||_||_d|_ |sr| dk r| dkr~d} t | |_ nd|_ | |_ dS)Nri) rwr>rrRrminrr}r~rrfrr) rrwr>rrRrr}r~rrrr?r?r@r-s,    zTableIterator.__init__ccsj|j}xV||jkr\t||j|j}|jdd|j||}|}|dkst| rTq|VqW|jdS)N)r}r~rirrrrQr)rrr~rlr?r?r@rWs  zTableIterator.__iter__cCs|jr|jjdS)N)rrwr)rr?r?r@rgszTableIterator.close)rcCs|jdk r4t|jtstd|jj|jd|_|S|rft|jtsLtd|jj|j|j|j d}n|j}|j |j|j |}|j |S)Nz0can only use an iterator or chunksize on a table)rRz$can only read_coordinates on a table)rRr}r~) rr:r>rrrrRrr}r~rr)rrrRresultsr?r?r@rks   zTableIterator.get_result)r3r)NNFNF)r3r)F)rWrXrYrerrfrryrwr r>rrrrrr?r?r?r@rs   rc @steZdZUdZdZdZdddgZee dBee eddd Z e e d d d Ze ed d dZe dddZed ddZeedddZed ddZe ed ddZejeedddZddZe d d!Ze d"d#Ze d$d%Ze d&d'Zd(d)ZdCd*d+Z d,d-Z!d.ed/d0d1Z"dDd2d3Z#ed4d5d6Z$d7d8Z%d9d:Z&d;d<Z'd.d=d>d?Z(d.d=d@dAZ)dS)EIndexCola an index column description class Parameters ---------- axis : axis which I reference values : the ndarray like converted values kind : a string description of this type typ : the pytables type pos : the position in the pytables Tfreqtz index_nameN)rFcnamecCst|tstd||_||_||_||_|p0||_||_||_ ||_ | |_ | |_ | |_ | |_| |_||_|dk r||j|t|jtstt|jtstdS)Nz`name` must be a str.)r:rErrr$typrFrorposrlrmrnrr_rmetadataset_posr)rrFrr$rprorrqrlrmrnrr_rrrr?r?r@rs(   zIndexCol.__init__)rcCs|jjS)N)rpitemsize)rr?r?r@rtszIndexCol.itemsizecCs |jdS)N_kind)rF)rr?r?r@ kind_attrszIndexCol.kind_attr)rqcCs$||_|dk r |jdk r ||j_dS)z. set the position of this column in the Table N)rqrpZ_v_pos)rrqr?r?r@rsszIndexCol.set_poscCsFttt|j|j|j|j|jf}djddt dddddg|DS) N,css |]\}}|d|VqdS)z->Nr?)rrkrlr?r?r@r sz$IndexCol.__repr__..rFrorrqr$) rMmapr5rFrorrqr$joinr)rtempr?r?r@rs  zIndexCol.__repr__)otherrcstfdddDS)z compare 2 col items c3s&|]}t|dt|dkVqdS)N)r*)rrj)r{rr?r@r sz"IndexCol.__eq__..rFrorrq)rFrorrq)r)rr{r?)r{rr@__eq__s zIndexCol.__eq__cCs |j| S)N)r|)rr{r?r?r@__ne__szIndexCol.__ne__cCs"t|jdsdSt|jj|jjS)z' return whether I am an indexed column r F)hasattrr_r*r ror5)rr?r?r@r5s zIndexCol.is_indexed)rrCruc Cst|tjstt||jjdk r.||j}t|j }t ||||}t }t|j |d<|j dk rrt|j |d<yt|f|}Wn0tk rd|krd|d<t|f|}YnXt||j}||fS)zV Convert the data from this selection to the appropriate pandas type. NrFrl)r:r;ndarrayrrdtypefieldsrorAr$_maybe_convertrrnrlr%r_set_tzrm)rrrvrCruval_kindrZ new_pd_indexr?r?r@converts"     zIndexCol.convertcCs|jS)z return the values)r)rr?r?r@ take_dataszIndexCol.take_datacCs|jjS)N)r_r+)rr?r?r@attrsszIndexCol.attrscCs|jjS)N)r_ description)rr?r?r@rszIndexCol.descriptioncCst|j|jdS)z# return my current col description N)r*rro)rr?r?r@colsz IndexCol.colcCs|jS)z return my cython values )r)rr?r?r@cvalues$szIndexCol.cvaluescCs t|jS)N)rr)rr?r?r@r)szIndexCol.__iter__cCsPt|jdkrLt|tr$|j|j}|dk rL|jj|krLtj ||j d|_dS)z maybe set a string col itemsize: min_itemsize can be an integer or a dict with this columns name with an integer size stringN)rtrq) rAr$r:rrrFrprtri StringColrq)rrrr?r?r@maybe_set_size,s   zIndexCol.maybe_set_sizecCsdS)Nr?)rr?r?r@validate_names:szIndexCol.validate_namesAppendableTable)handlerrOcCs:|j|_|j|j||j||j||jdS)N)r_ validate_col validate_attrvalidate_metadatawrite_metadataset_attr)rrrOr?r?r@validate_and_set=s    zIndexCol.validate_and_setcCs^t|jdkrZ|j}|dk rZ|dkr*|j}|j|krTtd|d|jd|jd|jSdS)z< validate this column: return the compared against itemsize rNz#Trying to store a string with len [z] in [z)] column but this column has a limit of [zC]! Consider using min_itemsize to preset the sizes on these columns)rAr$rrtrro)rrtcr?r?r@rEs zIndexCol.validate_col)rOcCsB|r>t|j|jd}|dk r>||jkr>td|d|jddS)Nzincompatible kind in col [z - r)r*rrvr$r)rrOZ existing_kindr?r?r@rXs zIndexCol.validate_attrc Csx|jD]}t||d}|j|ji}|j|}||kr|dk r||kr|d krt|||f}tj|tddd||<t ||dqt d|jd|d|d |d q|dk s|dk r|||<qWdS) z set/update the info for this indexable with the key/value if there is a conflict raise/warn as needed Nrlrn) stacklevelzinvalid info for [z] for [z], existing_value [z] conflicts with new value [r)rlrn) _info_fieldsr* setdefaultrFrattribute_conflict_docwarningswarnr\setattrr)rr<rkrlidxZexisting_valuewsr?r?r@ update_infoas   $zIndexCol.update_infocCs$|j|j}|dk r |jj|dS)z# set my state from the passed info N)rrF__dict__update)rr<rr?r?r@set_infos zIndexCol.set_infocCst|j|j|jdS)z set the kind for this column N)rrrvr$)rr?r?r@rszIndexCol.set_attr)rcCsD|jdkr@|j}|j|j}|dk r@|dk r@t|| r@tddS)z< validate that kind=category does not change the categories categoryNzEcannot append a categorical with different categories to the existing)rrr read_metadataror"r)rrZ new_metadataZ cur_metadatar?r?r@rs   zIndexCol.validate_metadatacCs|jdk r|j|j|jdS)z set the meta data N)rrrro)rrr?r?r@rs zIndexCol.write_metadata) NNNNNNNNNNNNN)N)N)*rWrXrYreis_an_indexableis_data_indexablerrErFrorrrgrfrtrvrsrrrr|r}r5r;rrrrrrrrrrrrrrrrrrr?r?r?r@rks\            rkc@s>eZdZdZeedddZeje e dddZ dd Z d S) GenericIndexColz< an index which is not represented in the data of the table )rcCsdS)NFr?)rr?r?r@r5szGenericIndexCol.is_indexed)rrCrucCs2t|tjstt|ttjt|}||fS)z Convert the data from this selection to the appropriate pandas type. Parameters ---------- values : np.ndarray nan_rep : str encoding : str errors : str )r:r;rrrr&arangerQ)rrrvrCrur?r?r@rs zGenericIndexCol.convertcCsdS)Nr?)rr?r?r@rszGenericIndexCol.set_attrN) rWrXrYrergrr5r;rrErrr?r?r?r@rs rc s0eZdZdZdZdZddgZd2edfdd Ze ed d d Z e ed d d Z ed ddZ e edddZedddZddZeeddddZeddZeeedddd Zeeddd!d"Zed#d$Zed%d&Ze d'd(Ze d)d*Zd+d,Zejeed-d.d/Z d0d1Z!Z"S)3DataCola3 a data holding column, by definition this is not indexable Parameters ---------- data : the actual data cname : the column name in the table to hold the data (typically values) meta : a string description of the metadata metadata : the actual metadata FrmrN)rFc s2tj||||||||| | | d | |_| |_dS)N) rFrr$rprqrormrr_rrr)superrrr7)rrFrr$rprorqrmrr_rrrrr7) __class__r?r@rszDataCol.__init__)rcCs |jdS)N_dtype)rF)rr?r?r@ dtype_attrszDataCol.dtype_attrcCs |jdS)N_meta)rF)rr?r?r@ meta_attrszDataCol.meta_attrcCsFttt|j|j|j|j|jf}djddt dddddg|DS) Nrwcss |]\}}|d|VqdS)z->Nr?)rrkrlr?r?r@r sz#DataCol.__repr__..rFrorr$shape) rMrxr5rFrorr$rryr)rrzr?r?r@rs zDataCol.__repr__)r{rcstfdddDS)z compare 2 col items c3s&|]}t|dt|dkVqdS)N)r*)rrj)r{rr?r@r  sz!DataCol.__eq__..rFrorrq)rFrorrq)r)rr{r?)r{rr@r| s zDataCol.__eq__)r7cCs@|dk s t|jdkstt|\}}||_||_t||_dS)N)rr_get_data_and_dtype_namer7_dtype_to_kindr$)rr7 dtype_namer?r?r@set_data s   zDataCol.set_datacCs|jS)z return the data )r7)rr?r?r@r szDataCol.take_datar6)rrcCs|j}|j}|j}|jdkr&d|jf}t|trJ|j}|j||jj d}ntt |sZt |rf|j |}nXt |rz|j|}nDt|rtj||dd}n&t|r|j||}n|j||j d}|S)zW Get an appropriately typed and shaped pytables.Col object for values. rI)r$r)rtr)rrtrrsizer:r-codes get_atom_datarFrrget_atom_datetime64r get_atom_timedelta64rriZ ComplexColrget_atom_string)rYrrrtrratomr?r?r@ _get_atom s$     zDataCol._get_atomcCstj||ddS)Nr)rtr)rir)rYrrtr?r?r@r6 szDataCol.get_atom_string)r$rcCsR|jdr$|dd}d|d}n"|jdr4d}n|j}|d}tt|S)z2 return the PyTables column class for this column uintNZUIntr6periodInt64Col)r1 capitalizer*ri)rYr$Zk4Zcol_nameZkcapr?r?r@get_atom_coltype: s    zDataCol.get_atom_coltypecCs|j|d|ddS)N)r$r)r)r)rYrr$r?r?r@rI szDataCol.get_atom_datacCstj|ddS)Nr)r)rir)rYrr?r?r@rM szDataCol.get_atom_datetime64cCstj|ddS)Nr)r)rir)rYrr?r?r@rQ szDataCol.get_atom_timedelta64cCst|jddS)Nr)r*r7)rr?r?r@rU sz DataCol.shapecCs|jS)z return my cython values )r7)rr?r?r@rY szDataCol.cvaluescCs`|r\t|j|jd}|dk r2|t|jkr2tdt|j|jd}|dk r\||jkr\tddS)zAvalidate that we have the same order as the existing & same dtypeNz4appended items do not match existing items in table!z@appended items dtype do not match existing items dtype in table!)r*rrvrLrrrr)rrOZexisting_fieldsZexisting_dtyper?r?r@r^ szDataCol.validate_attr)rrCrucCst|tjstt||jjdk r.||j}|jdk s.cSsg|]}tj|qSr?)r fromtimestamp)rr r?r?r@r srrI) categoriesrF)r8Or)rvrCru)#r:r;rrrrrrorprrr$rArrrrrmrasarrayobjectrravelr%Zfloat64r,anyastyperfZcumsum_valuesr-Z from_codesr_unconvert_string_arrayr)rrrvrCru convertedrr$rrrrrmrrrmaskr?r?r@rk s^             zDataCol.convertcCsHt|j|j|jt|j|j|j|jdk s2tt|j|j|jdS)z set the data for this column N) rrrvrrrrrr)rr?r?r@r szDataCol.set_attr) NNNNNNNNNNNN)#rWrXrYrerrrrErrgrrrrrr|rrr classmethodrrr rrrrrrrr;rrr __classcell__r?r?)rr@rsL         grc@sTeZdZdZdZddZeddZeeddd d Z ed d Z ed dZ dS)DataIndexableColz- represent a data column that can be indexed TcCst|jjstddS)Nz-cannot have non-object label DataIndexableCol)r%r is_objectr)rr?r?r@r szDataIndexableCol.validate_namescCstj|dS)N)rt)rir)rYrrtr?r?r@r sz DataIndexableCol.get_atom_stringr6)r$rcCs|j|dS)N)r$)r)rYrr$r?r?r@r szDataIndexableCol.get_atom_datacCs tjS)N)rir)rYrr?r?r@r sz$DataIndexableCol.get_atom_datetime64cCs tjS)N)rir)rYrr?r?r@r sz%DataIndexableCol.get_atom_timedelta64N) rWrXrYrerrrrrErrrr?r?r?r@r s  rc@seZdZdZdS)GenericDataIndexableColz* represent a generic pytables data column N)rWrXrYrer?r?r?r@r src@seZdZUdZedZeeee e f e  eededZd=edeeddd Zeed d d Zeee e e fd d dZeddZed ddZddZddZeddZeddZeddZ eddZ!ee d dd Z"eed d!d"Z#ed#d$Z$d%d&Z%d'd(Z&ed)d*Z'eed d+d,Z(ed-d.Z)d/d0Z*d>d2d3Z+d4d5Z,d?e-e e-e d6d7d8Z.d9d:Z/d@e-e e-e d6d;d<Z0d1S)AFixedz represent an object in my store facilitate read/write of various types of objects this is an abstract base class Parameters ---------- parent : HDFStore group : Node The group node where the table resides. r^r8FUTF-8rc)rrrCrucCsZt|tstt|tdk s"tt|tjs:tt|||_||_t||_ ||_ dS)N) r:ryrrrdr8rrrDrCru)rrrrCrur?r?r@r s  zFixed.__init__)rcCs*|jddko(|jddko(|jddkS)NrrI )version)rr?r?r@is_old_version! szFixed.is_old_versionc Csbtt|jjdd}y0tdd|jdD}t|dkrB|d}Wntk r\d }YnX|S) z compute and set our version pandas_versionNcss|]}t|VqdS)N)rf)rrJr?r?r@r * sz Fixed.version...rr)r)rrr)rAr*rr+rMr]rQrh)rrr?r?r@r% s   z Fixed.versioncCstt|jjddS)Nr')rAr*rr+)rr?r?r@r'1 szFixed.pandas_typecCs^|j|j}|dk rXt|ttfrDdjdd|D}d|d}|jdd|d S|jS) z* return a pretty representation of myself Nrwcss|]}t|VqdS)N)r5)rrJr?r?r@r ; sz!Fixed.__repr__..[rz12.12z (shape->))rrr:rLrMryr')rr>Zjshaper?r?r@r5 s zFixed.__repr__cCst|j|j_tt|j_dS)z set my pandas type & version N)rE pandas_kindrr'_versionr)rr?r?r@r`@ szFixed.set_object_infocCstj|}|S)N)r8)rZnew_selfr?r?r@r8E s z Fixed.copycCs|jS)N)r)rr?r?r@rI sz Fixed.shapecCs|jjS)N)rr)rr?r?r@rM szFixed.pathnamecCs|jjS)N)rr)rr?r?r@rQ sz Fixed._handlecCs|jjS)N)rr)rr?r?r@rU szFixed._filterscCs|jjS)N)rr)rr?r?r@rY szFixed._complevelcCs|jjS)N)rr)rr?r?r@r] szFixed._fletcher32cCs|jjS)N)rr+)rr?r?r@ra sz Fixed.attrscCsdS)z set our object attributes Nr?)rr?r?r@ set_attrse szFixed.set_attrscCsdS)z get our object attributes Nr?)rr?r?r@ get_attrsi szFixed.get_attrscCs|jS)z return my storable )r)rr?r?r@storablem szFixed.storablecCsdS)NFr?)rr?r?r@r_r szFixed.is_existscCst|jddS)Nr)r*r)rr?r?r@rv sz Fixed.nrowscCs|dkr dSdS)z' validate against an existing storable NTr?)rr{r?r?r@validatez szFixed.validateNcCsdS)z- are we trying to operate on an old version? Tr?)rrRr?r?r@validate_version szFixed.validate_versioncCs|j}|dkrdS|jdS)zr infer the axes of my storer return a boolean indicating if we have a valid storer or not NFT)rr)rr>r?r?r@r s zFixed.infer_axes)r}r~cCs tddS)Nz>cannot read on an abstract storer: subclasses should implement)r)rrRrr}r~r?r?r@r sz Fixed.readcKs tddS)Nz?cannot write on an abstract storer: subclasses should implement)r)rrr?r?r@ra sz Fixed.writecCs0tj|||r$|jj|jdddStddS)zs support fully deleting the node in its entirety (only) - where specification must be None T)rNz#cannot delete on an abstract storer)rrrr\rr)rrRr}r~r?r?r@r sz Fixed.delete)rrc)N)NNNN)NNN)1rWrXrYrerEr format_typer r r#r)obj_typerfrrCryrrrurrrgrrr rr'rr`r8rrrrrrrrrrr_rrrrrrrarr?r?r?r@r sX             rc@sJeZdZUdZedediZddejDZgZ e e  e dddZ d d Z d d Zd dZeedddZddZddZddZd/e eeeedddZd0e eeeeedddZe eddd Ze edd!d"Zd1e eeeeedd#d$Zd2d%eeeeed&d'd(Ze e d)d*d+Z!d3e e eed,d-d.Z"dS)4r3z a generified fixed version datetimercCsi|]\}}||qSr?r?)rrr r?r?r@r  szGenericFixed.)rcCs|jj|dS)N)_index_type_mapr)rrYr?r?r@_class_to_alias szGenericFixed._class_to_aliascCst|tr|S|jj|tS)N)r:r_reverse_index_maprr%)raliasr?r?r@_alias_to_class s zGenericFixed._alias_to_classcCs0|tkrddd}|S|tkr,ddd}|S|S)NcSs:tj|j|d}tj|dd}|dk r6|jdj|}|S)N)rl)rFUTC)r. _simple_newrr$ tz_localize tz_convert)rrlrmZdtaresultr?r?r@r` s z*GenericFixed._get_index_factory..fcSstj||d}tj|ddS)N)rl)rF)r/rr()rrlrmZparrr?r?r@r` s)NN)NN)r$r()rklassr`r?r?r@_get_index_factory s  zGenericFixed._get_index_factorycCs$|dk rtd|dk r tddS)zE raise if any keywords are passed which are not-None Nzqcannot pass a column specification when reading a Fixed format store. this store must be selected in its entiretyzucannot pass a where specification when reading from a Fixed format store. this store must be selected in its entirety)r)rrrRr?r?r@ validate_read s zGenericFixed.validate_readcCsdS)NTr?)rr?r?r@r_ szGenericFixed.is_existscCs|j|j_|j|j_dS)z set our object attributes N)rCrru)rr?r?r@r s zGenericFixed.set_attrsc CsVtt|jdd|_tt|jdd|_x(|jD]}t||tt|j|dq0WdS)z retrieve our attributes rCNrurc)rDr*rrCrAru attributesr)rrr?r?r@r s zGenericFixed.get_attrscKs |jdS)N)r)rr[rr?r?r@ra szGenericFixed.writeN)rkr}r~c Csddl}t|j|}|j}t|dd}t||jrD|d||}nztt|dd} t|dd} | dk rxtj| | d}n |||}| dkrt|d d} t || d d }n| d krtj |d d}|r|j S|SdS)z4 read an array for the specified node (off of group rN transposedF value_typer)rrrmT)rrzm8[ns]) rer*rr+r:ZVLArrayrAr;rZrrT) rrkr}r~rerrrretrrrmr?r?r@ read_array s&      zGenericFixed.read_array)rkr}r~rcCshtt|j|d}|dkr.|j|||dS|dkrVt|j|}|j|||d}|Std|dS)N_varietymulti)r}r~regularzunrecognized index variety: )rAr*rread_multi_indexrread_index_noder)rrkr}r~Zvarietyrrqr?r?r@ read_index s zGenericFixed.read_index)rkrqcCst|tr,t|j|dd|j||nt|j|ddtd||j|j}|j||j t |j |}|j |j _ |j|j _t|ttfr|jt||j _t|tttfr|j|j _t|tr|jdk rt|j|j _dS)Nrrrrq)r:r'rrwrite_multi_index_convert_indexrCru write_arrayrr*rr$r+rFr$r(rr index_classr*rlrm_get_tz)rrkrqrrr?r?r@ write_index) s     zGenericFixed.write_indexc Cst|j|d|jxtt|j|j|jD]\}\}}}t|rLt d|d|}t |||j |j }|j ||jt|j|} |j| j_|| j_t| j|d|||d|} |j | |q.WdS)N_nlevelsz=Saving a MultiIndex with an extension dtype is not supported._level_name_label)rrrP enumeraterlevelsrnamesrrr rCrur rr*rr$r+rF) rrkrqilev level_codesrF level_keyZ conv_levelr label_keyr?r?r@r @ s"  zGenericFixed.write_multi_indexcCst|j|d}g}g}g}xxt|D]l}|d|} t|j| } |j| ||d} |j| |j| j|d|} |j| ||d} |j| q(Wt|||ddS)Nrr)r}r~rT)rrrr) r*rrrrrOrFrr')rrkr}r~rPrrrrrrrrrr?r?r@rY s   zGenericFixed.read_multi_indexr8)rr}r~rc Cs>|||}d|jkr>tj|jjdkr>tj|jj|jjd}t|jj}d}d|jkrlt|jj }t|}|j tt |jdd}|j |}i} d|jkr|jd| d<d|jkrt |jdtr|jdjd | d<n|jd| d<|d kr|t|||j|jd fd ti| } n|t|||j|jd f| } || _ | S) Nrr)rrFr rrlrmzutf-8r)rCrur)r+r;prodrrZrrAr$rGrFrr*rr:bytesr=_unconvert_indexrCrur) rrr}r~r7r$rFr factoryrrqr?r?r@rp s<         zGenericFixed.read_index_node)rkrlcCsJtjd|j}|jj|j||t|j|}t|j|j _ |j |j _ dS)z write a 0-len array rIN)rI) r;rZrr create_arrayrr*rErr+rr)rrkrlZarrrr?r?r@write_array_empty s  zGenericFixed.write_array_empty)rkrlrc CsJt|tjtfstt|||jkr6|jj|j||j dk}d}t |j rVt d|snt |drn|j}d}d}|jdk rytjj|j }Wntk rYnX|dk r|s|jj|j|||j|jd}||dd<n |j||nJ|j jtjkr`tj|dd}|rn*|dkrnt|||f} tj| td d |jj|j|tj} | j |n|rt|j||nt!|j r|jj"|j||j#d d t$|j|j%_&nt'|j r|jj"|j||j(t$|j|} t)|j*| j%_*d | j%_&nHt+|j r$|jj"|j||j#d d t$|j|j%_&n|jj"|j|||t$|j|j%_,dS)NrFz]Cannot store a category dtype in a HDF5 dataset that uses format="fixed". Use format="table".rT)r)skipnar)ri8rr)-r:r;rr!rrrrr\rrrrr~rrriZAtomZ from_dtyperZ create_carrayrr!Zobject_r infer_dtypeperformance_docrrrZcreate_vlarray ObjectAtomrOrr viewr*r+rrasi8rrmr r) rrkrlrZ empty_arrayrrca inferred_typerZvlarrrr?r?r@r  sb            zGenericFixed.write_array)NN)NN)NN)NN)N)#rWrXrYrer$r(rrrrrrErrrrrgrr_rrrarrfrr%r rr'r rrrr!r r?r?r?r@r3 s0   % 2 r3csReZdZUdZdgZeeddZd e e e e dddZ fd d Z Z S) rNrDrFc Cs,yt|jjfSttfk r&dSXdS)N)rQrrrrh)rr?r?r@r szSeriesFixed.shapeN)r}r~cCs<|j|||jd||d}|jd||d}t|||jdS)Nrq)r}r~r)rqrF)rr rr)rF)rrRrr}r~rqrr?r?r@r s zSeriesFixed.readc s:tj|f||jd|j|jd|j|j|j_dS)Nrqr)rrarrqr rrFr)rr[r)rr?r@ra szSeriesFixed.write)NNNN)rWrXrYrrrrFrgrrrfrrarr?r?)rr@rN s  rNcsPeZdZUddgZeeddZd eeeedddZ fd d Z Z S) BlockManagerFixedrnblocksc Csy|j}d}xFt|jD]8}t|jd|d}t|dd}|dk r||d7}qW|jj}t|dd}|dk rt|d|d}ng}|j||Stk rdSXdS)Nrblock_itemsrrI) rrr-r*rZ block0_valuesrLrOrh)rrrrrrr?r?r@r& s"   zBlockManagerFixed.shapeN)r}r~cCs |j|||jjd}g}xJt|jD]<}||kr>||fnd \}} |jd||| d} |j| q*W|d} g} xht|jD]Z}|jd|d} |jd|d|| d}| | j | }t |j ||dd}| j|qWt | dkr t | dd }|j| d d }|St |d|ddS) Nrr)r}r~r.r/rrI)rrq)rF)rr8)NN)rrZ_get_block_manager_axisrrr rOr-rrr#rrQr+r)rrRrr}r~Z select_axisrrrraxrdfs blk_itemsrdfoutr?r?r@rA s(  zBlockManagerFixed.readc stj|f||j}|js&|j}|j|j_x@t|jD]2\}}|dkr\|j r\t d|j d||qZ3d?d@Z4dedAdBZ5dCdDZ6e7dEdFZ8dfe9edGdHdIZ:dge9ee9ee e e;e;fdJdKdLZdPdQZ?dhe@edRdSdTZAeBdUdVZCdidWdXdYdZZDe9eee9eeeefd[d\d]ZEdje9ee9ed^d_d`ZFdkee9ee9edadbdcZGZHS)lra represent a table: facilitate read/write of various types of tables Attrs in Table Node ------------------- These are attributes that are store in the main table node, they are necessary to recreate these tables when read back in. index_axes : a list of tuples of the (original indexing axis and index column) non_index_axes: a list of tuples of the (original index axis and columns on a non-indexing axis) values_axes : a list of the columns which comprise the data of this table data_columns : a list of the columns that we are allowing indexing (these become single columns in values_axes), or True to force all columns nan_rep : the string to use for nan representations for string objects levels : the names of levels metadata : the names of the metadata columns Z wide_tabler_rITNrcr8)rrruc sRtj||||d|pg|_|p$g|_|p.g|_|p8g|_| pDt|_| |_dS)N)rCru) rr index_axesr values_axesrtrr<rv) rrrrCrur9rr:rtr<rv)rr?r@r s      zTable.__init__)rcCs|jjddS)N_r)rAr])rr?r?r@table_type_short szTable.table_type_shortc Cs|jt|jrdj|jnd}d|d}d}|jrZdjdd|jD}d|d}djd d|jD}|jd |d |jd |j d |j d|d|d S)z* return a pretty representation of myself rwrz,dc->[rrcss|]}t|VqdS)N)rE)rrJr?r?r@r  sz!Table.__repr__..rcss|] }|jVqdS)N)rF)rrjr?r?r@r  sz12.12z (typ->z,nrows->z,ncols->z ,indexers->[r) rrQrtryrrr9r'r<rncols)rZjdcrverZjverZ jindex_axesr?r?r@r s  zTable.__repr__)rcCs"x|jD]}||jkr|SqWdS)z return the axis for c N)rrF)rrrjr?r?r@r s  zTable.__getitem__c Cs|dkr dS|j|jkr2td|jd|jdxd D]}t||d}t||d}||kr8x@t|D]4\}}||}||krftd|d |d |dqfWtd|d |d |dq8WdS) z$ validate against an existing table Nz'incompatible table_type with existing [z - rr9rr:zinvalid combination of [z] on appending data [z] vs current table [)r9rr:)rArr*rrr)rr{rsvovrsaxZoaxr?r?r@r s     zTable.validatecCs t|jtS)z@the levels attribute is 1 or a list in the case of a multi-index)r:rrL)rr?r?r@is_multi_index szTable.is_multi_indexcCsTddt|jjD}y |j|fStk rN}ztd|WYdd}~XnXdS)ze validate that we can store the multi-index; reset and return the new object cSs&g|]\}}|dk r|nd|qS)NZlevel_r?)rrlr?r?r@r sz-Table.validate_multiindex..zBduplicate names/columns in the multi-index when storing as a tableN)rrqrZ reset_indexr)rr[rrr?r?r@validate_multiindex s zTable.validate_multiindexcCstjdd|jDS)z/ based on our axes, compute the expected nrows cSsg|]}|jjdqS)r)rr)rrr?r?r@r sz(Table.nrows_expected..)r;rr9)rr?r?r@nrows_expected szTable.nrows_expectedcCs d|jkS)z has this table been created r_)r)rr?r?r@r_ szTable.is_existscCst|jddS)Nr_)r*r)rr?r?r@r szTable.storablecCs|jS)z. return the table group (this is my storable) )r)rr?r?r@r_ sz Table.tablecCs|jjS)N)r_r)rr?r?r@r sz Table.dtypecCs|jjS)N)r_r)rr?r?r@r" szTable.descriptioncCstj|j|jS)N)rrr9r:)rr?r?r@r& sz Table.axescCstdd|jDS)z0 the number of total columns in the values axes css|]}t|jVqdS)N)rQr)rrjr?r?r@r - szTable.ncols..)sumr:)rr?r?r@r=* sz Table.ncolscCsdS)NFr?)rr?r?r@ is_transposed/ szTable.is_transposedcCs(ttjdd|jDdd|jDS)z@return a tuple of my permutated axes, non_indexable at the frontcSsg|]}t|dqS)r)rf)rrjr?r?r@r8 sz*Table.data_orientation..cSsg|]}t|jqSr?)rfr)rrjr?r?r@r9 s)rMrrrr9)rr?r?r@data_orientation3 szTable.data_orientationcsRdddddjD}fddjD}fddjD}t|||S)z> return a dict of the kinds allowable columns for this object rqr)rrIcSsg|]}|j|fqSr?)ro)rrjr?r?r@rC sz$Table.queryables..csg|]\}}|dfqS)Nr?)rrr) axis_namesr?r@rD scs&g|]}|jtjkr|j|fqSr?)rFrrtro)rr )rr?r@rF s)r9rr:r)rZd1Zd2Zd3r?)rIrr@ queryables= s  zTable.queryablescCsdd|jDS)z return a list of my index cols cSsg|]}|j|jfqSr?)rro)rrr?r?r@rP sz$Table.index_cols..)r9)rr?r?r@ index_colsM szTable.index_colscCsdd|jDS)z! return a list of my values cols cSsg|] }|jqSr?)ro)rrr?r?r@rT sz%Table.values_cols..)r:)rr?r?r@ values_colsR szTable.values_cols)rkrcCs|jj}|d|dS)z+ return the metadata pathname for this key z/meta/z/meta)rr)rrkrr?r?r@_get_metadata_pathV szTable._get_metadata_path)rkrcCs0t|}|jj|j||d|j|j|jddS)z Write out a metadata array to the key as a fixed-format Series. Parameters ---------- key : str values : ndarray r_)rprCrurvN)r)rrxrMrCrurv)rrkrr?r?r@r[ s zTable.write_metadata)rkcCs0tt|jdd|ddk r,|jj|j|SdS)z) return the meta data array for this key rN)r*rrrrM)rrkr?r?r@rn szTable.read_metadatacCspt|j|j_|j|j_|j|j_|j|j_|j|j_|j|j_|j|j_|j |j_ |j |j_ |j |j_ dS)z! set our table type & indexables N) rErArrKrLrrtrvrCrurr<)rr?r?r@rt s        zTable.set_attrscCst|jddpg|_t|jddp$g|_t|jddp:t|_t|jdd|_tt|jdd|_t t|jdd|_ t|jd dpg|_ d d |j D|_ d d |j D|_dS) z retrieve our attributes rNrtr<rvrCrurcrcSsg|]}|jr|qSr?)r)rrjr?r?r@r sz#Table.get_attrs..cSsg|]}|js|qSr?)r)rrjr?r?r@r s)r*rrrtrr<rvrDrCrArur indexablesr9r:)rr?r?r@r szTable.get_attrscCs\|dk rX|jddkrX|jddkrX|jddkrXtdjdd|jD}tj|tdS) z- are we trying to operate on an old version? NrrIrrrcSsg|] }t|qSr?)rE)rrJr?r?r@r sz*Table.validate_version..)rincompatibility_docryrrr[)rrRrr?r?r@r s*zTable.validate_versioncCs^|dkr dSt|tsdS|j}x6|jD]*\}}|dkr>q,||kr,td|dq,WdS)z validate the min_itemsize doesn't contain items that are not in the axes this needs data_columns to be defined Nrzmin_itemsize has the key [z%] which is not an axis or data_column)r:rrJrr)rrrqrr r?r?r@validate_min_itemsize s zTable.validate_min_itemsizec sg}jjjxztjjD]j\}\}}t|}j|}|dk rLdnd}|d}t|d} t|||| |j||d} |j| q Wt j t |fdd|j fddtjj D|S) z1 create/cache the indexables if they don't exist Nrru)rFrrqr$rpr_rrrc st|tstt}|krt}t|}t|j}t|dd}t|dd}t|}j |}t|dd} |||||||j | ||d } | S)Nrurr) rFrorr$rqrpr_rrrr) r:rErrrr*_maybe_adjust_namerrrr_) rrrradj_namerrr$mdrr[)base_posrdescr table_attrsr?r@r` s.   zTable.indexables..fcsg|]\}}||qSr?r?)rrr)r`r?r@r sz$Table.indexables..)rr_rrrKr*rrkrOrrtrQrrL) r _indexablesrrrFrrTrrvr$ index_colr?)rUrrVr`rrWr@rN s0     # zTable.indexables)r$c CsX|js dS|dkrdS|dks(|dkr8dd|jD}t|ttfsL|g}t}|dk rb||d<|dk rr||d<|j}x|D]}t|j|d}|dk r"|j r|j }|j } |j } |dk r| |kr|j n| |d<|dk r| |kr|j n| |d<|j sP|jjdrtd |jf|q~||jd d kr~td |d |d|dq~WdS)aZ Create a pytables index on the specified columns. Parameters ---------- columns : None, bool, or listlike[str] Indicate which columns to create an index on. * False : Do not create any indexes. * True : Create indexes on all columns. * None : Create indexes on all columns. * listlike : Create indexes on the given columns. optlevel : int or None, default None Optimization level, if None, pytables defaults to 6. kind : str or None, default None Kind of index, if None, pytables defaults to "medium". Raises ------ TypeError if trying to create an index on a complex-type column. Notes ----- Cannot index Time64Col or ComplexCol. Pytables must be >= 3.0. NFTcSsg|]}|jr|jqSr?)rro)rrjr?r?r@rsz&Table.create_index..r#r$complexzColumns containing complex values can be stored but cannot be indexed when using table format. Either use fixed format, set index=False, or do not include the columns containing complex values to data_columns when initializing the table.rrIzcolumn z/ is not a data_column. In order to read column z: you must reload the dataframe into HDFStore and include z with the data_columns argument.)rrr:rMrLrr_r*r r5rqr#r$Z remove_indexrr1rr%rrh) rrr#r$kwr_rr rqZ cur_optlevelZcur_kindr?r?r@r% sF    zTable.create_index)r}r~rc Cs^t||||d}|j}g}x<|jD]2}|j|j|j||j|j|jd}|j |q$W|S)a Create the axes sniffed from the table. Parameters ---------- where : ??? start : int or None, default None stop : int or None, default None Returns ------- List[Tuple[index_values, column_values]] )rRr}r~)rvrCru) Selectionrrrr<rrvrCrurO) rrRr}r~ selectionrrjrjresr?r?r@ _read_axesHs   zTable._read_axes)rcCs|S)z return the data for this obj r?)rYr[rr?r?r@ get_objectjszTable.get_objectcst|s gS|d\}|jj|t}|jddkrN|rNtd|d||dkr`t}n |dkrlg}t|trt|t|}|jfdd |j Dfd d |DS) zd take the input data_columns and min_itemize and create a data columns spec rrr'z"cannot use a multi-index on axis [z] with data_columns TNcs g|]}|dkr|kr|qS)rr?)rr)existing_data_columnsr?r@rsz/Table.validate_data_columns..csg|]}|kr|qSr?r?)rr) axis_labelsr?r@rs) rQr<rrrrLr:rrr)rrtrrrrr<r?)rbrar@validate_data_columnsos$    zTable.validate_data_columns)r[rc0stts,|jj}td|dtddkr:dgfddD|jrzd}d d|jDt|j }|j }nd }|j } |j d kst t|j d krtd g} |dkrd}fddd Dd} j| } t| } |r|j|j|j|| ||.| |d }/t6|drr|j?|/_?|/j@||r|r|/jA||/S)!a0 Create and return the axes. Parameters ---------- axes: list or None The names or numbers of the axes to create. obj : DataFrame The object to create axes on. validate: bool, default True Whether to validate the obj against an existing object already written. nan_rep : A value to use for string column nan_rep. data_columns : List[str], True, or None, default None Specify the columns that we want to create to allow indexing on. * True : Use all available columns. * None : Use no columns. * List[str] : Use the specified columns. min_itemsize: Dict[str, int] or None, default None The min itemsize for a column in bytes. z/cannot properly create the storer for: [group->z,value->rNrcsg|]}j|qSr?)_get_axis_number)rrj)r[r?r@rsz&Table._create_axes..TcSsg|] }|jqSr?)r)rrjr?r?r@rsFrrIz.get_blk_items..r?)rfr6r?)rfr@ get_blk_items sz)Table._create_axes..get_blk_itemsz-cannot have non-object label DataIndexableColzIncompatible appended table [z]with existing table [Z values_block_) existing_colrrrvrCrurmr)r8) rFrorrprqr$rmrrrrrr7cSsg|]}|jr|jqSr?)rrF)rrr?r?r@rgs) rrrCrur9rr:rtr<rvr)rrI)Br:r#rrrrrr9rLrtrvr<rrrQrrrr"r;arrayrrrrWrOZ_get_axis_namer rCrurrsrr _reindex_axisrcr`r_get_blocks_and_itemsr:rrrrrE IndexErrorr_maybe_convert_for_string_atomrRrrrrrFr~rrmrrrrrrrrQr)0rrr[rrvrtrrr table_existsZnew_infonew_non_index_axesrrjZ append_axisindexerZ exist_axisr< axis_nameZ new_indexZnew_index_axesjrgr block_objr6r2Zvaxesrbb_itemsrrFrhrnew_namedata_convertedrSrpr$rmrrrrr7rrZdcsZ new_tabler?)rr[r@ _create_axess             "              zTable._create_axescCsddd}|jj}||j|}t|r|d\}} t| jt|} |j| |dj} t| j}|| |}x<|D]4} |j| g|dj} |j| j|j|| | jqlW|r\ddt||D} g}g}x|D]}t |j }y&| j |\}}|j ||j |Wqt tfk rN}z,djdd |D}td |d |WYdd}~XqXqW|}|}||fS) Ncsfdd|DS)Ncsg|]}jj|jqSr?)rrr7)rr8)rfr?r@rszFTable._get_blocks_and_items..get_blk_items..r?)rfr6r?)rfr@rgsz2Table._get_blocks_and_items..get_blk_itemsr)rcSs"i|]\}}||ft|jqSr?)rMtolist)rrtrur?r?r@r sz/Table._get_blocks_and_items..rwcss|]}t|VqdS)N)r5)ritemr?r?r@r sz.Table._get_blocks_and_items..z+cannot match existing table structure for [z] on appending data)r5r6rQr%rrrLrrrMrrrOrlrryr)rsrnror:rtrgr6r2rrbZ new_labelsrfrZby_itemsZ new_blocksZ new_blk_itemsZearrtrurZjitemsr?r?r@rks@          zTable._get_blocks_and_itemsr\)r]c s|dk rt|}|dk rRjrRtjts.tx"jD]}||kr6|jd|q6Wx jD]\}}t|||qZW|jdk rx2|jj D]$\}}fdd} | ||qWS)z process axes filters NrcsxjD]}j|}j|}|dk s,t||krdjrJ|jtj}||}j|d|S||krt t |j }t |}t t rd|}||}j|d|SqWtd|ddS)N)rrIzcannot find the field [z] for filtering!)Z _AXIS_ORDERSrd _get_axisrrBunionr%rrr2r*rr:r#r)fieldfiltrqZ axis_numberZ axis_valuesZtakersr)r[oprr?r@process_filters"       z*Table.process_axes..process_filter) rLrBr:rrinsertrrjrNrp) rr[r]rrrlabelsr}r~rr?)r[rrr@ process_axess  !zTable.process_axes)rnrrrcCs|dkrt|jd}td|d}dd|jD|d<|rl|dkrJ|jpHd}tj|||p\|jd }||d <n|jdk r|j|d <|S) z< create the description of the table from the axes & values Ni'r_)rFrcSsi|]}|j|jqSr?)rpro)rrjr?r?r@r sz,Table.create_description..r )rnrorr) maxrErrrrirrr)rrornrrrrr?r?r@create_descriptions       zTable.create_description)r}r~c Cs|j||jsdSt||||d}|j}|jdk rxR|jjD]D\}}}|j||j|jdd} ||| j ||j|j }qDWt |S)zf select coordinates (row numbers) from a table; return the coordinates object F)rRr}r~NrI)r}r~) rrr\ select_coordsrNrprririlocrr%) rrRr}r~r]Zcoordsr}rr~r7r?r?r@r s  "zTable.read_coordinates)rr}r~cCs|j|jsdS|dk r$tdx|jD]v}||jkr,|jsPtd|dt|jj |}|j |j |j ||||j |j|jd}tt|d|j|dSq,Wtd|d dS) zj return a single column from the table, generally only indexables are interesting FNz4read_column does not currently accept a where clausezcolumn [z=] can not be extracted individually; it is not data indexable)rvrCrurI)rFz] not found in the table)rrrrrFrrr*r_r rr<rrvrCrur)rrmr)rrrRr}r~rjrZ col_valuesr?r?r@r$s&      zTable.read_column)NrcNNNNNN)N)NNN)NN)TNNN)N)NNN)NNN)IrWrXrYrerrrErArrrrkr9r rfrrrr:rtrrrr<ryrrgr<rrrrrBrDrEr_rr_rrrr=rGrHrJrKrLrMr;rrrrrrrQrrNrr%rr_rr`rcr#rx staticmethodrkrrrrrr?r?)rr@r s           JU +d 1=rc@s6eZdZdZdZd eeeedddZddZdS) rVz a write-once read-many table: this format DOES NOT ALLOW appending to a table. writing is a one-time operation the data are stored in a format that allows for searching the data on disk rLN)r}r~cCs tddS)z[ read the indices and the indexing array, calculate offset rows and return z!WORMTable needs to implement readN)r)rrRrr}r~r?r?r@rZs zWORMTable.readcKs tddS)z write in a format that we can search later on (but cannot append to): write out the indices and the values using _write_array (e.g. a CArray) create an indexing table so that we can search z"WORMTable needs to implement writeN)r)rrr?r?r@rafszWORMTable.write)NNNN) rWrXrYrerArrfrrar?r?r?r@rVQsrVc @sveZdZdZdZdddZdeeedd d Z e j e e j ee j e e j d d d Z deeeedddZdS)rz* support the new appendable table formats Z appendableNFTcCs| r|jr|jj|jd|j||||| | d}x|jD] }|jq:W|js|j|||| d}|j| |d<|jj |jf||j |j _ x|jD]}|j ||qW|j || ddS)Nr_)rr[rrrrvrt)rornrrr)rs)r_rr\rrxrrrrZ create_tabler<rr write_data)rr[rrOrornrrrrrrsrvrtrr_rjoptionsr?r?r@rats0     zAppendableTable.write)rrscs|jj}|j}g}|rXx@|jD]6}t|jjdd}t|tj r|j |j dddqWt |r|d}x|ddD] }||@}qvW|j }nd}dd |jD} t | } | dkst| d d |jD} d d | D} g} xBt| D]6\} }|f|j|| | j}| j | | j|qW|dkr0d }tjt|||jd }||d}xt|D]t} | |t| d||krP|j|fdd | D|dk r|ndfdd | Ddq\WdS)z` we form the data into a 2-d including indexes,values,mask write chunk-by-chunk r)ru1F)r8rINcSsg|] }|jqSr?)r)rrjr?r?r@rsz.AppendableTable.write_data..cSsg|] }|jqSr?)r)rrjr?r?r@rscSs,g|]$}|jtjtj|j|jdqS)rI)Z transposer;Zrollrr)rr r?r?r@rsi)rcsg|]}|qSr?r?)rrj)end_istart_ir?r@rscsg|]}|qSr?r?)rr )rrr?r@rs)indexesrr)rrrEr:r,r7rr:r;rrOrrQrr9rrrreshaperZrirwrite_data_chunk)rrrsrrmasksrjrmrnindexesrZbvaluesrr Z new_shaperowschunksr?)rrr@rsJ       zAppendableTable.write_data)rrrrc Csx|D]}tj|jsdSqW|djd}|t|krHtj||jd}|jj}t|}x t|D]\} } | ||| <qbWx$t|D]\} }|||| |<qW|dk r|jj t dd} | j s|| }t|r|j j ||j jdS)z Parameters ---------- rows : an empty memory space where we are putting the chunk indexes : an array of the indexes mask : an array of the masks values : an array of the values Nr)rF)r8)r;rrrQrZrrrrrrrr_rOr) rrrrrr rrrrrrr?r?r@rs&    z AppendableTable.write_data_chunk)r}r~cCsh|dkst| rh|dkr<|dkr<|j}|jj|jddn(|dkrJ|j}|jj||d}|jj|S|jstdS|j}t ||||d}|j }t |j }t|} | rd|j } t| | dkj} t| sdg} | d| kr| j| | ddkr| jdd| j} xLt| D]@} |jt| | }|j||jd||jddd| } qW|jj| S)NT)r)r}r~rIrrr)rQrrr\rr_Z remove_rowsrrr\rr)Z sort_valuesdiffrLrqrOrrreversedrr)rrRr}r~rr_r]rZ sorted_serieslnrrZpgrrr?r?r@rsB     "  zAppendableTable.delete) NFNNNNNNFNNT)F)NNN)rWrXrYrerArarrfrrr;rrrrr?r?r?r@ros( .?(rc@speZdZUdZdZdZdZeZe e ee fe e dddZee dd d Zdeeeed d dZd S)rTz* support the new appendable table formats rBrJr)rcCs|jdjdkS)NrrI)r9r)rr?r?r@rG`sz"AppendableFrameTable.is_transposed)rcCs|r |j}|S)z these are written transposed )r)rYr[rr?r?r@r`dszAppendableFrameTable.get_objectN)r}r~cs:j|jsdSj|||d}tjrJjjjddtnt}fddtj D}t|dksxt |d}||d} g} x^tj D]N\} } | j krq|| \} }|jddkrt j | }nt| }|jd}|dk r|j|d d jr*|}|}t| t| d dd }n|j}t| t| d dd }|}|jdkrvt|tjrv|jd|jdf}t|tjrt|j||d }n,t|trt|||d }nt|g||d }|j|jkjst |j|jf| j|qWt| dkr | d}n t| dd}t|||d}j |||d}|S)N)rRr}r~rcs"g|]\}}|jdkr|qS)r)r9)rrr0)rr?r@rsz-AppendableFrameTable.read..rIrr'rT)inplacerF)rF)rrq)r)r]r)!rrr_rQrr<rrrrrr:r' from_tuplesr% set_namesrGr*rrr:r;rrrr#ZdtypesrrrOr+r\r)rrRrr}r~rr<ZindsindrqframesrrjZ index_valsrr rrZindex_Zcols_r3r]r?)rr@rksV $       "  zAppendableFrameTable.read)NNNN)rWrXrYrerrArr#rr r r)rgrrGrr`rrfrr?r?r?r@rTXs rTcsveZdZdZdZdZdZeZe e dddZ e e dd d Z dfd d Zdeeeeedfdd ZZS)rRz* support the new appendable table formats rGrHr)rcCsdS)NFr?)rr?r?r@rGsz#AppendableSeriesTable.is_transposed)rcCs|S)Nr?)rYr[rr?r?r@r`sz AppendableSeriesTable.get_objectNc s<t|ts|jpd}|j|}tjf||jjd|S)z- we are going to write this as a frame table r)r[rt)r:r#rFZto_framerrarry)rr[rtrrF)rr?r@ras   zAppendableSeriesTable.write)r}r~rcs|j}|dk rF|rFt|jts"tx"|jD]}||kr*|jd|q*Wtj||||d}|rn|j|jdd|j dddf}|j dkrd|_ |S)Nr)rRrr}r~T)rr) rBr:rrLrrrr set_indexrrF)rrRrr}r~rBrr>)rr?r@rs   zAppendableSeriesTable.read)N)NNNN)rWrXrYrerrArr)rrgrrGrr`rarrfrrr?r?)rr@rRs rRcs(eZdZdZdZdZfddZZS)rSz* support the new appendable table formats rGrIc sJ|jpd}|j|\}|_t|j}|j|||_tjfd|i|S)z- we are going to write this as a frame table rr[)rFrDrrLrOrrra)rr[rrFr )rr?r@ras    z AppendableMultiSeriesTable.write)rWrXrYrerrArarr?r?)rr@rSsrSc@sZeZdZdZdZdZdZeZe e dddZ e dd Z d d Z ed d ZddZdS)rQz< a table that read/writes the generic pytables table format rBrCr)rcCs|jS)N)r)rr?r?r@r'szGenericTable.pandas_typecCst|jddp|jS)Nr_)r*r)rr?r?r@r szGenericTable.storablecCsLg|_d|_g|_dd|jD|_dd|jD|_dd|jD|_dS)z retrieve our attributes NcSsg|]}|jr|qSr?)r)rrjr?r?r@rsz*GenericTable.get_attrs..cSsg|]}|js|qSr?)r)rrjr?r?r@rscSsg|] }|jqSr?)rF)rrjr?r?r@rs)rrvrrNr9r:rt)rr?r?r@r s zGenericTable.get_attrsc Cs|j}|jd}|dk rdnd}tdd|j||d}|g}xlt|jD]^\}}t|ts\tt ||}|j|}|dk r|dnd}t |||g||j||d} |j | qFW|S)z2 create the indexables from the table description rqNrr)rFrr_rrr)rFrqrrpr_rrr) rrrr_rZ_v_namesr:rErr*rrO) rrrTrrYrXrrrrr?r?r@rNs*   zGenericTable.indexablescKs tddS)Nz cannot write on an generic table)r)rrr?r?r@ra:szGenericTable.writeN)rWrXrYrerrArr#rrgrEr'rrrrNrar?r?r?r@rQs  #rQcsheZdZdZdZeZdZej dZ e e dddZ dfd d Zdeeeed fd d ZZS)rUz a frame with a multi-index rKrz ^level_\d+$)rcCsdS)NZappendable_multir?)rr?r?r@r<Fsz*AppendableMultiFrameTable.table_type_shortNc sl|dkrg}n|dkr |jj}|j|\}|_x"|jD]}||kr8|jd|q8Wtjf||d|S)NTr)r[rt)rryrDrrrra)rr[rtrr)rr?r@raJs  zAppendableMultiFrameTable.write)r}r~csDtj||||d}|jj}|jjfdd|jjD|_|S)N)rRrr}r~cs g|]}jj|rdn|qS)N) _re_levelssearch)rrC)rr?r@rbsz2AppendableMultiFrameTable.read..)rrrrrqrr)rrRrr}r~r3)r)rr@rUs  zAppendableMultiFrameTable.read)N)NNNN)rWrXrYrerAr#rrrecompilerrgrEr<rarrfrrr?r?)rr@rU>s  rU)r[rrrcCs|j|}t|}|dk r"t|}|dks4|j|rB|j|rB|St|j}|dk rlt|jj|dd}|j|stddg|j}|||<|jt|}|S)NF)sort) r{r2equalsuniquerslicerrrM)r[rrr{r0Zslicerr?r?r@rjhs   rj)rmrcCstj|}|S)z- for a tz-aware type, return an encoded zone )rZ get_timezone)rmzoner?r?r@rs r)rrmrrcCszt|tr"|jdks"|j|ks"t|dk rdt|dd}|j}t|}t||d}|jdj|}n|rvt j |dd}|S)a coerce the values to a DatetimeIndex if tz is set preserve the input shape if possible Parameters ---------- values : ndarray or Index tz : str or tzinfo coerce : if we do not have a passed timezone, coerce to M8[ns] ndarray NrF)rFrzM8[ns])r) r:r$rmrr*rrArrr;r)rrmrrFr?r?r@rs   r)rFrqrCrurc Csjt|tst|j}t|\}}t|}tj|}t|trbt ||||t |ddt |dd|dSt|t rtt dt j|dd} tj|} | dkrtjdd | Dtjd }t ||dtj|d S| d krt| ||}|jj} t ||d tj| |d S| dkrt |||||dSt|tjr4|jtks8t|dksJt|tj}t |||||d SdS)Nrlrm)rr$rprlrmrnzMultiIndex not supported here!F)r"rcSsg|] }|jqSr?) toordinal)rr r?r?r@rsz"_convert_index..)r)rnrintegerfloating)rr$rprnr)rr)r:rErrFrrrrr&rkr*r'rrr%r;rZint32riZ Time32Col_convert_string_arrayrrtrrrr') rFrqrCrurnrrr$rr+rrtr?r?r@r sL            r )r$rCrurc Cs|dkrt|}n|dkr$t|}n|dkrxytjdd|Dtd}Wqtk rttjdd|Dtd}YqXnT|dkrtj|}n@|d krt|d||d }n&|d krtj|d }ntd||S)NrrrcSsg|]}tj|qSr?)rr)rr r?r?r@rsz$_unconvert_index..)rcSsg|]}tj|qSr?)rr)rr r?r?r@rsrfloatr)rvrCrurrzunrecognized index type )rr)r$r*r;rrrr)r7r$rCrurqr?r?r@rs$    r)rFcCs|js |jS|jj}tj|jdd}|dkr6tdn(|dkrHtdn|dkpV|dks^|jS|j|dd }t|t r~|d }|j} tj| dd}|dkrx\t t |j d D]F} |j | } tj| dd}|dkr|jj| } td | d |d qWt| ||j| j } | j |j ks*t| j |j f| j}t|tr\t|j|pX|jdpXd }t|pfd |}|dk r|j|}||kr|}| jd|dd} | S)NF)r"rz+[date] is not implemented as a table columnrz>too many timezones in this block, create separate data columnsrr)ZdowncastrzCannot serialize the column [z!] because its data contents are [z] object dtyperz|S)r8)rrrrFrr%rZfillnar:rLrrQrigetr7rprrrrtrrfrrrr)rFr.rhrrrvrCrurr+r7rrrrwrtZecir?r?r@rmsH         rm)r7rCrurcCs\t|r(t|jjj||jj|j}t|j}t dt j |}t j |d|d}|S)a Take a string-like that is object dtype and coerce to a fixed size string type. Parameters ---------- data : np.ndarray[object] encoding : str errors : str Handler for encoding errors. Returns ------- np.ndarray[fixed-length-string] rIS)r)rQr)rrEencoderrrrr libwritersmax_len_string_arrayr;r)r7rCruensuredrtr?r?r@r?s  rcCs|j}tj|jtd}t|rvtjt|}d|}t |dt r^t |j j ||dj}n|j|ddjtdd}|dkrd}tj||}|j|S) a* Inverse of _convert_string_array. Parameters ---------- data : np.ndarray[fixed-length-string] nan_rep : the storage repr of NaN encoding : str errors : str Handler for encoding errors. Returns ------- np.ndarray[object] Decoded data. )rUr)ruF)r8Nre)rr;rrrrQrrrr:rr)rEr=rrZ!string_array_replace_from_nan_repr)r7rvrCrurrtrr?r?r@r^s  r)rrrCrucCs6t|tstt|t|r2t|||}||}|S)N)r:rErr _need_convert_get_converter)rrrCruconvr?r?r@rs  r)r$rCrucs8|dkrddS|dkr&fddStd|dS)NrcSstj|ddS)NzM8[ns])r)r;r)rJr?r?r@rKsz _get_converter..rcst|ddS)N)rvrCru)r)rJ)rCrur?r@rKsz invalid kind )r)r$rCrur?)rCrur@rs r)r$rcCs|dkr dSdS)NrrTF)rrr?)r$r?r?r@rsr)rFrc CshyN|ddkrL|ddkrL|ddkrLtjd|}|rL|jd}d|}Wntk rbYnX|S)z Prior to 0.10.1, we named values blocks like: values_block_0 an the name values_0, adjust the given name if necessary. Parameters ---------- name : str version : Tuple[int, int, int] Returns ------- str rrIrrzvalues_block_(\d+)Zvalues_)rrrrl)rFrrgrpr?r?r@rRs$  rR) dtype_strrcCst|}|jds|jdr"d}n|jdr2d}n|jdrBd}n|jdsV|jdr\d}nn|jdrld}n^|jd r|d }nN|jd rd }n>|jd rd }n.|jd rd}n|dkrd}ntd|d|S)zA Find the "kind" string describing the given dtype name. rrrrZrfrrr timedeltarrrrrzcannot interpret dtype of [r)rAr1r)rr$r?r?r@rs.       r)r7cCsbt|tr|j}|jjjdd}|jjdkr@tj|j d}nt|t rP|j }tj|}||fS)zJ Convert the passed data into a storable form and a dtype string. rrrMr$)rr) r:r-rrrFr]r$r;rr(r(r))r7rr?r?r@rs    rc@sDeZdZdZd eeeeedddZddZdd Z d d Z dS) r\z Carries out a selection operation on a tables.Table object. Parameters ---------- table : a Table object where : list of Terms (or convertible to) start, stop: indices to start and/or stop selection N)r_r}r~c CsP||_||_||_||_d|_d|_d|_d|_t|ryt j |dd}|dksZ|dkrt j |}|j t jkr|j|j}}|dkrd}|dkr|jj}t j||||_nNt|j jt jr|jdk r||jkjs|jdk r||jkjrtd||_Wntk rYnX|jdkrL|j||_|jdk rL|jj\|_|_dS)NF)r"rbooleanrz3where must have index locations >= start and < stop)r_rRr}r~ conditionrNZtermsrrrr%r;rrZbool_rr issubclassrrrrgenerateevaluate)rr_rRr}r~inferredr?r?r@rs@       zSelection.__init__cCsv|dkr dS|jj}yt|||jjdStk rp}z,dj|j}td|d|d|WYdd}~XnXdS)z) where can be a : dict,list,tuple,string N)rJrCrwzThe passed where expression: z contains an invalid variable reference all of the variable references must be a reference to an axis (e.g. 'index' or 'columns'), or a data_column The currently defined references are: r)r_rJr0rC NameErrorryrr)rrRrPrZqkeysr?r?r@r,s zSelection.generatecCsX|jdk r(|jjj|jj|j|jdS|jdk rB|jjj|jS|jjj|j|jdS)z( generate the selection N)r}r~) rr_Z read_whererpr}r~rrr)rr?r?r@rBs   zSelection.selectcCs|j|j}}|jj}|dkr$d}n|dkr4||7}|jdkrD|}n|dkrT||7}|jdk rz|jjj|jj||ddS|jdk r|jStj ||S)z( generate the selection NrT)r}r~r) r}r~r_rrZget_where_listrprr;r)rr}r~rr?r?r@rNs    zSelection.select_coords)NNN) rWrXrYrerrrfrrrrr?r?r?r@r\s - r\) rjNNFNTNNNNrcr9) Nr|rcNNNNFN)N)F)rer8rrrrrrtypingrrrrrr r r rZnumpyr;Zpandas._configr r Z pandas._libsrrrZpandas._libs.tslibsrZpandas._typingrrrZpandas.compat._optionalrZpandas.compat.pickle_compatrZ pandas.errorsrZpandas.util._decoratorsrZpandas.core.dtypes.commonrrrrrrrrr Zpandas.core.dtypes.genericr!Zpandas.core.dtypes.missingr"rr#r$r%r&r'r(r)r*r+r,Zpandas.core.arraysr-r.r/Zpandas.core.commoncorecommonrZ pandas.core.computation.pytablesr0r1Zpandas.core.indexes.apir2Zpandas.io.commonr3Zpandas.io.formats.printingr4r5rer6r7r8rrBrArDrGrPrfrUrrVrZWarningr[rOr\rr]Z duplicate_docr&r=rZ dropna_docZ format_docZ config_prefixZregister_optionZis_boolZis_one_of_factoryrdrgrirErr{rrryrrkrrrrrr3rNr,rOrrVrrTrRrSrQrUrjrrrr rrmrrrrrrRrrr\r?r?r?r@s (     ,  0       \*$]q!3V ZXjd1A* <A % !