Deephaven and PyArrow
This guide shows how to use Deephaven's Arrow integration with PyArrow. PyArrow is a Python library for Apache Arrow, which is a columnar memory format similar to Deephaven's table format. The integration lets you:
- Read Feather V2 files into Deephaven tables. Feather V2 is the on-disk form of the Arrow IPC (interprocess communication) format, and PyArrow's
write_featherwrites it by default. - Convert between Deephaven tables and PyArrow tables.
- Write Deephaven tables to Feather files, by converting them to PyArrow tables and passing the result to PyArrow's
write_feather.
The deephaven.arrow submodule provides three functions for these tasks:
Read a Feather file
Use read_feather to read a Feather file into a Deephaven table as iris:
Convert between Deephaven and PyArrow tables
Note
Converting between Deephaven and PyArrow tables copies the converted columns' data in memory. Converting a large table needs free memory for more than one extra copy of that data while the conversion runs, in the JVM heap as well as in PyArrow's memory.
For a refreshing table, to_arrow converts a consistent snapshot of the table's current rows. The PyArrow table doesn't update when the source table ticks.
Use to_arrow to convert the iris table from Read a Feather file to a PyArrow table twice: once with all columns, and once with only the Class column.
The printed schema shows that some columns aren't plain Arrow types. to_arrow can return dictionary-encoded or run-end-encoded columns. A dictionary-encoded column stores each distinct value once, and each row holds an index into that list. A run-end-encoded column stores each run of repeated values once.
Starting in version 43.0, Deephaven picks these encodings itself by default: dictionary encoding for string and other object columns with few distinct values, and run-end encoding for columns whose values repeat in runs. The Class column gets both. To control these encodings, see Barrage schema annotation.
to_table accepts encoded columns, but read_feather can't read them correctly from a Feather file. This guide writes pa_iris to a Feather file later, so it decodes the columns first:
The helper turns dictionary-encoded values back into plain values by casting them to their value type, and expands run-end-encoded columns with PyArrow's run_end_decode. A column can be both, as Class is here: its runs hold dictionary-encoded values. run_end_decode can't expand runs of dictionary values, so the helper decodes the values inside each run-end-encoded chunk first. It then slices the rebuilt chunk to the rows the original covers, and only then expands it. This matters if you sliced the table: the helper expands only the rows you kept, not the whole chunk they came from. Passing the decoded type to pa.chunked_array lets the helper handle a column with no chunks, such as one from an empty table.
Then, use to_table to convert pa_iris back to a Deephaven table twice. The first call copies all columns. The second call passes cols to copy every column except Class.
to_table also converts tables you build with PyArrow itself, as long as their column types are supported:
Supported column types
to_table accepts these Arrow column types:
- Exact types listed in
deephaven.arrow.SUPPORTED_ARROW_TYPES, such aspa.int64(),pa.string(), andpa.timestamp("ns").to_tablemaps each of these directly to a Deephaven type. - Some parameterized types, which take arguments such as a precision, an element type, or a time zone: 128-bit and 256-bit decimals (
pa.decimal128andpa.decimal256), lists, list views, fixed-size lists, maps, fixed-size binary, unions, and timestamps with a time zone.SUPPORTED_ARROW_TYPESdoesn't list these types, soto_tableleaves their mapping to the Deephaven engine. The element types of a list, map, or union must be supported as well. - Dictionary-encoded and run-end-encoded columns whose values have one of the types above.
deephaven.arrow.SUPPORTED_ARROW_ENCODINGSlists these encodings. The column becomes a plain Deephaven column of its value type.
To see the full list of exact types, print SUPPORTED_ARROW_TYPES:
to_table raises an error for any other column type, including large strings, large binary, large lists, and structs. To check whether it accepts a type, call deephaven.arrow.is_deephaven_compatible, which covers all three groups above.
Write a Feather file
To write a Deephaven table to a Feather file, convert it to a PyArrow table with to_arrow and decode any encoded columns, as in Convert between Deephaven and PyArrow tables. Then pass the result to PyArrow's write_feather. This example writes the decoded pa_iris to a file and reads it back with read_feather:
read_feather reads fewer column types than to_table: it can't read list, map, union, duration, or most time-of-day columns. To bring such a file back into Deephaven, read it with PyArrow's read_table and convert the result with to_table.