Deephaven and PyArrow

This guide shows how to use Deephaven's Arrow integration with PyArrow. PyArrow is a Python library for Apache Arrow, which is a columnar memory format similar to Deephaven's table format. The integration lets you:

  • Read Feather V2 files into Deephaven tables. Feather V2 is the on-disk form of the Arrow IPC (interprocess communication) format, and PyArrow's write_feather writes it by default.
  • Convert between Deephaven tables and PyArrow tables.
  • Write Deephaven tables to Feather files, by converting them to PyArrow tables and passing the result to PyArrow's write_feather.

The deephaven.arrow submodule provides three functions for these tasks:

Read a Feather file

Use read_feather to read a Feather file into a Deephaven table as iris:

Convert between Deephaven and PyArrow tables

Note

Converting between Deephaven and PyArrow tables copies the converted columns' data in memory. Converting a large table needs free memory for more than one extra copy of that data while the conversion runs, in the JVM heap as well as in PyArrow's memory.

For a refreshing table, to_arrow converts a consistent snapshot of the table's current rows. The PyArrow table doesn't update when the source table ticks.

Use to_arrow to convert the iris table from Read a Feather file to a PyArrow table twice: once with all columns, and once with only the Class column.

The printed schema shows that some columns aren't plain Arrow types. to_arrow can return dictionary-encoded or run-end-encoded columns. A dictionary-encoded column stores each distinct value once, and each row holds an index into that list. A run-end-encoded column stores each run of repeated values once.

Starting in version 43.0, Deephaven picks these encodings itself by default: dictionary encoding for string and other object columns with few distinct values, and run-end encoding for columns whose values repeat in runs. The Class column gets both. To control these encodings, see Barrage schema annotation.

to_table accepts encoded columns, but read_feather can't read them correctly from a Feather file. This guide writes pa_iris to a Feather file later, so it decodes the columns first:

The helper turns dictionary-encoded values back into plain values by casting them to their value type, and expands run-end-encoded columns with PyArrow's run_end_decode. A column can be both, as Class is here: its runs hold dictionary-encoded values. run_end_decode can't expand runs of dictionary values, so the helper decodes the values inside each run-end-encoded chunk first. It then slices the rebuilt chunk to the rows the original covers, and only then expands it. This matters if you sliced the table: the helper expands only the rows you kept, not the whole chunk they came from. Passing the decoded type to pa.chunked_array lets the helper handle a column with no chunks, such as one from an empty table.

Then, use to_table to convert pa_iris back to a Deephaven table twice. The first call copies all columns. The second call passes cols to copy every column except Class.

to_table also converts tables you build with PyArrow itself, as long as their column types are supported:

Supported column types

to_table accepts these Arrow column types:

  • Exact types listed in deephaven.arrow.SUPPORTED_ARROW_TYPES, such as pa.int64(), pa.string(), and pa.timestamp("ns"). to_table maps each of these directly to a Deephaven type.
  • Some parameterized types, which take arguments such as a precision, an element type, or a time zone: 128-bit and 256-bit decimals (pa.decimal128 and pa.decimal256), lists, list views, fixed-size lists, maps, fixed-size binary, unions, and timestamps with a time zone. SUPPORTED_ARROW_TYPES doesn't list these types, so to_table leaves their mapping to the Deephaven engine. The element types of a list, map, or union must be supported as well.
  • Dictionary-encoded and run-end-encoded columns whose values have one of the types above. deephaven.arrow.SUPPORTED_ARROW_ENCODINGS lists these encodings. The column becomes a plain Deephaven column of its value type.

To see the full list of exact types, print SUPPORTED_ARROW_TYPES:

to_table raises an error for any other column type, including large strings, large binary, large lists, and structs. To check whether it accepts a type, call deephaven.arrow.is_deephaven_compatible, which covers all three groups above.

Write a Feather file

To write a Deephaven table to a Feather file, convert it to a PyArrow table with to_arrow and decode any encoded columns, as in Convert between Deephaven and PyArrow tables. Then pass the result to PyArrow's write_feather. This example writes the decoded pa_iris to a file and reads it back with read_feather:

read_feather reads fewer column types than to_table: it can't read list, map, union, duration, or most time-of-day columns. To bring such a file back into Deephaven, read it with PyArrow's read_table and convert the result with to_table.