Arrow Flight and Deephaven

Apache Arrow is a high-performance, language-independent columnar memory format for flat and nested data. Deephaven's Arrow Flight integration lets you import and export data with remote Flight clients over gRPC. Deephaven's Barrage protocol extends Flight, so Flight clients can also subscribe to ticking tables and receive only what changes each cycle.

This guide shows how Arrow types map to Deephaven column types, how to choose the Arrow types a table uses on the network (its wire types) when you import or export it, and where to learn how to register your own type mappings.

Arrow type support matrix

Deephaven supports a wide range of Arrow types. The following table lists each supported Arrow type, the Deephaven type it maps to by default, and its supported coercions. Deephaven column types are Java types. For example, long is a Java primitive, and Instant is the Java class java.time.Instant. A coercion is a non-default Deephaven type that Deephaven can convert the Arrow type to on import, and convert back to the Arrow type on export. You request one on import with the deephaven:type metadata tag, and on export by attaching a Flight schema to the table.

Integral coercion

Coercion from a larger integral type to a smaller one, or from an unsigned type to the signed type of the same width, follows Java's narrowing primitive conversion, which keeps only the low-order bits. Some values truncate to null, because Deephaven reserves one value of each primitive type as its null marker. For example, 0x2080 becomes null when coerced to a byte, because QueryConstants.NULL_BYTE is 0x80. Similarly, -1 becomes null when coerced to a char, because QueryConstants.NULL_CHAR is 0xFFFF.

Decimal coercion

Coercing a decimal value to a type with lower precision rounds or truncates it. For example, coercion to any integral type drops the fractional part of the decimal value.

Unsupported Arrow types

Deephaven does not support the following Arrow types:

• Large types such as LargeUtf8, LargeBinary, LargeList, and LargeListView
• Struct
• Utf8View and BinaryView

Supporting the large types would require simulating 64-bit arrays.

Set the Flight schema

Import

When a Flight client uploads data to Deephaven with a DoPut call, the client can specify the destination column types by adding metadata tags to the Flight schema's fields.

The metadata keys are as follows:

• deephaven:type: the canonical class name of the Deephaven type to use for the column. For a list field, this can be an array class, such as int[], or a vector class, such as io.deephaven.vector.LongVector.
• deephaven:componentType: the canonical class name of the element type to use for the column. For an ObjectVector column, set it to choose the element class. Without it, Deephaven uses the default type of the list's element field. For an array column or a primitive vector column, Deephaven infers the element type from deephaven:type.

Any field without type tags gets the default Deephaven type for its Arrow wire type, as listed in the Arrow type support matrix.

The import_table method of the Python client replaces each field's metadata with its own:

• It sets deephaven:type from a fixed mapping for simple Arrow types.
• It leaves deephaven:type off for parameterized types, such as lists, maps, 128-bit and 256-bit decimals, and timestamps with a time zone. Those columns get their default Deephaven type.
• It never sets deephaven:componentType.
• It raises an error for any other Arrow type, including dictionary-encoded and run-end encoded columns.

To upload a type that import_table rejects, or to choose the column types yourself, upload with a Flight client such as pyarrow.flight, and put the type tags in each field's metadata.

Let's upload a table with pyarrow.flight and download it again. The schema asks Deephaven to store a 32-bit integer field as a long column. The upload's Flight descriptor, scope followed by a variable name, publishes the table to the server's query scope as uploaded_table. The download asks for that variable with the ticket s/uploaded_table. Run this code in a separate Python process that has pyarrow installed, not in the Deephaven server console. Replace <your-pre-shared-key> with the key the server prints to its log on startup:

The downloaded column is still int32 on the wire, even though the server stores it as a long column, because Deephaven keeps the uploaded schema (see Round-tripping):

Because this example sends the pre-shared key on every call, the server starts a new session for each call. A long-running client should instead reuse the session token the server returns, or use the Python client, which manages its session for you.

The Groovy guide's Upload a table example shows a complete upload from an Arrow Java client. Its Duration-to-long mapping works only after you register the custom mapping on the server.

Export

You can specify the Flight schema, and so each column's wire type, to use when exporting a table to a Flight client. Use with_attributes to put an Arrow Java org.apache.arrow.vector.types.pojo.Schema object in the table's attributes under the key BarrageSchema. This overrides the schema Deephaven generates from the table's column types.

To change only some fields, start from the generated schema, which the Java method BarrageUtil.schemaFromTable returns. The generated schema may run-end encode or dictionary encode a column whose values repeat often. Check each field's type and dictionary before you replace its type.

Building the Schema object from Python means working with Arrow Java types through jpy. Barrage schema annotation outlines the steps, and the Groovy Barrage schema annotation guide has worked examples you can adapt.

Round-tripping

When a client uploads a table with DoPut, Deephaven stores the Flight schema in the table's BarrageSchema attribute. This is convenient, because you can then download the table in exactly the same wire format.

Most table operations drop this attribute, even when they keep every column's type unchanged. Filters (such as where), sort, reverse, flatten, first_by, and last_by keep it, and so do the constituent tables of a partition_by. To preserve the original wire types after any other operation, re-attach a schema with with_attributes as described in Export. The schema needs one field for each of the result table's columns, so after an operation that adds, drops, or renames columns, adjust the schema's fields to match.

Turn off the default toString export

By default, when Deephaven has no writer for a column's type, it exports the column's values as UTF-8 strings, using each value's toString. This is not always desirable. Some types have no meaningful string form, and others can't be converted to a string without losing data. To turn off this behavior for every export from the server, call the Java method disableToStringUnknownTypes through jpy:

Custom mappings

You implement custom type mappings in Java by registering readers and writers with the DefaultChunkReaderFactory and DefaultChunkWriterFactory classes. See the Groovy Arrow Flight guide for a worked example that registers a mapping, uploads a table from an Arrow Java Flight client, and downloads it again.

Related documentation

• Barrage schema annotation
• What is Barrage?
• Python client quickstart
• Barrage Extensions Package Summary
• Arrow Flight Package Summary