Arrow Flight and Deephaven

Apache Arrow is a high-performance, language-independent columnar memory format for flat and nested data. Deephaven's Arrow Flight integration lets you import and export data with remote Flight clients over gRPC. Deephaven's Barrage protocol extends Flight, so Flight clients can also subscribe to ticking tables and receive only what changes each cycle.

This guide shows how Arrow types map to Deephaven column types, how to choose the Arrow types a table uses on the network (its wire types) when you import or export it, and how to register your own type mappings.

Arrow type support matrix

Deephaven supports a wide range of Arrow types. The following table lists each supported Arrow type, the Deephaven type it maps to by default, and its supported coercions. A coercion is a non-default Deephaven type that Deephaven can convert the Arrow type to on import, and convert back to the Arrow type on export. You request one on import with the deephaven:type metadata tag, and on export by attaching a Flight schema to the table.

Integral coercion

Coercion from a larger integral type to a smaller one, or from an unsigned type to the signed type of the same width, follows Java's narrowing primitive conversion, which keeps only the low-order bits. Some values truncate to null, because Deephaven reserves one value of each primitive type as its null marker. For example, 0x2080 becomes null when coerced to a byte, because QueryConstants.NULL_BYTE is 0x80. Similarly, -1 becomes null when coerced to a char, because QueryConstants.NULL_CHAR is 0xFFFF.

Decimal coercion

Coercing a decimal value to a type with lower precision rounds or truncates it. For example, coercion to any integral type drops the fractional part of the decimal value.

Unsupported Arrow types

Deephaven does not support the following Arrow types:

• Large types such as LargeUtf8, LargeBinary, LargeList, and LargeListView
• Struct
• Utf8View and BinaryView

Supporting the large types would require simulating 64-bit arrays.

Set the Flight schema

Import

When a Flight client uploads data to Deephaven with a DoPut call, the client can specify the destination column types by adding metadata tags to the Flight schema's fields.

The metadata keys are as follows:

• deephaven:type: the canonical class name of the Deephaven type to use for the column. For a list field, this can be an array class, such as int[], or a vector class, such as io.deephaven.vector.LongVector.
• deephaven:componentType: the canonical class name of the element type to use for the column. For an ObjectVector column, set it to choose the element class. Without it, Deephaven uses the default type of the list's element field. For an array column or a primitive vector column, Deephaven infers the element type from deephaven:type.

Any field without type tags gets the default Deephaven type for its Arrow wire type, as listed in the Arrow type support matrix.

The Upload a table example shows how to set the schema metadata from an Arrow Java Flight client. It maps Duration to long, which isn't a supported coercion, so it works only after you register the custom mapping.

Export

You can specify the Flight schema, and so each column's wire type, to use when exporting a table to a Flight client. Use withAttributes to store an Arrow Java org.apache.arrow.vector.types.pojo.Schema object in the table's attributes under the key Table.BARRAGE_SCHEMA_ATTRIBUTE. The key's value is BarrageSchema. The schema you attach overrides the one Deephaven generates from the table's column types.

When you build the schema:

• Start from BarrageUtil.schemaFromTable, which returns the default schema, and change only the fields you need.
• Check each field's type and dictionary before you replace its type. The default schema may run-end encode or dictionary encode a column whose values repeat often. Barrage schema annotation shows how to check which encoding a client receives.

The following example fetches the default schema and changes the export wire type of column D from Int(64, signed) to Timestamp(MILLISECOND). Deephaven's default long writer treats each value as nanoseconds and converts it to milliseconds on the wire. The column has no repeated values, so Deephaven doesn't encode its field, and the example can replace the field's type directly:

For focused examples that use Table.BARRAGE_SCHEMA_ATTRIBUTE for Union, Map, run-end encoded, and dictionary-encoded wire types, see Barrage schema annotation.

Round-tripping

When a client uploads a table with DoPut, Deephaven stores the Flight schema in the table's Table.BARRAGE_SCHEMA_ATTRIBUTE attribute, whose value is BarrageSchema. This is convenient, because you can then download the table in exactly the same wire format.

Most table operations drop this attribute, even when they keep every column's type unchanged. Filters (such as where), sort, reverse, flatten, firstBy, and lastBy keep it, and so do the constituent tables of a partitionBy. To preserve the original wire types after any other operation, re-attach a schema with withAttributes as shown in Export. The schema needs one field for each of the result table's columns, so after an operation that adds, drops, or renames columns, adjust the schema's fields to match.

Turn off the default toString export

By default, when Deephaven has no writer for a column's type, it exports the column's values as UTF-8 strings, using each value's toString. This is not always desirable. Some types have no meaningful string form, and others can't be converted to a string without losing data. To turn off this behavior for every export from the server, call disableToStringUnknownTypes:

Custom mappings

You can register new Arrow-to-Deephaven type mappings, or replace the reader or writer for an existing pair. You can't change the default Deephaven type for an Arrow type, which applies when no deephaven:type metadata is present.

The DefaultChunkReaderFactory and DefaultChunkWriterFactory classes maintain the registry of import/export mappings. Their source code shows more examples of reusing Deephaven's internal helpers in your own readers and writers.

Let's go through a full example of how to add and use a custom mapping from an Arrow Duration to primitive long and back. The client steps run in an Arrow Flight client, such as a separate Java or Groovy process that has Arrow Flight on its classpath.

1. On the server, register the custom mapping with DefaultChunkReaderFactory and DefaultChunkWriterFactory.
2. On the client, prepare a remote client to connect to the Deephaven server.
3. On the client, upload a table to the query scope variable uploadedTable.
4. On the server, verify the uploaded table's column type and data.
5. On the client, download the table from the server and print the returned data.

Register the custom mapping

This step runs on the Deephaven server. It registers a custom reader and writer for the Arrow Duration type. The reader converts incoming Duration values to a primitive long column in nanoseconds. The writer converts the nanoseconds back to the field's time unit on export.

Prepare the remote client

This step and the next run in the Flight client, not in the Deephaven server console. The client connects to the server and authenticates with the server's pre-shared key, which a default server prints to its log on startup.

Upload a table

This step runs in the Flight client. It uploads millisecond Duration values and tags the field with deephaven:type so that Deephaven stores the column as a long.

Verify the upload on the server

Run these snippets in the Deephaven server console to verify that the client uploaded the table correctly. First, check the column type:

The console output, written out here as a table, shows that the destination column type is long:

Next, print the table's data:

The console output, written out here as a table, shows values in nanoseconds. The client uploaded milliseconds, and the custom reader multiplied each value by the time-unit factor. Your values differ from these because the client uploads random durations:

Download the table

This step runs in the Flight client. You don't need to set Table.BARRAGE_SCHEMA_ATTRIBUTE for this download. Deephaven keeps the uploaded schema on the table and uses it when you download that table, or a table derived from it by an operation that keeps the attribute. Round-tripping lists those operations. Being able to round-trip a table from a client without setting a custom export schema is a quality-of-life feature of Deephaven.

For the same upload, the output looks like this. The first ten values match the table above:

Related documentation

• Barrage schema annotation
• What is Barrage?
• Barrage Extensions Package Summary
• Arrow Flight Package Summary