Barrage schema annotation

Deephaven tables support Object-typed columns that can hold arbitrary Java objects. When exporting these tables over Flight using the Barrage format, Deephaven uses Apache Arrow schemas to describe the data. By default, if a column is typed as Object, the Arrow schema may not capture the intended structure of the data, which can lead to inefficient serialization or loss of type information. Put the schema in the table's BarrageSchema attribute (the Java constant Table.BARRAGE_SCHEMA_ATTRIBUTE) to inject explicit Arrow schema information, which ensures that the Flight export uses the correct wire format.

Use this when your Deephaven column type is too generic for the intended wire type (for example, Object columns that should be exported as Union or Map).

When to use schema annotation

Schema annotation is needed when:

  • Exporting Object-typed columns that contain Union types (e.g., columns that may hold either String or Double values)
  • Exporting Map columns where key/value types need explicit Arrow type definitions
  • The default schema inference produces incorrect or inefficient wire formats

How it works

  1. Extract a base schema with BarrageUtil.schemaFromTable.
  2. Replace the target field with explicit Arrow types (e.g., ArrowType.Utf8, ArrowType.Union, ArrowType.Map).
  3. Attach the schema using with_attributes.

Note

with_attributes returns a new table. Most table operations, including select, view, and update, drop the schema attribute; filters, sort, reverse, flatten, first_by, last_by, and partition_by keep it. Apply the schema as the last step before export.

Supported types

The following complex Arrow types can be annotated:

  • Union (Dense or Sparse): For columns containing multiple possible types
  • Map: For key-value pair columns with explicit key/value type definitions
  • Nested combinations: Maps with Union values, etc.
  • Run-end encoded and dictionary-encoded fields, for columns whose values repeat

You don't always need to annotate a column to get an encoding. The server properties BarrageUtil.ree.autoDetectEnabled and BarrageUtil.dictionary.autoDetectEnabled are both on by default. While either is enabled, the server may run-end encode or dictionary encode a column of a table you never annotated, so a Flight client can receive encoded columns from an unannotated table. An encoding in an attached schema always applies, whatever these properties are set to.

Working examples

Schema annotation requires direct manipulation of Apache Arrow Java types via jpy. This involves careful handling of Java constructor overloads, null values, and collection types that can be complex in Python.

For working, tested examples, see the Groovy Barrage schema annotation guide. The Groovy examples demonstrate:

  • Annotating Union<String, Double> columns
  • Annotating Map<String, String> columns
  • Annotating Map<String, Integer> columns
  • Annotating Map<String, Union> columns

The Groovy patterns can be adapted for Python use with jpy, but require attention to how Python maps to Java types.