Barrage Schema Annotation

Deephaven tables support Object-typed columns that can hold arbitrary Java objects. When exporting these tables over Flight using the Barrage format, Deephaven uses Apache Arrow schemas to describe the data. By default, if a column is typed as Object, the Arrow schema may not capture the intended structure of the data, which can lead to inefficient serialization or loss of type information. Use the Table.BARRAGE_SCHEMA_ATTRIBUTE to inject explicit Arrow schema information, which ensures that the Flight export uses the correct wire format.

Use this when your Deephaven column type is too generic for the intended wire type (for example, Object columns that should be exported as Union or Map). This guide includes examples of the Union and Map types, which are currently supported by Deephaven.

How It Works

  1. Extract a base schema with BarrageUtil.schemaFromTable(...). Manages basic type mapping for primitive types and collections of primitives.
  2. Replace the target field with explicit Arrow types.
  3. Attach the schema using withAttributes(Map.of(Table.BARRAGE_SCHEMA_ATTRIBUTE, newSchema)).

Note

withAttributes(...) returns a new table. If you later transform the table (for example, with select, view, or update), attributes may not be preserved and you may need to re-apply the schema. Ideally, you would apply the schema as late as possible before export to minimize this risk.

Example: Annotate Union<String, Double> Columns

The following example creates a table with a column of Objects (limited for this example to String and Double). The Arrow schema annotates the column as a dense union with String and Double branches. The final table can be exported over Flight / Barrage without error.

Example: Annotate Map<String, String> Columns

The following example creates a table with a column of Map<String, Double>. The Arrow schema annotates the column as an Arrow Map with the correct types for key and values. The final table can be exported over Flight / Barrage without error.

Example: Annotate Map<String, Integer> Columns

The following example creates a table with a column of Map<String, Integer>. The Arrow schema annotates the column as an Arrow Map with String keys and Integer values. The final table can be exported over Flight / Barrage without error.

Example: Annotate Map<String, Union> Columns

This example demonstrates the use of Union for values in a Map with String keys. The Union can contain a Double, String, Long, or Integer.