Barrage schema annotation
Deephaven tables support Object-typed columns that can hold arbitrary Java objects. When exporting these tables over Flight using the Barrage format, Deephaven uses Apache Arrow schemas to describe the data. By default, if a column is typed as Object, the Arrow schema may not capture the intended structure of the data, which can lead to inefficient serialization or loss of type information. Use the Table.BARRAGE_SCHEMA_ATTRIBUTE to inject explicit Arrow schema information, which ensures that the Flight export uses the correct wire format.
Use this when your Deephaven column type is too generic for the intended wire type (for example, Object columns that should be exported as Union or Map), or when you want to opt into a wire-level compression such as Run-End Encoding. This guide includes examples of the Union, Map, and RunEndEncoded types and of dictionary encoding, all of which are supported by Deephaven.
How it works
- Extract a base schema with
BarrageUtil.schemaFromTable(...). It maps every column to a default Arrow type; columns whose type has no native mapping, including genericObjectcolumns, default toUtf8(string). Encoding auto-detection is enabled by default, so the base schema may already include run-end or dictionary encodings for columns that qualify; with it disabled, every column gets its default Arrow type. - Replace the target field with explicit Arrow types.
- Attach the schema using
withAttributes(Map.of(Table.BARRAGE_SCHEMA_ATTRIBUTE, newSchema)).
Note
withAttributes(...) returns a new table. Filters (where, whereIn, whereNotIn), sorts (sort, sortDescending), reverse, flatten, firstBy, lastBy, and attribute-only operations such as withAttributes keep the schema attribute; partitionBy copies it to each constituent table. Other transformations, such as select, view, update, head, or aggBy, drop it, and you must re-apply the schema. Apply the schema as late as possible before export.
Example: Annotate Union<String, Double> columns
The following example creates a table with a column of Objects (limited for this example to String and Double). The Arrow schema annotates the column as a dense union with String and Double branches. The final table can be exported over Flight / Barrage without error.
Example: Annotate Map<String, String> columns
The following example creates a table with a column of Map<String, String>. The Arrow schema annotates the column as an Arrow Map with the correct types for key and values. The final table can be exported over Flight / Barrage without error.
Example: Annotate Map<String, Integer> columns
The following example creates a table with a column of Map<String, Integer>. The Arrow schema annotates the column as an Arrow Map with String keys and Integer values. The final table can be exported over Flight / Barrage without error.
Example: Annotate Map<String, Union> columns
This example demonstrates using Union for values in a Map with String keys. The Union can contain a Double, String, Long, or Integer.
Example: Run-End Encoded (REE) columns
Run-End Encoding is a wire-level optimization for columns with many repeated values. Instead of sending every value, the column is serialized as two child arrays:
run_ends— a non-nullable integer array of cumulative 1-based end indices, one per run. The last value always equals the logical length of the encoded array, which is the column's row count. When a subscriber requests columns as lists, that length is the number of values inside the list, not the record batch's length of1.values— the values that will be repeated in the run.
A column of 1,000 rows where the same integer repeats 100 times in a row costs 10 run_end entries + 10 value entries instead of 1,000 integers. Deephaven stores the column flat (unchanged type); REE is a transport-only optimization. The run_ends integer width is determined by the Arrow field structure you supply via BARRAGE_SCHEMA_ATTRIBUTE. Int16, Int32, and Int64 are supported; use Int32 unless you have a specific reason to use another width. Note that Int16 run_ends constrain the effective batch size to at most Short.MAX_VALUE / 32,767 rows per record batch.
To confirm that the column really is sent run-end encoded, see Verify the encoding from a subscriber below.
Example: Dictionary-Encoded columns
Dictionary Encoding is a wire-level optimization for low-cardinality columns. Instead of sending each value in full, Deephaven sends each unique value to a subscriber once (in a DictionaryBatch message), until the dictionary is reset, and replaces each row with a compact integer index.
A string column with 1,000 rows drawn from only 5 distinct values costs 5 full string entries (in the dictionary) + 1,000 integer indices, rather than 1,000 full strings. Deephaven stores the column flat (unchanged type); dictionary encoding is a transport-only optimization.
The DictionaryEncoding index width controls the integer type used for indices:
Int32(32-bit signed) — handles up to about 2.1 billion distinct values; suitable for almost all use cases.Int8(8-bit signed) — the most compact option, but limits the dictionary to at most 128 distinct values.Int16(16-bit signed) — more compact thanInt32, but limits the dictionary to at most 32,768 distinct values.Int64(64-bit signed) — accepted, but indices are computed as 32-bit integers, so it adds no capacity overInt32.
Caution
Dictionary updates are sent as deltas, so entries accumulate as new unique values appear. To prevent unbounded growth on the server and client, Deephaven resets the dictionary before writing a message when the dictionary has grown larger than the table or viewport, or when the message's new values could push it past what the index type can represent (128 distinct values for Int8, 32,768 for Int16). A reset flushes the current dictionary and accumulates only newly encountered values. A column whose values change over time therefore does not hit the limit just because it has held more distinct values in total than the index can represent. Deephaven still throws an error at serialization time if the values in a single update or snapshot exceed what the index type can represent. Prefer Int32 unless the number of distinct values a column holds at one time stays within the smaller limit.
Verify the encoding from a subscriber
Deephaven sends the export schema to every subscriber. A Deephaven (Java engine) subscriber, such as another server or a URI, stores it on the resulting table under the same Table.BARRAGE_SCHEMA_ATTRIBUTE. Reading that attribute back tells you exactly which encoding each column was sent with.
Run the Run-End Encoded example above so that table_w_attributes exists, then subscribe to it — from a second Deephaven instance, or from the same instance over a URI (the target must allow anonymous authentication):
This prints:
status arrived as RunEndEncoded with Int32 run ends, exactly as annotated, while value was sent unencoded. Running the same check against the dictionary-encoded example prints dictionary_encoded=true for status instead.
Subscribing to a table with no BARRAGE_SCHEMA_ATTRIBUTE prints false for both facets of every column, unless the server has encoding auto-detection enabled. The BarrageUtil.ree.autoDetectEnabled and BarrageUtil.dictionary.autoDetectEnabled properties are both on by default; while either is enabled, the server may choose an encoding on its own for a table you never annotated, and this check is how you see what it picked.
Use println wire_schema.toJson() to dump the entire negotiated schema, including each field's deephaven:type metadata.
Note
These encodings do not change the Deephaven column type — the subscriber's status column is still a String, and the subscriber's TableDefinition is identical either way. Both encodings are transport-only optimizations, so the schema attribute is the only thing that tells you how the bytes were sent.
Caution
Only some operations propagate the attribute (filters, sorts, reverse, flatten, firstBy, lastBy, and attribute-only operations; see the note in How it works). Read it from the table returned by resolve rather than from a derived table.
From the producer
The server logs this decision each time it computes a table's Barrage schema. That includes every export, and also direct calls to BarrageUtil.schemaFromTable, so a table can appear more than once. Raise the level of the io.deephaven.extensions.barrage.util.BarrageUtil logger — in your logging configuration, or at runtime with ch.qos.logback.classic.Logger#setLevel — to DEBUG for a one-line summary each time, or to TRACE to also dump the complete Arrow schema:
The summary reports where the encodings came from — an explicit BARRAGE_SCHEMA_ATTRIBUTE, or auto-detection — which is how you confirm what the server chose for a table you did not annotate yourself. Tables are named by their Table.BARRAGE_PERFORMANCE_KEY_ATTRIBUTE when it is set, and by the table description otherwise.