Barrage metrics for performance monitoring

Barrage is the name of Deephaven's IPC table transport. This guide explains what statistics are recorded and how to access them. For a step-by-step account of where in the life of an update each statistic is recorded, see Interpret Barrage metrics.

Access Barrage metrics tables

You can access these tables as follows:

This is what the subscriptions table looks like when there are live subscriptions:

This is what the snapshots table looks like after processing a few requests:

Barrage subscription metrics summary

Subscription statistics are presented in percentiles bucketed over a time period (see BarragePerformanceLog.cycleDurationMillis in Extra configuration). Each row describes one statistic for one publisher — a table and update interval, one tree or rollup subscription, or one receiver's local copy of a table — over one period. A statistic with no samples in the period has no row. The StatType column names the statistic, Count is the number of samples in the period, and Pct50, Pct75, Pct90, Pct95, Pct99, and Max are the distribution of the sampled values. The TableId and TableKey columns identify the table (or tree or rollup subscription, or receiver's local copy), and Time marks the end of the period. Subscriptions to the same table with different update intervals are recorded as separate rows that share the same TableId and TableKey.

Sender statistics are recorded by the server that publishes the table. Receiver statistics are recorded by the process that subscribes to or snapshots the table, in its own subscription metrics table, and only when that process uses the Java Barrage client — for example, a Deephaven server that uses barrage_session or a URI, or the Java client. Clients such as pydeephaven and the JavaScript client record no receiver statistics.

These are the values of StatType:

Barrage snapshot metrics summary

Snapshot statistics are recorded for snapshots requested with a Barrage snapshot request or Arrow Flight DoGet. Each row also includes the TableId and TableKey of the table and the RequestTime at which the request was received. A refreshing table produces one row per request. A static table is sent in chunks and produces one row per chunk, all with the same RequestTime: SnapshotNanos is the total so far, and WriteNanos and WriteBytes are for that chunk.

Identify a table

Tables are identified by their TableId and TableKey. For sender statistics, the TableId is the source table's System.identityHashCode, written in hexadecimal. For receiver statistics, it is the identity hash code of the subscriber's local copy of the table. For tree and rollup table subscriptions, it is the identity hash code of the subscription rather than of a table. The TableKey defaults to Table.getDescription but can be overridden by setting the BarragePerformanceTableKey attribute on the table you publish via with_attributes.

Extra configuration

Here are server-side flags that change the behavior of Barrage metrics.

• -DBarragePerformanceLog.enableAll: record metrics for tables that do not have an explicit TableKey (default: true).
• -DBarragePerformanceLog.cycleDurationMillis: the interval to flush aggregated statistics (default: 60000 - once per minute).

Control subscription snapshot size

When a client subscribes to a table, or changes an existing subscription (for example, by moving its viewport), the server sends a snapshot of the requested data. For large tables, constructing this snapshot requires holding the data in memory on the server side, which can lead to out-of-memory (OOM) errors.

To address this, Deephaven can break a large snapshot into a series of smaller chunks. This behavior is controlled by the following properties:

• -DBarrageMessageProducer.subscriptionGrowthEnabled: When true (the default), the server limits the size of each snapshot chunk. When false, the server sends the entire snapshot at once (unlimited size).

These properties control the chunk size for growing subscriptions (when subscription growth is enabled), and for every snapshot of a static table requested with a Barrage snapshot request or DoGet, which is always sent in chunks:

• -DBarrageUtil.minSnapshotCellCount: The minimum number of cells (rows × columns) per snapshot chunk. Default: 8192.
• -DBarrageUtil.maxSnapshotCellCount: The maximum number of cells per snapshot chunk. Default: 16777216 (approximately 16 million).

The server adaptively adjusts the chunk size between these bounds based on how long each snapshot takes to generate, targeting a percentage of the update graph cycle time:

• -DBarrageUtil.targetSnapshotPercentage: The fraction of the update graph's target cycle duration that each snapshot chunk should take to generate. Default: 0.25.

Reducing publisher memory usage

For systems serving very large tables to many subscribers, you can reduce publisher-side memory usage by lowering maxSnapshotCellCount. Setting minSnapshotCellCount equal to maxSnapshotCellCount fixes the chunk size and disables adaptive sizing:

This configuration limits each snapshot chunk to at most 1 million cells — well below the 16 million default maximum. Subscribers receive the full table data as a series of snapshot chunks rather than all at once. While a subscription is still growing, the server sends the next chunk as soon as the previous one is done, without waiting for the next update interval.

Compact pending deltas

The server records one delta — the set of changes from a single update graph cycle — for each table and update interval, shared by every subscriber to that table with that interval, then sends the accumulated deltas when the interval elapses. When subscribers are served less often than the table ticks, the server therefore holds every intervening cycle's data at once, even though the message subscribers eventually receive is usually the size of the combined change rather than the sum of the individual ones. (For a blink table it is the sum, since every row must be delivered.) Pending memory grows with both the number of cycles per update interval and the size of each cycle's changes.

To limit that growth, the server combines the pending deltas in the background, before the interval elapses. Compacting costs processor time and saves memory, so the server pays for it only where there is memory to reclaim. The server never compacts blink tables, or a queue of fewer than two pending deltas. Otherwise, it compares the storage the pending deltas occupy against the storage they would occupy compacted, and compacts when the saving clears both of two thresholds:

held is the storage the pending deltas currently occupy, which the server reports as PendingDeltaBytes. compacted is an estimate of what they would occupy after compacting. Both measure the capacity of the chunks allocated rather than the rows stored in them, so a queue of many small updates is scored on the memory it actually holds.

The two thresholds answer different questions. The fraction asks whether compacting is worthwhile at all; the floor asks whether the saving is large enough to be worth the work.

• -DBarrageMessageProducer.compactionEnabled: When true (the default), the server compacts a table's pending deltas between update intervals. When false, deltas accumulate untouched until the interval elapses.
• -DBarrageMessageProducer.compactionMinFreedFraction: The fraction of the pending deltas' storage that compacting must release for the server to do it. Default: 0.5. A higher value copies less data but lets the pending deltas grow larger — at 0.9 the server copies about a ninth as much and holds about ten times the compacted footprint.
• -DBarrageMessageProducer.compactionFloorBytes: The number of bytes compacting must release, whatever fraction of the total that represents. Default: 4194304 (4 MiB). This keeps a stream of very small updates from compacting on every cycle, where the work costs the same as a compaction that reclaims far more.
• -DBarrageMessageProducer.deltaChunkSize: The number of rows in each chunk a delta records. Default: 65536, the largest pooled chunk capacity (which is itself set by -DChunkPoolConstants.largestPooledChunkLog2Capacity, default 16). A value that is not a power of two rounds up to the next one.

Write to subscribers in parallel

Each time the server propagates an update to a table's subscribers, it writes each subscriber its own view of the update, which holds only the columns, rows, and encoding that subscriber asked for. Writing a message serializes it into the gRPC stream's buffers. Serializing is processor work that does not wait for the client, so the server can write to several subscribers at once. Parallel writes need a message that goes to more than one subscriber and BarrageMessageProducer.propagationThreads greater than 1. Even then, the other threads come from a pool that every table shares, and when that pool is busy, the propagation thread writes to the remaining subscribers itself.

• -DBarrageMessageProducer.propagationThreads: The most threads that write one update to a table's subscribers at once, counting the thread that runs the propagation. Default: the number of available processors. Above 1, the other threads come from a pool that every table shares, which never holds more than this number less one. A value of 1 or less writes to subscribers one after another on the propagation thread, and the server makes no pool. The same threads also copy a shared write to its subscribers (see Share one write among identical subscribers), so this setting limits that too: with 1 or less, the server copies to one subscriber after another.

Share one write among identical subscribers

A subscriber to a whole table, rather than to a viewport of it, is a full-table subscriber. Full-table subscribers that would receive identical bytes share one write of each update. The server writes the update once and copies the bytes to each of them, which saves the processor time of writing it again for every subscriber.

Full-table subscribers share a write when they:

• Have received the whole table, rather than still receiving it in parts (see Control subscription snapshot size).
• Use the same subscription options, such as the batch size, the maximum message size, and how nulls are sent.
• Subscribe to the same columns.

Viewport subscribers always get a write of their own.

The server copies a shared write to its subscribers one message at a time:

• Each message holds exactly one batch, either a record batch or a dictionary batch. A message never holds both.
• For a table with dictionary-encoded columns, each dictionary batch is its own message, sent ahead of the record batch that uses its values.
• The server copies each message to every subscriber before it writes the next, so sharing holds at most one extra batch in memory for each write in progress.
• The server copies a large message to the subscribers in parallel: when the message's size, multiplied by one less than the number of subscribers that share the write, reaches BarrageMessageProducer.parallelCopyMinBytes. The copies use the same threads as parallel writes, so they also depend on BarrageMessageProducer.propagationThreads.
• A record batch or dictionary batch is no larger than the subscriber's maximum message size, which is BarrageMessageWriterImpl.maxOutboundMessageSize unless the subscriber sets its own (see Message batching). The exception is a single row, or a single dictionary value, too large to fit, which the server sends anyway.

When a table has dictionary-encoded columns, subscribers that share a write may have been sent different amounts of each dictionary, for example when one of them subscribed more recently. Before the shared write, the server sends each subscriber that is behind the values it lacks: one message per dictionary, or several when the values would exceed the maximum message size. After that, subscribers that share a write stay level, so they need no more of these messages until another subscriber joins them.

• -DBarrageMessageProducer.dedupIdenticalSubscribers: When true (the default), identical full-table subscribers share one write of each update. When false, every subscriber gets a write of its own.
• -DBarrageMessageProducer.parallelCopyMinBytes: The amount of copying, in bytes, at which the server copies a message to the subscribers that share a write in parallel: the message's size multiplied by one less than the number of those subscribers. Default: 4194304 (4 MiB). The server copies smaller messages to one subscriber after another. With BarrageMessageProducer.propagationThreads of 1 or less, it copies every message to one subscriber after another.

Compress Barrage data

By default, the server sends Barrage snapshots and subscriptions uncompressed. To let it compress a table's data, call the table's with_transport_compression method with the names of the compression types the server may use, in order of preference, such as the built-in "zstd", "gzip", "snappy", and "deflate". A gRPC client may list the compression types it can decode in its grpc-accept-encoding request header. When a client fetches the table, the server uses the first type in the table's list that the client also lists, and sends uncompressed data when none match or the client lists none.

with_transport_compression stores the list in the table's TransportCompression attribute. Operations whose result holds only the table's values keep the attribute: filtering, sorting, reversing, flattening, dropping or renaming columns, partition_by constituents, first_by and last_by. Operations that compute new values or bring in another table's data drop it, including view, update_view, joins, rollups and trees; so do operations such as head, select and update. Their results have no attribute set, so they are sent uncompressed unless the server has a server-wide list, described next. For a rollup or tree table, call with_transport_compression on the rollup or tree table rather than on its source.

To compress tables without changing the queries that create them, set a server-wide list of compression types, most preferred first, with the TransportCompression.defaultTypes property, for example -DTransportCompression.defaultTypes=zstd,snappy. The server uses the server-wide list for every table that has no TransportCompression attribute set. Without the property, only tables with their own TransportCompression attribute are compressed.

With a server-wide list, compression is opt-out rather than opt-in. A table with its own TransportCompression attribute uses its own list instead of the server-wide list. To exclude a table, call with_transport_compression([]). This sets the attribute to an empty list, which means "never compress". To undo with_transport_compression so that the server-wide list applies again, call reset_transport_compression, which removes the attribute. Passing None to with_transport_compression does the same. Operations keep or drop an empty list just as they do any other list. A table made by an operation that drops the attribute, such as a join or view, has no attribute set, so the server-wide list applies to it, even when its source was excluded.

These are the compression types each client lists by default. A client that lists none always receives uncompressed data.

Remote tables between servers are a separate case. A Deephaven server that fetches a table from another server, with barrage_session or by resolving a dh:// URI, connects through the Java client that runs inside the server, even though the query that fetches the table is written in Python. So it lists the same compression types as the Java client, not the ones pydeephaven lists.

Choose the list based on what the server's CPU can afford. The following figures come from TransportCompressionBenchmark, a JMH benchmark in the Deephaven Core repository, run on a one-million-row table of mixed trade data. They were measured with JDK 21 on one thread of an Apple M1 Max and of an Intel Core i9-14900KS, and the speeds are megabytes of uncompressed data per second. Absolute speeds depend on the hardware and the data, so use them to compare the compression types rather than to predict throughput:

zstd suits most tables. gzip compresses slightly smaller but is about 20 times slower to compress, so list it last, and only when clients that cannot decode zstd, such as pydeephaven, should also receive compressed data. deflate is the algorithm of gzip in the zlib format: it compresses to the same size at the same speed, and decompresses slightly slower. Every client that decodes deflate also decodes gzip, so there is little reason to list deflate ahead of gzip. snappy compresses fastest but saves the least, and it saves nothing on data that is already close to random.

To narrow the compression types a client lists, use the setting for that client. An empty list asks for uncompressed data.

• A Java client: ClientConfig.builder().acceptCompression(...).
• A remote table fetched with barrage_session: its accept_compression argument.
• A remote table fetched by resolving a dh:// URI: the -DBarrageTableResolver.acceptCompression property on the fetching server, for example -DBarrageTableResolver.acceptCompression=zstd.

The Java client can also compress the data it uploads with DoPut, including the rows it adds to input tables: set ClientConfig.builder().requestCompression(...), for example to "zstd". Uploads are uncompressed by default. A server that cannot decode the type fails the upload, so choose gzip, which every Deephaven server decodes unless gzip is turned off as described below, when the server might not support the others.

To add a compression type beyond zstd, gzip, snappy, and deflate, such as lz4, write a Java class that implements io.deephaven.grpc.compression.CompressionCodecProvider and returns an io.grpc.Codec for the type, named by the codec's getMessageEncoding(). Register the class as a Java ServiceLoader service, for example with @AutoService(CompressionCodecProvider.class), and put its jar on the server's classpath. Tables can then list the type by its name, and the server uses it for clients that list it. A Java client lists it by default when the same jar is on its classpath; other clients list only the types in the table above. A class whose codec has the name of a built-in type, such as zstd, replaces the built-in codec wherever the jar is on the classpath, for example to compress at a different level.

To turn off a compression type on a server, set the type's TransportCompression.<type>.enabled property to false, for example -DTransportCompression.gzip.enabled=false, or put the same setting in the server's configuration file. Every type is enabled unless the type's property disables it. The server never compresses data with a disabled codec and never accepts data compressed with the codec: a table that lists the type is sent with the next type on the table's list, and a client request compressed with the type fails. The property applies to added types too.

Additional Barrage configuration

The following properties control other aspects of Barrage behavior:

Update interval

• -Dbarrage.minUpdateInterval: The default update interval, in milliseconds, for subscriptions that do not request one. Default: 1000 (1 second). When the table ticks, the server waits at least this long after the last update it sent before sending the next; adding or changing a subscription, and each step of a growing subscription, sends immediately. Lower values reduce latency but increase CPU and network usage.

Message batching

• -DBarrageMessageWriterImpl.batchSize: Maximum rows per Arrow record batch, and values per dictionary batch, when the client does not specify a batch size. Default: Integer.MAX_VALUE. Reduce this if clients have trouble processing very large batches.
• -DBarrageMessageWriterImpl.initialBatchSize: The starting row count for the first record batch of each message (capped by batchSize, and retried smaller if the batch is too large). Default: 4096. Later batches in the same message are sized from the observed bytes per row, with a margin, to stay under the maximum message size.
• -DBarrageMessageWriterImpl.maxOutboundMessageSize: Maximum size (in bytes) for outbound messages. Default: 104857600 (100 MB). This matches the default incoming message limit for Java clients. The server splits record batches, and the dictionary batches of dictionary-encoded columns, to stay under it.

Troubleshooting

Use the metrics tables described above to diagnose common Barrage issues.

High SnapshotNanos

If SnapshotNanos is consistently high:

• The source table may be very large. Consider using viewports or filtering data before subscription. For tree and rollup tables, every update is sent as a snapshot, so SnapshotNanos reflects ordinary update cost.
• The update graph may be holding a lock. Check for long-running operations blocking the cycle.
• Make sure subscription growth has not been disabled, and consider smaller chunk sizes (see Control subscription snapshot size).

High WriteNanos or WriteBytes

If WriteNanos is high or WriteBytes is large:

• WriteNanos measures encoding each message and handing it to gRPC, not network transmission. Large messages take longer to encode.
• Consider subscribing to fewer columns or using viewports to reduce data volume.
• A longer update interval sends fewer, larger messages. It reduces total bytes when the same rows change repeatedly between updates, but each message, and each WriteBytes value, gets larger.

High PropagateNanos

If PropagateNanos is consistently high:

• PropagateNanos includes writing each subscriber's message, so check WriteNanos first.

• Many subscribers may be connected to the same table. Their writes overlap only as far as BarrageMessageProducer.propagationThreads and idle processors allow: check that the property is above 1 and not well below the number of subscribers (see Write to subscribers in parallel), and whether the server's processors are already busy. Consider load balancing across multiple server instances.

• Subscribers whose options or viewports differ each need a write of their own. Full-table subscribers with matching options share one (see Share one write among identical subscribers).

• The server may be under memory pressure. Check JVM heap usage and garbage collection metrics.

Subscription errors

Common subscription issues:

• Ticket not found (for example, ticket '<id>' not found for a shared ticket): The ticket ID is wrong, or the published table was released or the session that published it closed. A recently released table may instead fail with a "dependency released" error. Keep the publishing session open, and keep the published table from being released, for as long as others need the ticket.
• Authentication failures: Verify that the auth_type and auth_token match the server configuration.
• Connection refused: Ensure the server is running and the host/port are correct. Check firewall rules.

Memory issues

If the server experiences out-of-memory errors during subscriptions:

• Make sure subscription growth has not been disabled (-DBarrageMessageProducer.subscriptionGrowthEnabled defaults to true).
• Lower maxSnapshotCellCount or targetSnapshotPercentage to reduce peak memory usage.
• Monitor the metrics tables to identify which tables consume the most resources.

Related documentation

• Interpret Barrage metrics
• What is Barrage?
• Capture Python client tables
• Periodic Update Graph configuration