Performance tables cheat sheet

Deephaven makes several performance tables available. They track metrics about running queries and server state. The tables fall into three groups: performance log tables of raw data, derived tables built from those logs, and Barrage performance tables.

The performance log tables hold the raw data and suit custom performance statistics. They record queries and update cycles across the whole server. By default, the query performance logs leave out an operation that takes less than a millisecond and performs no data reads or metadata operations, and the update performance log combines very short updates into one Aggregated Small Updates row per interval. The derived tables suit dashboards: the per-query tables filter to one evaluation number, and the tree tables organize every query or operation as a tree.

For what the performance log and derived tables contain and how to use them, see How to use Deephaven's performance tables. For what the Barrage tables record, see Interpret Barrage metrics.

Note

The word "log" in this context refers to the append-only nature of these tables, not to log files.

Performance log tables

The log-table methods are static methods of TableLoggers, which the Groovy console statically imports, so you can call them without a class name. Each returns one of the raw performance log tables:

MethodWhat it records
updatePerformanceLogUpdate performance data for each operation that processes updates, over each interval
updatePerformanceAncestorsLogThe ancestors of each operation in the update graph, for tracing an operation to its sources
queryOperationPerformanceLogPerformance data for the individual operations within each query
queryPerformanceLogPerformance data for each query as a whole
processMetricsLogMetrics collected for the current Deephaven engine process
processInfoLogA static table of information about the current Deephaven engine process
serverStateLogMemory use, update graph, and garbage collection statistics, sampled periodically

Derived tables

Several derived tables are views over the log tables. Their names typically drop the Log suffix. The derived tables add computed columns, such as:

  • TimeSecs, the elapsed time in seconds.
  • QueryMemUsedPct, memory use as a fraction of the maximum heap size.

The derived-table methods are static methods of PerformanceQueries. The per-query methods take that query's evaluation number, the unique ID Deephaven assigns to each query. Find the evaluation number in the EvaluationNumber column of the Query Performance Log. The other methods take no argument. The main derived tables are:

MethodWhat it records
queryPerformancePerformance data for one query, given its evaluation number
queryOperationPerformancePerformance data for each operation in one query, such as each where or update call, given the query's evaluation number
queryUpdatePerformanceUpdate performance data for the operations in one query, given its evaluation number
queryUpdatePerformanceMapA map of summary tables for one query's updates, such as the worst intervals and summary statistics, given its evaluation number
serverStateThe derived view of the Server State Log: memory use and availability as fractions of the maximum heap, the fraction of time spent in garbage collection, and update graph cycle statistics
queryPerformanceAsTreeTableEvery query, as a tree table
queryOperationPerformanceAsTreeTableEvery query and its operations, merged into one tree table

The Groovy console also defines a queryUpdatePerformanceSet closure. It takes an evaluation number and assigns each table in the queryUpdatePerformanceMap result to a script variable named by its key.

The following example creates the derived tables other than queryUpdatePerformanceMap, using 1 as the evaluation number:

Barrage performance

Barrage is Deephaven's interprocess communication (IPC) protocol for streaming tables between processes. Deephaven records two sets of statistics for it:

  • Subscription metrics, for subscriptions, including a snapshot that BarrageTableResolver.snapshot takes through a subscription.
  • Snapshot metrics, for snapshot requests such as an Arrow Flight DoGet.

Which process records a statistic depends on its side of the connection:

  • Sender statistics: the process that publishes the table records them.
  • Receiver statistics: a process that subscribes to or snapshots a table through the Java Barrage client records them, in its own subscription metrics table. This includes the Java client and a Deephaven server that uses a BarrageSession, a URI, or BarrageTableResolver. Clients such as pydeephaven and the JavaScript client record no receiver statistics.

For details, see Barrage metrics for performance monitoring.

Access the subscription and snapshot metrics tables as follows:

Identify a table

The TableId and TableKey columns identify what each row describes:

  • For sender statistics, TableId is the source table's System.identityHashCode, written in hexadecimal.
  • For receiver statistics, TableId is the identity hash code of the subscriber's local copy of the table.
  • Subscriptions to one table at different update intervals are tracked separately but share the same TableId and TableKey.
  • For tree table and rollup table subscriptions, TableId is the identity hash code of the subscription rather than of a table.
  • For every table, TableKey defaults to the table's description, which Table.getDescription returns. To override it, set the Table.BARRAGE_PERFORMANCE_KEY_ATTRIBUTE table attribute with withAttributes. For a tree or rollup table subscription, the key defaults to the tree or rollup table's description; set the attribute on the tree or rollup table itself.

withAttributes returns a new table. Publish attrTable, not t, to record metrics under MyTableKey.

Note

The web client often applies transformations, such as sorts and filters, to a table before it subscribes to it. Most transformations drop the Table.BARRAGE_PERFORMANCE_KEY_ATTRIBUTE attribute, which has these effects:

  • Deephaven records the transformed table only when BarragePerformanceLog.enableAll is true, which is the default.
  • The transformed table's TableKey is its own description.
  • If a non-web client also subscribes to the original table, the metrics table shows rows for both tables, with different TableId values.

Barrage subscription metrics

Deephaven aggregates subscription statistics over each flush interval, which BarragePerformanceLog.cycleDurationMillis sets, and records them as percentiles. Each row holds one statistic for one tracked source over one flush interval. A tracked source is one of the following:

A statistic with no samples in an interval has no row. The subscription metrics table has the following columns:

ColumnDescription
TableIdThe identity hash code of the tracked source's table, subscription, or local copy, as described under Identify a table
TableKeyThe tracked source's key, which defaults to the table's description, as described under Identify a table
StatTypeThe statistic the row records, from the next table
TimeThe time at which the flush interval ended
CountThe number of samples in the flush interval
Pct50, Pct75, Pct90, Pct95, Pct99The 50th, 75th, 90th, 95th, and 99th percentiles of the samples
MaxThe largest sample

The following table lists the StatType values. Its Sender / Receiver column says which side records each one. For where in an update's life each statistic is recorded, see Interpret Barrage metrics.

StatTypeSender / ReceiverDescription
EnqueueNanosSenderThe time it took to record changes that occurred during a single update graph cycle
AggregateNanosSenderThe time it took to aggregate pending updates into a message; recorded for every compaction and for every range a propagation packages, even a range of one update
PropagateNanosSenderThe elapsed time of one propagation phase: writing one message to the subscribers it goes to
SnapshotNanosSenderThe time it took to take one snapshot step for a new or changed subscription; for tree table and rollup table subscriptions, every update is sent as a snapshot and records a value
UpdateJobNanosSenderThe time it took to run one full cycle of the off-thread propagation logic
WriteNanosSenderThe elapsed time of one write of the update, to one subscriber or to every subscriber that shares an update
WriteBytesSenderThe size in bytes of one write of the update, every message in it including dictionary batches; recorded once for each subscriber it was sent to
PendingDeltaCountSenderHow many pending updates the server was holding, un-propagated, after recording an update graph cycle's changes; a compacted update counts once, even though it stands for every cycle it coalesced
PendingDeltaBytesSenderApproximate heap footprint, in bytes, of the chunk storage those pending updates own
DeserializationNanosReceiverThe time it took to read and deserialize one incoming gRPC message; an update split across several messages records one value per message
ProcessUpdateNanosReceiverThe time it took to apply a single message (update or snapshot): during an update graph cycle for a subscription, or as the message arrives for a snapshot
RefreshNanosReceiverThe time it took to apply all queued messages during a single update graph cycle; recorded only for subscriptions, once per cycle, even when nothing is queued

Barrage snapshot metrics

Deephaven records snapshot metrics for snapshots requested with a Barrage snapshot request or an Arrow Flight DoGet. It records one row per snapshot request for a refreshing table. For a static table, Deephaven sends the snapshot in chunks and records one row per chunk, all with the same RequestTime. Besides TableId and TableKey, the snapshot metrics table has the following columns:

ColumnDescription
RequestTimeThe time at which the server received the snapshot request
QueueNanosThe time from receiving the request until work on it began, including waiting for the requested table to be ready and for a thread
SnapshotNanosThe time it took to construct a consistent snapshot of the source table
WriteNanosThe time it took to write the snapshot or chunk
WriteBytesThe size in bytes of the snapshot or chunk, every message in it including dictionary batches

For a chunked static-table snapshot, SnapshotNanos in each chunk's row is the total snapshot time so far, while WriteNanos and WriteBytes cover only that chunk.

Configuration

The following server-side flags change the behavior of Barrage metrics:

  • -DBarragePerformanceLog.enableAll: whether to record metrics for tables that do not have an explicit TableKey (default: true)
  • -DBarragePerformanceLog.cycleDurationMillis: the interval at which to flush aggregated statistics (default: 60000, once per minute)