Skip to main content
Version: Java (Groovy)

Performance tables cheat sheet

Deephaven makes several performance log tables available.


The word "log" in this context refers to the append-only nature of these tables, not to log files in the traditional sense.

These tables track various metrics across Deephaven, and in normal circumstances will be populated with raw performance data pertaining to your running queries and server state.

upl = updatePerformanceLog()
qopl = queryOperationPerformanceLog()
qpl = queryPerformanceLog()
pml = processMetricsLog()
pil = processInfoLog()
ssl = serverStateLog()

Several derived tables are defined on top of them, typically by removing the suffix Log from the name; these derived tables accumulate data and present it in more user-friendly column types (e.g., time units in fractional seconds instead of nanoseconds for CPU time metrics), and as such are more useful for dashboards, where the Log tables are more useful as sources of raw data for user-defined computation of performance statistics. For the derived tables in the list below that are related to query performance, the functions that create them take one query evaluation number id as a parameter to show results for a particular query (where the Log tables show data for all queries).

import io.deephaven.engine.table.impl.util.PerformanceQueries

qup = PerformanceQueries.queryUpdatePerformance(1)
qp = PerformanceQueries.queryPerformance(1)
qop = PerformanceQueries.queryOperationPerformance(1)
ss = PerformanceQueries.serverState().tail(32)

Barrage Performance

Barrage is the name of our IPC table transport. There are two sets of statistics. The subscription metrics are recorded for ticking subscriptions. The snapshot metrics are recorded for one-off requests such as via an Arrow Flight DoGet.

Tables are identified by their TableId and TableKey. The TableId is determined by the source table's System.identityHashCode(). The TableKey defaults to Table#getDescription() but can be overridden by setting a table attribute.

table = emptyTable(0)
table.setAttribute(Table.BARRAGE_PERFORMANCE_KEY_ATTRIBUTE, "MyTableKey")

The web client applies transformations to every table that it subscribes to. If a table is also subscribed by a non-web client then statistics for the original table and the transformed table will both appear in the metrics table. Their TableId will differ. Most transformations clear the TableKey attribute, such as when a column is sorted or filtered by the user through the GUI.

Here are server side flags that change the behavior of barrage metrics.

  • -DBarragePerformanceLog.enableAll: record metrics for tables that do not have an explicit TableKey (default: true)
  • -DBarragePerformanceLog.cycleDurationMillis: the interval to flush aggregated statistics (default: 60000 - once per minute)

You can access these tables as follows:

import io.deephaven.extensions.barrage.BarragePerformanceLog

subs = BarragePerformanceLog.getInstance().getSubscriptionTable()
snaps = BarragePerformanceLog.getInstance().getSnapshotTable()

Barrage Subscription Metrics

Subscription statistics are presented in percentiles bucketed over a time period.

Here are the various metrics that are recorded by the deephaven-core server:

Stat TypeSender / ReceiverDescription
EnqueueMillisSenderThe time it took to record changes that occurred during a single update graph cycle
AggregateMillisSenderThe time it took to aggregate multiple updates within the same interval
PropagateMillisSenderThe time it took to deliver an aggregated message to all subscribers
SnapshotMillisSenderThe time it took to snapshot data for a new or changed subscription
UpdateJobMillisSenderThe time it took to run one full cycle of the off-thread propagation logic
WriteMillisSenderThe time it took to write the update to a single subscriber
WriteMegabitsSenderThe payload size of the update in megabits
DeserializationMillisReceiverThe time it took to read and deserialize the update from the wire
ProcessUpdateMillisReceiverThe time it took to apply a single update during the update graph cycle
RefreshMillisReceiverThe time it took to apply all queued updates during a single udpate graph cycle

Barrage Snapshot Metrics

Snapshot statistics are presented once per request.

Stat TypeDescription
QueueMillisThe time it took waiting for a thread to process the request
SnapshotMillisThe time it took to construct a consistent snapshot of the source table
WriteMillisThe time it took to write the snapshot
WriteMegabitsThe payload size of the snapshot in megabits