How to use Deephaven's performance tables for system monitoring

This guide shows how to use Deephaven's internal performance tables.

Deephaven provides several ways to monitor query performance and resolve issues as they arise. Many of them are available directly in the user interface. For example, to see how long a query has been running, open the History Panel and look at the command details. You can also hover over the console command, and the tooltip shows the formatted duration of the query.

The most common performance issues are slow initialization, a query failing to update quickly, and a sluggish user interface. If a query seems to be taking too long or throws an error, Deephaven's performance tables are a great place to start. Performance tables are a powerful tool for analyzing performance and debugging issues. They show where the time goes, so you can troubleshoot queries on your own.

Which table to open depends on the problem:

The example below shows how to find slow operations in the Query Operation Performance Tree Table, then opens the Query Operation Performance Log and the Update Performance Log directly for a small query.

Evaluation numbers

Several perfmon functions take an evaluation number as their eval_number argument. An evaluation number uniquely identifies a query. A query is one request to the server, such as:

  • A command you run in the console by pressing Enter.
  • A sort, filter, or custom column you apply in the UI.
  • A request from a client API application.

A subquery that runs as part of another query gets its own evaluation number. In the subquery's row of the performance tables, the ParentEvaluationNumber column holds the evaluation number of the query that started it. The individual operations within a query, such as single method calls, share the query's evaluation number, and each has its own OperationNumber in the Query Operation Performance Log.

You can find evaluation numbers in the EvaluationNumber column of the tables that query_performance_log and query_operation_performance_log return. The Query Performance Log and Query Operation Performance Log sections describe these tables.

Available performance tables

Deephaven's perfmon module contains functions that create and open performance tables in the Deephaven IDE.

Example: find slow operations

The tables available in perfmon contain a lot of information, and it can seem overwhelming without some context. The following examples show two ways to find the operations that slow a query down. The first is quicker, but the performance tree tables show only the work each query does when it starts. The second works from the raw logs, so it also covers the update cycles of ticking tables, and you can add your own columns.

In the performance tree tables

The performance tree tables are useful tools for pinpointing the cause of unexpected slow performance in your queries. They are also the quickest way to find slow operations, because you don't need to add any columns. Open the Query Operation Performance Tree Table with query_operation_performance_tree_table and sort it in descending order by the UsageNanos column so that the slowest queries appear at the top. The screenshots below come from a session that ran the performance table query below, so your rows will differ:

The Query Operation Performance Tree Table, sorted by the UsageNanos column

Expand a parent row to see the sub-operations that make up its total time and find the ones that slow the query down:

The sorted Query Operation Performance Tree Table, expanded to show sub-operations

In the performance logs

To add your own columns, or to examine a ticking query, work from the raw logs. Let's explore an example that focuses on two of the tables: the Query Operation Performance Log and the Update Performance Log.

Below is the entire query.

Performance table query

Some key pieces of the above query are worth noting:

  • The functions fast and slow compute the same result. slow is slow because it waits a quarter of a second before it returns the result.
  • qopl_update adds a DurationNanos column, the time each operation took when its query started.
  • upl_update adds two columns, IntervalDurationNanos and Ratio. Ratio is the fraction of the interval spent performing a particular operation.
  • sort_descending orders upl_update by Ratio, so the operations that take the most time appear at the top.

Two of the derived views already have these columns. Query Operation Performance has DurationNanos, and Query Update Performance has IntervalDurationNanos and Ratio. Each view shows only one query, though. Adding the columns to the raw logs covers every query at once.

For the static tables t, t2, and t3, open qopl_update. Each row is one operation that ran when its query started, and DurationNanos shows how long it took. The Update([Y]) row for t3 takes about 25 seconds, because slow sleeps for a quarter of a second on each of its 100 rows. Update([X]) for t2 is fast. The first time the query runs, most of that row's time is formula compilation.

For the ticking tables td, td2, and td3, open upl_update:

The above upl_update table

In the image above, Update([Y]), which calls slow, takes a whopping 25% of each full interval while td ticks:

  • The ticking table td adds one row every second.
  • For each new row, the engine sits and waits a quarter of a second for slow to return.

Clearly, slow is the bottleneck in this code. Real slowdowns are rarely this obvious, but sorting by Ratio puts the operations worth optimizing at the top.

Performance table reference

This section describes the raw performance logs first, then the derived views and tree tables built from them. Each table's section lists its columns in a collapsible block.

Raw performance logs

Query Performance Log

The Query Performance Log (QPL) contains query-level performance data. Each row describes one query, so a Deephaven instance that runs many queries has many rows. For the individual operations within each query, see the Query Operation Performance Log. Create the QPL with query_performance_log.

The most significant columns in this table are:

  • UsageNanos — Time spent running this query in nanoseconds, summed over all threads.
  • FreeMemory — The free memory at the end of this query.
  • TotalMemory — The total size of the JVM heap memory at the end of this query.
  • FreeMemoryChange — The change in free memory during this query. Other threads can change free memory at the same time, so pair it with AllocatedBytes.
Query Performance Log columns
Column NameData TypeDescription
EvaluationNumberlongAn increasing integral value for requests made to the worker.
ParentEvaluationNumberlongWhen this query is a subquery, the evaluation number of the query that started it.
SessionIdStringA unique identifier for the user session performing the query.
DescriptionStringA textual description of the query, generally including the name and some of the arguments.
StartTimeInstantThe time this query began.
EndTimeInstantThe time this query finished.
UsageNanoslongTime spent running this query, in nanoseconds, summed over every thread that worked on it. Parallel work can make it larger than the wall-clock time from StartTime to EndTime.
CpuNanoslongHow long, in nanoseconds, this query used the CPU.
UserCpuNanoslongHow long, in nanoseconds, this query used the CPU in user-space. Subtract UserCpuNanos from CpuNanos to get system time.
FreeMemorylongThe free memory at the end of this query.
TotalMemorylongThe total size of the JVM heap memory at the end of this query.
FreeMemoryChangelongThe change in free memory during this query. Other threads can change free memory at the same time, so this value alone doesn't show what the query used. To find operations that use significant heap, combine this column with profiling and the AllocatedBytes column, which also counts memory that became eligible for garbage collection during or after the query.
TotalMemoryChangelongThe change in total JVM heap memory during this query.
CollectionslongThe number of garbage collections while executing this query.
CollectionTimeNanoslongThe number of nanoseconds spent in the garbage collector while executing this query.
AllocatedByteslongThe number of bytes allocated by this query.
PoolAllocatedByteslongThe number of bytes allocated from pools by this query.
DataReadNanoslongHow long, in nanoseconds, this query spent blocking while reading data.
DataReadCountlongThe number of blocking data reads that occurred during this query.
DataReadByteslongHow many bytes this query read while performing blocking operations accounted for in DataReadNanos and DataReadCount.
MetadataOperationNanoslongHow long, in nanoseconds, this query spent on blocking metadata operations.
MetadataOperationCountlongThe number of blocking metadata operations that occurred during this query.
WasInterruptedbooleanWas this query interrupted?
ExceptionStringIf this query produced an exception, the exception's message. Otherwise null.
AuthContextStringThe authentication context used for executing this query.
WorkerHeapSizelongThe maximum heap size of the worker, in bytes.

Query Operation Performance Log

The Query Operation Performance Log (QOPL) contains performance data for the individual operations within each query. It has one row for each operation. By default, it omits operations that finish very quickly without reading data or performing metadata operations. Create the QOPL with query_operation_performance_log.

The QOPL and the Update Performance Log (UPL) record different work:

  • The QOPL records the work an operation does when its query runs, such as a table operation's initialization.
  • The UPL summarizes, over an interval, the work operations do afterward to process updates to ticking tables.

The QOPL describes each operation in its Description column, while the UPL uses EntryDescription. To match a UPL row to the QOPL row for the same operation, use the EvaluationNumber and OperationNumber columns, which both tables share. The QOPL always keeps the row for an operation that has UPL rows, even when that operation finished quickly, so this match is exact.

The most significant columns in this table are:

  • StartTime — The time at which this operation started.
  • EndTime — The time at which this operation ended.
  • OperationNumber — A monotonically increasing sequence number for each operation in a query.
  • FreeMemory — The free memory at the end of this operation.
  • TotalMemory — The total size of the JVM heap memory at the end of this operation.
  • FreeMemoryChange — The change in free memory during this operation.
Query Operation Performance Log columns
Column NameData TypeDescription
EvaluationNumberlongAn increasing integral value for requests made to the worker.
ParentEvaluationNumberlongWhen this query is a subquery, the evaluation number of the query that started it.
OperationNumberintAn identifier for an operation within an evaluation.
ParentOperationNumberintThe OperationNumber of the operation that encloses this one in the same query. Null when the operation is not nested inside another operation.
DepthintThe operation's nesting level: 0 for an operation directly under its query, plus 1 for each enclosing operation.
SessionIdStringA unique identifier for the user session performing the operation.
DescriptionStringA textual description of the operation, generally including the name and some of the arguments.
CallerLineStringThe file and line number where the operation was initiated, if available.
IsCompilationbooleanDoes this represent a formula compilation operation?
StartTimeInstantThe time this operation began.
EndTimeInstantThe time this operation finished.
UsageNanoslongTime spent running this operation, in nanoseconds, summed over every thread that worked on it. Parallel work can make it larger than the wall-clock time from StartTime to EndTime.
CpuNanoslongHow long, in nanoseconds, this operation used the CPU.
UserCpuNanoslongHow long, in nanoseconds, this operation used the CPU in user-space. Subtract UserCpuNanos from CpuNanos to get system time.
FreeMemorylongThe free memory at the end of this operation.
TotalMemorylongThe total size of the JVM heap memory at the end of this operation.
FreeMemoryChangelongThe change in free memory during this operation.
TotalMemoryChangelongThe change in total JVM heap memory during this operation.
CollectionslongThe number of garbage collections while executing this operation.
CollectionTimeNanoslongThe number of nanoseconds spent in the garbage collector while executing this operation.
AllocatedByteslongThe number of bytes allocated by this operation.
PoolAllocatedByteslongThe number of bytes allocated from pools by this operation.
InputSizeLonglongThe number of rows in the input for this operation.
DataReadNanoslongHow long, in nanoseconds, this operation spent blocking while reading data.
DataReadCountlongThe number of blocking data reads that occurred during this operation.
DataReadByteslongHow many bytes this operation read while performing blocking operations accounted for in DataReadNanos and DataReadCount.
MetadataOperationNanoslongHow long, in nanoseconds, this operation spent on blocking metadata operations.
MetadataOperationCountlongThe number of blocking metadata operations that occurred during this operation.
WasInterruptedbooleanWas this operation interrupted?
AuthContextStringThe authentication context used for executing this operation.
WorkerHeapSizelongThe maximum heap size of the worker, in bytes.

Update Performance Log

The update graph is the engine component that updates ticking tables. The Update Performance Log (UPL) describes the time and resources the update graph uses to keep each table operation's results up to date. Create the UPL with update_performance_log. The UPL reports in intervals:

  • Each interval lasts one minute by default. The UpdatePerformanceTracker.reportIntervalMillis configuration property sets its length in milliseconds.
  • At the end of each interval, each operation that processed updates gets its own row, except for small operations.
  • Small operations, which used very little time in the interval and did not read data or perform metadata operations, share a single Aggregated Small Updates row.
  • A row named UpdatePerformanceTracker Flush records the time the tracker spends writing the interval's rows.

The most significant columns in this table are:

  • EntryDescription — The operation this row describes.
  • IntervalStartTime and IntervalEndTime — The beginning and end of the logging interval.
  • UsageNanos — How long, in nanoseconds, the operation spent updating during the interval, summed over all threads.
  • CpuNanos — How long, in nanoseconds, the operation used the CPU during the interval.
  • RowsAdded, RowsRemoved, RowsModified, and RowsShifted — The rows the operation processed in the interval.
Update Performance Log columns
Column NameData TypeDescription
EntryIdlongA numeric identifier for this performance entry. Use it to identify the same operation across cycles.
EvaluationNumberlongThe evaluation number of the corresponding entry in the Query Operation Performance Log.
OperationNumberintThe operation number of the corresponding entry in the Query Operation Performance Log.
EntryDescriptionStringA textual description of the operation, generally including the name and some of the arguments.
EntryCallerLineStringThe file and line number where the operation was initiated, if available.
IntervalStartTimeInstantThe beginning of the interval this row represents.
IntervalEndTimeInstantThe end of the interval this row represents.
UsageNanoslongTime this operation spent processing updates in this interval, in nanoseconds, summed over all threads.
CpuNanoslongHow long, in nanoseconds, this operation used the CPU in this interval.
UserCpuNanoslongHow long, in nanoseconds, this operation used the CPU in user-space in this interval. Subtract UserCpuNanos from CpuNanos to get system time.
RowsAddedlongHow many rows were added in this interval.
RowsRemovedlongHow many rows were removed in this interval.
RowsModifiedlongHow many rows were modified in this interval.
RowsShiftedlongHow many rows were shifted in this interval. The count can be too high. The engine counts every row key in each shifted range, including keys with no row, so a shift over a sparse region overstates the number of rows.
InvocationCountlongHow many times this entry was executed in this interval.
MinFreeMemorylongThe minimum free heap memory sampled at the start or end of any of this operation's runs in this interval.
MaxTotalMemorylongThe maximum total heap memory sampled at the start or end of any of this operation's runs in this interval.
CollectionslongThe number of garbage collections while executing this operation in this interval.
CollectionTimeNanoslongThe number of nanoseconds spent in the garbage collector while executing this operation in this interval.
AllocatedByteslongThe number of bytes allocated by this operation in this interval.
PoolAllocatedByteslongThe number of bytes allocated from pools by this operation in this interval.
DataReadNanoslongHow long, in nanoseconds, this operation spent blocking while reading data in this interval.
DataReadCountlongThe number of blocking data reads that occurred during this operation in this interval.
DataReadByteslongHow many bytes this operation read while performing blocking operations accounted for in DataReadNanos and DataReadCount.
MetadataOperationNanoslongHow long, in nanoseconds, this operation spent on blocking metadata operations in this interval.
MetadataOperationCountlongThe number of blocking metadata operations that occurred during this operation in this interval.
AuthContextStringThe authentication context used for executing this operation.
UpdateGraphStringThe name of the update graph for this entry.
WorkerHeapSizelongThe maximum heap size of the worker, in bytes.

Update Performance Ancestors Log

The Update Performance Ancestors Log contains the UpdateGraph, EntryId, and EntryDescription columns from the Update Performance Log and an additional Ancestors column. Each row describes one operation, which is a node in the update graph. The Ancestors column lists the operations whose updates feed into it, so you can trace a slow operation back to its sources. An operation that gains new ancestors later, such as a join, gets another row, and the functions below combine its rows. Create the Update Performance Ancestors Log with update_performance_ancestors_log.

Update Performance Ancestors Log columns
Column NameData TypeDescription
UpdateGraphStringThe name of the update graph for this entry.
EntryIdlongA numeric identifier for this performance entry. Use it to identify the same operation across cycles.
EntryDescriptionStringA textual description of the operation, generally including the name and some of the arguments.
AncestorsLongVectorA vector of ancestor EntryId values.

To draw the ancestor graph of one or more entries, pass their EntryId values, the Update Performance Log, and the Update Performance Ancestors Log to one of these perfmon functions:

Note

You must have Graphviz installed to produce an ancestors image with ancestor_image or ancestor_svg. The published Deephaven Docker images don't include Graphviz, so install it in a custom image to use these functions. ancestor_dot doesn't need Graphviz.

In the following example, entry_id is the EntryId of the operation you want to examine, and upl and upal are the tables created in Available performance tables:

Process Metrics Log

This table contains metrics collected for the current Deephaven engine process. Create it with process_metrics_log.

Process Metrics Log columns
Column NameData TypeDescription
ProcessUniqueIdStringThe process's unique identifier String.
TimestampInstantThe time this metric sample was recorded.
NameStringThe name of the metric being reported.
IntervalStringThe length of the interval being reported, such as 1s or 15m.
TypeStringThe metric type, such as state or counter.
NlongThe number of samples.
SumlongThe sum of all samples in this interval.
LastlongThe last sample in this interval.
MinlongThe minimum sample in this interval.
MaxlongThe maximum sample in this interval.
AvglongThe mean of all samples in this interval.
Sum2longThe sum of the squares of all samples in this interval.
StdevlongThe standard deviation of all samples in this interval.

Process Info Log

This static table contains process information for the current Deephaven engine process. Create it with process_info_log.

Process Info Log columns
Column NameData TypeDescription
IdStringAn identifier string for this process.
TypeStringThe category of the property described in this row.
KeyStringA key describing the process or property.
ValueStringThe value of the property or process described by the Key.

Server State Log

This table contains JVM information about memory use, garbage collection, and the update graph, sampled periodically. Columns with UGP in their names describe the update graph's refresh cycles. A cycle is on budget when it finishes within the update graph's target cycle duration. Create the Server State Log with server_state_log.

Server State Log columns
Column NameData TypeDescription
IntervalStartTimeInstantThe start time of the interval.
IntervalDurationMicrosintThe number of microseconds in the interval.
TotalMemoryMiBintThe JVM heap memory currently reserved, in MiB.
FreeMemoryMiBintThe free JVM heap memory, in MiB.
IntervalCollectionsshortThe number of garbage collection events in the interval.
IntervalCollectionTimeMicrosintThe number of microseconds (approximately) spent collecting garbage in the interval.
IntervalUGPCyclesOnBudgetshortThe number of update graph cycles in the interval that finished within the target cycle duration.
IntervalUGPCyclesTimeMicrosint[]An array of update graph cycle durations in the interval, in microseconds.
IntervalUGPCyclesSafePointsshortThe number of JVM safepoint pauses during update graph cycles in the interval.
IntervalUGPCyclesSafePointTimeMicrosintThe number of microseconds spent in JVM safepoint pauses during update graph cycles in the interval.

Derived views

The perfmon functions in this section build views on top of the raw logs. They add computed columns, such as durations in seconds and memory use as a fraction of the heap.

Three of the views show one query:

To use them, find the query in the Query Performance Log, read its EvaluationNumber, and pass that number to the function. In the examples below, n is that evaluation number.

Server State takes no argument.

Query Performance

query_performance takes an evaluation number and returns that query's performance data from the Query Performance Log, including how long the query took to run and how its memory use changed.

Query Performance columns
Column NameData TypeDescription
EvaluationNumberlongAn increasing integral value for requests made to the worker.
ParentEvaluationNumberlongWhen this query is a subquery, the evaluation number of the query that started it.
QueryMemUsedlongThe JVM heap memory in use at the end of the query, in bytes (TotalMemory - FreeMemory). This is the whole heap, not only this query's memory.
QueryMemFreelongThe heap memory remaining before the max heap size is reached, in bytes.
QueryMemUsedPctdoubleQueryMemUsed as a fraction of the max heap size (WorkerHeapSize).
EndTimeInstantThe time this query finished.
TimeSecsdoubleThe time this query took in seconds.
NetMemoryChangelongFreeMemoryChange - TotalMemoryChange: the change in used heap memory during this query, with the sign reversed. A negative value means heap use grew.
SessionIdStringA unique identifier for the user session performing the query.
DescriptionStringA textual description of the query, generally including the name and some of the arguments.
StartTimeInstantThe time this query began.
UsageNanoslongTime spent running this query, in nanoseconds, summed over every thread that worked on it. Parallel work can make it larger than the wall-clock time from StartTime to EndTime.
CpuNanoslongHow long, in nanoseconds, this query used the CPU.
UserCpuNanoslongHow long, in nanoseconds, this query used the CPU in user-space. Subtract UserCpuNanos from CpuNanos to get system time.
FreeMemorylongThe free memory at the end of this query.
TotalMemorylongThe total size of the JVM heap memory at the end of this query.
FreeMemoryChangelongThe change in free memory during this query.
TotalMemoryChangelongThe change in total JVM heap memory during this query.
CollectionslongThe number of garbage collections while executing this query.
CollectionTimeNanoslongThe number of nanoseconds spent in the garbage collector while executing this query.
AllocatedByteslongThe number of bytes allocated by this query.
PoolAllocatedByteslongThe number of bytes allocated from pools by this query.
DataReadNanoslongHow long, in nanoseconds, this query spent blocking while reading data.
DataReadCountlongThe number of blocking data reads that occurred during this query.
DataReadByteslongHow many bytes this query read while performing blocking operations accounted for in DataReadNanos and DataReadCount.
MetadataOperationNanoslongHow long, in nanoseconds, this query spent on blocking metadata operations.
MetadataOperationCountlongThe number of blocking metadata operations that occurred during this query.
WasInterruptedbooleanWas this query interrupted?
ExceptionStringIf this query produced an exception, the exception's message. Otherwise null.
AuthContextStringThe authentication context used for executing this query.
WorkerHeapSizelongThe maximum heap size of the worker, in bytes.
DurationNanoslongThe wall-clock time from StartTime to EndTime, in nanoseconds.

Query Operation Performance

query_operation_performance takes an evaluation number and returns that query's operations from the Query Operation Performance Log, including how long each operation took to execute and how resource consumption changed while it ran. Operations include table operations such as where, update, and natural_join, as well as internal functions.

Query Operation Performance columns
Column NameData TypeDescription
EvaluationNumberlongAn increasing integral value for requests made to the worker.
ParentEvaluationNumberlongWhen this query is a subquery, the evaluation number of the query that started it.
OperationNumberintAn identifier for an operation within an evaluation.
ParentOperationNumberintThe OperationNumber of the operation that encloses this one in the same query. Null when the operation is not nested inside another operation.
EndTimeInstantThe time this operation finished.
TimeSecsdoubleThe time this operation took in seconds.
NetMemoryChangelongFreeMemoryChange - TotalMemoryChange: the change in used heap memory during this operation, with the sign reversed. A negative value means heap use grew.
DepthintThe operation's nesting level: 0 for an operation directly under its query, plus 1 for each enclosing operation.
SessionIdStringA unique identifier for the user session performing the operation.
DescriptionStringA textual description of the operation, generally including the name and some of the arguments.
CallerLineStringThe file and line number where the operation was initiated, if available.
IsCompilationbooleanDoes this represent a formula compilation operation?
StartTimeInstantThe time this operation began.
UsageNanoslongTime spent running this operation, in nanoseconds, summed over every thread that worked on it. Parallel work can make it larger than the wall-clock time from StartTime to EndTime.
CpuNanoslongHow long, in nanoseconds, this operation used the CPU.
UserCpuNanoslongHow long, in nanoseconds, this operation used the CPU in user-space. Subtract UserCpuNanos from CpuNanos to get system time.
FreeMemorylongThe free memory at the end of this operation.
TotalMemorylongThe total size of the JVM heap memory at the end of this operation.
FreeMemoryChangelongThe change in free memory during this operation.
TotalMemoryChangelongThe change in total JVM heap memory during this operation.
CollectionslongThe number of garbage collections while executing this operation.
CollectionTimeNanoslongThe number of nanoseconds spent in the garbage collector while executing this operation.
AllocatedByteslongThe number of bytes allocated by this operation.
PoolAllocatedByteslongThe number of bytes allocated from pools by this operation.
InputSizeLonglongThe number of rows in the input for this operation.
DataReadNanoslongHow long, in nanoseconds, this operation spent blocking while reading data.
DataReadCountlongThe number of blocking data reads that occurred during this operation.
DataReadByteslongHow many bytes this operation read while performing blocking operations accounted for in DataReadNanos and DataReadCount.
MetadataOperationNanoslongHow long, in nanoseconds, this operation spent on blocking metadata operations.
MetadataOperationCountlongThe number of blocking metadata operations that occurred during this operation.
WasInterruptedbooleanWas this operation interrupted?
AuthContextStringThe authentication context used for executing this operation.
WorkerHeapSizelongThe maximum heap size of the worker, in bytes.
DurationNanoslongThe wall-clock time from StartTime to EndTime, in nanoseconds.

Query Update Performance

query_update_performance takes an evaluation number and returns that query's update performance data from the Update Performance Log.

Query Update Performance columns
Column NameData TypeDescription
EvaluationNumberlongAn increasing integral value for requests made to the worker.
OperationNumberintAn identifier for an operation within an evaluation.
RatiodoubleThe fraction of the logging interval this operation spent processing updates (UsageNanos / IntervalDurationNanos). 1 means the entire interval. Parallel work can push it above 1.
QueryMemUsedlongAn upper estimate of JVM heap memory in use during the interval, in bytes (MaxTotalMemory - MinFreeMemory). This is the whole heap, not only this operation's memory.
QueryMemUsedPctdoubleThe fraction of the max heap size (WorkerHeapSize) that QueryMemUsed represents.
IntervalEndTimeInstantThe end of the interval this row represents.
RowsPerSeclongRows added, removed, and modified (NRows) per second of the interval.
RowsPerCPUSeclongNRows per second of CPU time (CpuNanos).
RowsPerOpSeclongNRows per second the operation spent processing updates (UsageNanos).
EntryDescriptionStringA textual description of the operation, generally including the name and some of the arguments.
EntryIdlongA numeric identifier for this performance entry. Use it to identify the same operation across cycles.
EntryCallerLineStringThe file and line number where the operation was initiated, if available.
IntervalStartTimeInstantThe beginning of the interval this row represents.
UsageNanoslongTime this operation spent processing updates in this interval, in nanoseconds, summed over all threads.
CpuNanoslongHow long, in nanoseconds, this operation used the CPU in this interval.
UserCpuNanoslongHow long, in nanoseconds, this operation used the CPU in user-space in this interval. Subtract UserCpuNanos from CpuNanos to get system time.
RowsAddedlongHow many rows were added in this interval.
RowsRemovedlongHow many rows were removed in this interval.
RowsModifiedlongHow many rows were modified in this interval.
RowsShiftedlongHow many rows were shifted in this interval. The count can be too high. The engine counts every row key in each shifted range, including keys with no row, so a shift over a sparse region overstates the number of rows.
InvocationCountlongHow many times this entry was executed in this interval.
MinFreeMemorylongThe minimum free heap memory sampled at the start or end of any of this operation's runs in this interval.
MaxTotalMemorylongThe maximum total heap memory sampled at the start or end of any of this operation's runs in this interval.
CollectionslongThe number of garbage collections while executing this operation in this interval.
CollectionTimeNanoslongThe number of nanoseconds spent in the garbage collector while executing this operation in this interval.
AllocatedByteslongThe number of bytes allocated by this operation in this interval.
PoolAllocatedByteslongThe number of bytes allocated from pools by this operation in this interval.
DataReadNanoslongHow long, in nanoseconds, this operation spent blocking while reading data in this interval.
DataReadCountlongThe number of blocking data reads that occurred during this operation in this interval.
DataReadByteslongHow many bytes this operation read while performing blocking operations accounted for in DataReadNanos and DataReadCount.
MetadataOperationNanoslongHow long, in nanoseconds, this operation spent on blocking metadata operations in this interval.
MetadataOperationCountlongThe number of blocking metadata operations that occurred during this operation in this interval.
AuthContextStringThe authentication context used for executing this operation.
UpdateGraphStringThe name of the update graph for this entry.
WorkerHeapSizelongThe maximum heap size of the worker, in bytes.
IntervalDurationNanoslongThe duration of the interval in nanoseconds.
QueryMemFreelongThe heap memory remaining before the max heap size is reached, in bytes.
NRowslongThe total number of rows added, removed, and modified in the interval.

For more views of the same data, query_update_performance_map takes the same evaluation number and returns a dictionary of tables built from this data, with the keys QueryUpdatePerformance, UpdateWorst, WorstInterval, UpdateMostRecent, UpdateAggregate, and UpdateSummaryStats.

Server State

The Server State table contains basic memory, garbage collection (GC), and update graph statistics for the current engine process, sampled periodically. server_state builds it from the Server State Log. Columns that start with UGP describe the refresh cycles of the update graph.

Server State columns
Column NameData TypeDescription
IntervalStartInstantThe beginning of the interval this row represents.
IntervalSecsdoubleThe length of the interval being reported (in seconds).
IntervalEndInstantThe end of the interval this row represents.
UsedMemMiBintThe amount of used memory in MiB.
AvailMemMiBintThe amount of available memory in MiB.
MaxMemMiBintThe maximum amount of memory in MiB.
AvailMemRatiodoubleThe ratio of available memory to maximum memory.
UsedMemRatiodoubleThe ratio of used memory to maximum memory.
GcTimeRatiofloatThe ratio of GC time to total time.
UGPCycleslongThe number of update graph cycles.
UGPOnBudgetRatiofloatThe fraction of update graph cycles that finished within the target cycle duration.
UGPCycleMaxSecsdoubleThe maximum cycle time in seconds.
UGPCycleMedianSecsdoubleThe median cycle time in seconds.
UGPCycleMeanSecsdoubleThe mean cycle time in seconds.
UGPCycleP90SecsdoubleThe 90th percentile cycle time in seconds.
UGPTimeRatiofloatThe ratio of update graph cycle time to total time.
UGPSafePointTimeRatiofloatThe ratio of JVM safepoint pause time during update graph cycles to total time.

Tree tables

Query Performance Tree Table

query_performance_tree_table returns a tree table of query performance data. It shows the same information as the Query Performance Log, in tree form. Like a traditional call graph, it places every query that has a parent query into an expandable group. In the image below, a sort applied in the UI ran as the query with EvaluationNumber = 75. Expanding its row shows the subquery that did the sort, EvaluationNumber = 76:

The Query Performance Tree Table, with the row where EvaluationNumber = 75 expanded to show its subquery

To see individual sub-operations as a tree table, call query_operation_performance_tree_table.

Query Performance Tree Table columns
Column NameData TypeDescription
EvaluationNumberlongThe evaluation number of the corresponding entry in the Query Performance Log.
ParentEvaluationNumberlongThe evaluation number of this row's parent in the Query Performance Log.
SessionIdStringA unique identifier for the user session performing the query.
DescriptionStringA textual description of the query, generally including the name and some of the arguments.
StartTimeInstantThe time this query began.
EndTimeInstantThe time this query finished.
UsageNanoslongTime spent running this query, in nanoseconds, summed over every thread that worked on it. Parallel work can make it larger than the wall-clock time from StartTime to EndTime.
CpuNanoslongHow long, in nanoseconds, this query used the CPU.
UserCpuNanoslongHow long, in nanoseconds, this query used the CPU in user-space. Subtract UserCpuNanos from CpuNanos to get system time.
FreeMemorylongThe free memory at the end of this query.
TotalMemorylongThe total size of the JVM heap memory at the end of this query.
FreeMemoryChangelongThe change in free memory during this query.
TotalMemoryChangelongThe change in total JVM heap memory during this query.
CollectionslongThe number of garbage collections while executing this query.
CollectionTimeNanoslongThe number of nanoseconds spent in the garbage collector while executing this query.
AllocatedByteslongThe number of bytes allocated by this query.
PoolAllocatedByteslongThe number of bytes allocated from pools by this query.
DataReadNanoslongHow long, in nanoseconds, this query spent blocking while reading data.
DataReadCountlongThe number of blocking data reads that occurred during this query.
DataReadByteslongHow many bytes this query read while performing blocking operations accounted for in DataReadNanos and DataReadCount.
MetadataOperationNanoslongHow long, in nanoseconds, this query spent on blocking metadata operations.
MetadataOperationCountlongThe number of blocking metadata operations that occurred during this query.
WasInterruptedbooleanWas this query interrupted?
ExceptionStringIf this query produced an exception, the exception's message. Otherwise null.
AuthContextStringThe authentication context used for executing this query.
WorkerHeapSizelongThe maximum heap size of the worker, in bytes.

Query Operation Performance Tree Table

query_operation_performance_tree_table returns a tree table of every query and its operations. It combines the rows of the Query Performance Log and the Query Operation Performance Log. Like a traditional call graph, it places each operation in an expandable group under its query or parent operation. Unlike Query Operation Performance, it takes no evaluation number, so use it when you don't yet know which query is slow.

The Query Operation Performance Tree Table

Query Operation Performance Tree Table columns
Column NameData TypeDescription
EvalKeyStringThe row's key in the tree: the EvaluationNumber for a query row, or EvaluationNumber:OperationNumber for an operation row.
ParentEvalKeyStringThe EvalKey of this row's parent, used to construct the tree.
SessionIdStringA unique identifier for the user session performing the operation.
DescriptionStringA textual description of the operation, generally including the name and some of the arguments.
StartTimeInstantThe time this operation began.
EndTimeInstantThe time this operation finished.
UsageNanoslongTime spent running this operation, in nanoseconds, summed over every thread that worked on it. Parallel work can make it larger than the wall-clock time from StartTime to EndTime.
CpuNanoslongHow long, in nanoseconds, this operation used the CPU.
UserCpuNanoslongHow long, in nanoseconds, this operation used the CPU in user-space. Subtract UserCpuNanos from CpuNanos to get system time.
FreeMemorylongThe free memory at the end of this operation.
TotalMemorylongThe total size of the JVM heap memory at the end of this operation.
FreeMemoryChangelongThe change in free memory during this operation.
TotalMemoryChangelongThe change in total JVM heap memory during this operation.
CollectionslongThe number of garbage collections while executing this operation.
CollectionTimeNanoslongThe number of nanoseconds spent in the garbage collector while executing this operation.
AllocatedByteslongThe number of bytes allocated by this operation.
PoolAllocatedByteslongThe number of bytes allocated from pools by this operation.
DataReadNanoslongHow long, in nanoseconds, this operation spent blocking while reading data.
DataReadCountlongThe number of blocking data reads that occurred during this operation.
DataReadByteslongHow many bytes this operation read while performing blocking operations accounted for in DataReadNanos and DataReadCount.
MetadataOperationNanoslongHow long, in nanoseconds, this operation spent on blocking metadata operations.
MetadataOperationCountlongThe number of blocking metadata operations that occurred during this operation.
WasInterruptedbooleanWas this operation interrupted?
ExceptionStringIf this query produced an exception, the exception's message. Otherwise null. Always null for operation rows.
AuthContextStringThe authentication context used for executing this operation.
WorkerHeapSizelongThe maximum heap size of the worker, in bytes.
DepthintThe operation's nesting level: 0 for an operation directly under its query, plus 1 for each enclosing operation. Null for query rows.
CallerLineStringThe file and line number where the operation was initiated, if available.
IsCompilationbooleanDoes this represent a formula compilation operation? Null for query rows.
InputSizeLonglongThe number of rows in the input for this operation. Null for query rows.
EvaluationNumberlongThe evaluation number of the query this row belongs to.
ParentEvaluationNumberlongWhen this row's query is a subquery, the evaluation number of the query that started it.
OperationNumberintAn identifier for an operation within an evaluation.
ParentOperationNumberintThe OperationNumber of the operation that encloses this one in the same query. Null when the operation is not nested inside another operation.

Other performance tables

The tables above are not the only ones perfmon provides. The Barrage subscription and snapshot performance logs record statistics for Barrage, the protocol Deephaven uses to send tables between processes. Create them with barrage_subscription_performance_log and barrage_snapshot_performance_log. The Barrage Performance section of the performance tables cheat sheet shows how to use them, and the Barrage metrics guide explains what they record.

For every function in the module, see the perfmon Pydoc.

Get help

If you can't find an obvious cause for an error or slow performance in your query, you might have found a bug. File a bug report and include the following:

  • The version of the Deephaven system.
  • The complete stack trace, not just the Caused by: lines.
  • A code snippet or a description of the query.
  • Any logs from your user console. If you run Deephaven in Docker, include the logs from your terminal.
  • Support logs exported from the browser. In the Settings menu, click the Export Logs button to download a zip with the browser logs and the current state of the application.

You can also get help by asking questions in our Slack Community or on the Community Discussions page.