ProcessMetricsLogCoreV2

The ProcessMetricsLogCoreV2 table captures internal metrics that were previously written to the stats.log CSV. The table is disabled by default.

To enable the ProcessMetricsLogCoreV2 table for all processes, set the following property to true:

IrisLogDefaults.writeDatabaseProcessMetrics=true

Caution

Enabling this for all processes will generate a large amount of data, which can impact performance. It is recommended that this be enabled only for debugging purposes or for specific processes where detailed metrics are required.

To enable this for a single Persistent Query or Code Studio:

Click the Show Advanced dropdown (for a Persistent Query, it is in the Settings tab).
Add -DIrisLogDefaults.writeDatabaseProcessMetrics=true to the Extra JVM Arguments field.

Columns

Column Name	Column Type	Description
`Date`	String	The date on which the information was generated. This is the partitioning column.
`AuthenticatedUser`	String	The authenticated user (e.g., the user who logged in via password, SAML, etc.) that owns the worker.
`EffectiveUser`	String	The effective user (e.g., the "operate as" user if the authenticated user is using the "operate as" option) that owns the worker.
`ProcessInfoId`	String	The `ProcessInfoID` of the process that generated this event.
`Timestamp`	DateTime	The timestamp for this event.
`Name`	String	The name of the metric.
`Interval`	String	The timing interval for the metric.
`Type`	String	The type of the metric.
`N`	long	Data specific to the metric type.
`Sum`	long	Data specific to the metric type.
`Last`	long	Data specific to the metric type.
`Min`	long	Data specific to the metric type.
`Max`	long	Data specific to the metric type.
`Avg`	long	Data specific to the metric type.
`Sum2`	long	Data specific to the metric type.
`Stdev`	long	Data specific to the metric type.

Understanding the statistical fields

The ProcessMetricsLogCoreV2 table records statistical data about various system metrics over time. Each row represents a snapshot of a metric during a specific time interval. The statistical fields provide different views of the metric's behavior during that interval:

N - The number of observations or events recorded during the interval.
Sum - The total sum of all values recorded during the interval.
Last - The most recent value recorded in the interval.
Min - The minimum (smallest) value observed during the interval.
Max - The maximum (largest) value observed during the interval.
Avg - The average value across all observations (calculated as Sum / N).
Sum2 - The sum of squares of all values (used for variance calculations).
Stdev - The standard deviation of values during the interval.

Example: Counter metrics

For a counter metric like Memory-GC-G1.YoungGenCollectionCount:

N=5 means 5 garbage collection events occurred during the interval.
Sum=250 means a total of 250 milliseconds spent in GC across those events.
Last=60 means the most recent GC took 60 milliseconds.
Min=30 means the shortest GC was 30 milliseconds.
Max=80 means the longest GC was 80 milliseconds.
Avg=50 means the average GC duration was 50 milliseconds.

To view counter metrics:

// Get all Memory-GC-G1 metrics for today
gc_metrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2").where(
    "Date=today()", "Name.startsWith(`Memory-GC.G1`)"
).view("Timestamp", "Name", "Type", "N", "Sum", "Last", "Min", "Max", "Avg")

// Get all counter-type metrics
counter_metrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2").where(
    "Date=today()", "Type=`counter`"
).view("Timestamp", "Name", "N", "Sum", "Last", "Min", "Max", "Avg")

# Get all Memory-GC-G1 metrics for today
gc_metrics = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date=today()", "Name.startsWith(`Memory-GC.G1`)"])
    .view(["Timestamp", "Name", "Type", "N", "Sum", "Last", "Min", "Max", "Avg"])
)

# Get all counter-type metrics
counter_metrics = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date=today()", "Type=`counter`"])
    .view(["Timestamp", "Name", "N", "Sum", "Last", "Min", "Max", "Avg"])
)

Example: State metrics

For a state metric like Memory-Heap.Used:

N=10 means 10 samples were taken during the interval.
Last=4294967296 means the current heap usage is 4,294,967,296 bytes (4 GB).
Min=3221225472 means the minimum heap usage during the interval was ~3 GB.
Max=5368709120 means the maximum heap usage during the interval was ~5 GB.
Avg shows the average heap usage across all samples.

To view state metrics:

// Get all memory metrics for today
memory_metrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2").where(
    "Date=today()", "Name.startsWith(`Memory-`)"
).view("Timestamp", "Name", "Last", "Min", "Max", "Avg")

// Focus on heap usage specifically with conversion to GB
heap_usage = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2").where(
    "Date=today()", "Name=`Memory-Heap.Used`"
).update("LastGB=Last/1073741824.0").view("Timestamp", "LastGB", "Min", "Max")

# Get all memory metrics for today
memory_metrics = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date=today()", "Name.startsWith(`Memory-`)"])
    .view(["Timestamp", "Name", "Last", "Min", "Max", "Avg"])
)

# Focus on heap usage specifically with conversion to GB
heap_usage = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date=today()", "Name=`Memory-Heap.Used`"])
    .update("LastGB=Last/1073741824.0")
    .view(["Timestamp", "LastGB", "Min", "Max"])
)

Metric types

The Type column indicates the category of metric being recorded. Deephaven uses two primary metric types:

Counter

A counter tracks cumulative events or operations over time. Counter metrics are used to measure things that accumulate, such as:

Number of read or write operations
Number of bytes transferred
Number of operations completed

For counter metrics, the statistical fields track the individual events:

N - The count of events that occurred during the interval.
Sum - The total accumulated value across all events (e.g., total bytes).
Last - The most recent individual event value.
Min/Max - The smallest/largest individual event values.
Avg - The average event value (Sum / N).

Example: Memory-GC-G1.YoungGenCollectionCount with Type='counter' tracks individual garbage collection events.

State

A state metric represents a sampled measurement of current system state. State metrics are used to measure things that fluctuate, such as:

Current memory usage
Current thread counts
Current queue depths

For state metrics, the statistical fields track the sampled values:

N - The number of samples taken during the interval
Sum - The sum of all sampled values
Last - The most recent sampled value (typically the most useful for state metrics)
Min/Max - The minimum/maximum values observed during sampling
Avg - The average value across all samples (Sum / N)

Example: Memory-Heap.Used with Type='state' tracks the current heap memory usage at each sample point.

Common metrics

Deephaven processes log various metrics, including:

Memory-Heap.Used - JVM heap memory usage (state)
Memory-NonHeap.Used - JVM non-heap memory usage, e.g., metaspace (state)
Memory-Direct.Used - Direct buffer memory usage (state)
Memory-GC-G1.* - G1 garbage collection metrics (counter)
Thread.* - Thread-related metrics (state)

Querying metrics

To analyze ProcessMetricsLogCoreV2 data, you can query the table and filter by specific metric names:

// Get all available metric names for today
metric_names = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2").where(
    "Date=today()"
).selectDistinct("Name").sort("Name")

// Get memory usage over time
memory_metrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2").where(
    "Date=today()", "Name.startsWith(`Memory-`)"
).view("Timestamp", "Name", "Last", "Min", "Max", "Avg")

# Get all available metric names for today
metric_names = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where("Date=today()")
    .select_distinct("Name")
    .sort("Name")
)

# Get memory usage over time
memory_metrics = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date=today()", "Name.startsWith(`Memory-`)"])
    .view(["Timestamp", "Name", "Last", "Min", "Max", "Avg"])
)