Performance monitoring

This guide covers cluster-wide performance monitoring for administrators, including internal tables, external monitoring integration, and alerting strategies.

Note

For monitoring individual query performance, see Monitor queries.

Overview

Effective cluster monitoring requires tracking:

Layer	Metrics	Tools
Infrastructure	CPU, memory, disk, network	Prometheus Node Exporter, `iostat`, `vmstat`
Application	Process health, PQ status, connections	Status Dashboard, internal tables
Query	Execution time, resource usage, errors	Internal performance tables

Internal tables for monitoring

Deephaven logs operational data to internal tables in the DbInternal namespace.

Key monitoring tables

Table	Purpose	Key columns
`ProcessMetricsLogCoreV2`	JVM and system metrics	`Name`, `Last`, `Min`, `Max`, `Avg`
`ProcessEventLog`	Process log messages	`Process`, `Level`, `LogEntry`
`PersistentQueryStateLog`	PQ state changes	`Name`, `Status`, `Owner`
`ResourceUtilization`	Dispatcher resource usage	`HeapUsageMB`, `HeapAvailableMB`, `WorkerCount`
`ServerStateLogCoreV2`	Worker JVM metrics	`TotalMemoryMiB`, `FreeMemoryMiB`, `IntervalCollections`

Process metrics queries

// Memory metrics across all processes
memoryMetrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2")
    .where("Date = today()", "Name.startsWith(`Memory-`)")
    .view("Timestamp", "ProcessInfoId", "Name", "Last", "Max")
    .sort("Timestamp")

// GC metrics
gcMetrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2")
    .where("Date = today()", "Name.startsWith(`Memory-GC-`)")
    .sort("Timestamp")

# Memory metrics across all processes
memory_metrics = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date = today()", "Name.startsWith(`Memory-`)"])
    .view(["Timestamp", "ProcessInfoId", "Name", "Last", "Max"])
    .sort("Timestamp")
)

# GC metrics
gc_metrics = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date = today()", "Name.startsWith(`Memory-GC-`)"])
    .sort("Timestamp")
)

Persistent Query monitoring

// Current PQ states
pqStates = db.liveTable("DbInternal", "PersistentQueryStateLog")
    .where("Date = today()")
    .lastBy("Owner", "Name")
    .view("Owner", "Name", "Status", "Timestamp", "ServerHost")

// Failed or errored PQs
failedPqs = pqStates.where("Status in `Failed`, `Error`, `Disconnected`")

# Current PQ states
pq_states = (
    db.live_table("DbInternal", "PersistentQueryStateLog")
    .where("Date = today()")
    .last_by(["Owner", "Name"])
    .view(["Owner", "Name", "Status", "Timestamp", "ServerHost"])
)

# Failed or errored PQs
failed_pqs = pq_states.where("Status in `Failed`, `Error`, `Disconnected`")

Resource utilization

// Dispatcher resource usage
resourceUtil = db.liveTable("DbInternal", "ResourceUtilization")
    .where("Date = today()")
    .view("Timestamp", "ResourceProcessName", "HeapUsageMB", "HeapAvailableMB", "WorkerCount", "Comment")
    .sort("Timestamp")

# Dispatcher resource usage
resource_util = (
    db.live_table("DbInternal", "ResourceUtilization")
    .where("Date = today()")
    .view(
        [
            "Timestamp",
            "ResourceProcessName",
            "HeapUsageMB",
            "HeapAvailableMB",
            "WorkerCount",
            "Comment",
        ]
    )
    .sort("Timestamp")
)

Status Dashboard (Prometheus)

The Status Dashboard provides a Prometheus-compatible metrics endpoint for external monitoring.

Default configuration

Property	Default	Description
`StatusDashboard.prometheus.port`	8112	Prometheus scrape port
`StatusDashboard.prometheus.namespace`	Deephaven	Metrics namespace
`StatusDashboard.useSsl`	true	Enable HTTPS
`StatusDashboard.useAuthentication`	true	Require authentication

Available metrics

The Status Dashboard exports:

Process health: Controller, dispatchers, PQ status
Certificate expiration: Days until SSL certificates expire
Data lag: Latency of internal table updates
Custom PQ metrics: User-defined monitoring

Prometheus integration

Configure Prometheus to scrape the Status Dashboard:

scrape_configs:
  - job_name: "deephaven"
    scheme: https
    basic_auth:
      username: dashboard_user
      password_file: /path/to/password
    static_configs:
      - targets: ["deephaven-server:8112"]

See Status Dashboard for complete configuration.

Key metrics to monitor

Process health

These processes must be running for the cluster to function:

Controller: Manages PQ lifecycle and cluster state
Dispatcher: Starts and manages workers
TDCP: Caches intraday data for workers
DIS: Ingests and serves intraday data

Use dh_monit summary to check process status.

Resource utilization

Appropriate resource thresholds are system-dependent. Establish baselines during normal operations and monitor for deviation.

How to monitor:

// Heap and GC metrics
memoryMetrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2")
    .where("Date = today()", "Name.startsWith(`Memory-`)")
    .view("Timestamp", "ProcessInfoId", "Name", "Last", "Max")

# Heap and GC metrics
memory_metrics = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date = today()", "Name.startsWith(`Memory-`)"])
    .view(["Timestamp", "ProcessInfoId", "Name", "Last", "Max"])
)

What to look for:

Heap usage after GC: Query Memory-Heap.Used / Memory-Heap.Max. Rising trends may indicate memory pressure.
GC pause time: Query Memory-GC-* metrics. Increasing pauses affect responsiveness.
CPU utilization: Use htop or query CPU metrics from ProcessMetricsLogCoreV2.
Disk utilization: Use df -h and iostat -xz 5 to check capacity and I/O.

Data pipeline

Metrics to track:

DIS write throughput: Significant drops indicate ingestion issues.
Merge completion: Delays affect historical data availability.
Internal table lag: Affects monitoring and audit data freshness.

Persistent Queries

Metrics to track:

PQ status: Error/Failed states require investigation.
PQ restart frequency: High restart rates indicate instability.
Worker acquisition time: Slow acquisition indicates dispatcher capacity issues.

Alerting strategies

Recommended alerts

Critical (immediate action):

Controller or dispatcher down
DIS not running
Disk space critically low
Certificate expiring soon

Warning (investigate soon):

PQ in Failed/Error state
Resource utilization consistently above baseline
GC pauses increasing
Merge jobs behind schedule

Informational:

PQ restarts
Worker acquisitions
Configuration changes

Alert configuration

Using Grafana with Prometheus:

Import the example dashboard from /usr/illumon/latest/etc/grafanaDashboard.json.
Configure alert rules based on your established baselines.
Set up notification channels (email, Slack, PagerDuty).

Monitoring commands

Process status

# Check all Deephaven processes
dh_monit summary

# Check specific process
dh_monit status iris_controller
dh_monit status db_tdcp
dh_monit status db_dis

System metrics

# CPU and memory overview
top -b -n 1 | head -20

# I/O statistics
iostat -xz 5

# Network connections
netstat -an | grep -E ':(8123|22014|9092)' | wc -l

Log analysis

# Recent errors in controller log
grep -i error /var/log/deephaven/iris_controller/PersistentQueryController.log.current | tail -20

# PQ failures
grep -i "Failed\|Error" /var/log/deephaven/iris_controller/PersistentQueryController.log.current

Quick reference

Internal table quick access

// All key monitoring tables for today (sorted by Timestamp)
pel = db.liveTable("DbInternal", "ProcessEventLog").where("Date=today()").sort("Timestamp")
pml = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2").where("Date=today()").sort("Timestamp")
pqsl = db.liveTable("DbInternal", "PersistentQueryStateLog").where("Date=today()").sort("Timestamp")
rul = db.liveTable("DbInternal", "ResourceUtilization").where("Date=today()").sort("Timestamp")
ssl = db.liveTable("DbInternal", "ServerStateLogCoreV2").where("Date=today()").sort("Timestamp")

# All key monitoring tables for today (sorted by Timestamp)
pel = (
    db.live_table("DbInternal", "ProcessEventLog")
    .where("Date=today()")
    .sort("Timestamp")
)
pml = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where("Date=today()")
    .sort("Timestamp")
)
pqsl = (
    db.live_table("DbInternal", "PersistentQueryStateLog")
    .where("Date=today()")
    .sort("Timestamp")
)
rul = (
    db.live_table("DbInternal", "ResourceUtilization")
    .where("Date=today()")
    .sort("Timestamp")
)
ssl = (
    db.live_table("DbInternal", "ServerStateLogCoreV2")
    .where("Date=today()")
    .sort("Timestamp")
)

Monitoring checklist

✅ Status Dashboard configured and accessible
✅ Prometheus scraping Deephaven metrics
✅ Node Exporter running on all hosts
✅ Grafana dashboards configured
✅ Alert rules defined for critical metrics
✅ Notification channels configured
✅ Internal table queries available for investigation

Performance monitoring

Overview

Internal tables for monitoring

Key monitoring tables

Process metrics queries

Persistent Query monitoring

Resource utilization

Status Dashboard (Prometheus)

Default configuration

Available metrics

Prometheus integration

Key metrics to monitor

Process health

Resource utilization

Data pipeline

Persistent Queries

Alerting strategies

Recommended alerts

Alert configuration

Monitoring commands

Process status

System metrics

Log analysis

Quick reference

Internal table quick access

Monitoring checklist

Related documentation