---
title: Performance monitoring
sidebar_label: Monitoring
---

This guide covers cluster-wide performance monitoring for administrators, including internal tables, external monitoring integration, and alerting strategies.

> [!NOTE]
> For monitoring individual query performance, see [Monitor queries](../../performance/monitor-queries.md).

## Overview

Effective cluster monitoring requires tracking:

| Layer          | Metrics                                | Tools                                        |
| -------------- | -------------------------------------- | -------------------------------------------- |
| Infrastructure | CPU, memory, disk, network             | Prometheus Node Exporter, `iostat`, `vmstat` |
| Application    | Process health, PQ status, connections | Status Dashboard, internal tables            |
| Query          | Execution time, resource usage, errors | Internal performance tables                  |

## Internal tables for monitoring

Deephaven logs operational data to [internal tables](../internal-tables/internal-tables.md) in the `DbInternal` namespace.

### Key monitoring tables

| Table                     | Purpose                   | Key columns                                              |
| ------------------------- | ------------------------- | -------------------------------------------------------- |
| `ProcessMetricsLogCoreV2` | JVM and system metrics    | `Name`, `Last`, `Min`, `Max`, `Avg`                      |
| `ProcessEventLog`         | Process log messages      | `Process`, `Level`, `LogEntry`                           |
| `PersistentQueryStateLog` | PQ state changes          | `Name`, `Status`, `Owner`                                |
| `ResourceUtilization`     | Dispatcher resource usage | `HeapUsageMB`, `HeapAvailableMB`, `WorkerCount`          |
| `ServerStateLogCoreV2`    | Worker JVM metrics        | `TotalMemoryMiB`, `FreeMemoryMiB`, `IntervalCollections` |

### Process metrics queries

```python
# Memory metrics across all processes
memory_metrics = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date = today()", "Name.startsWith(`Memory-`)"])
    .view(["Timestamp", "ProcessInfoId", "Name", "Last", "Max"])
    .sort("Timestamp")
)

# GC metrics
gc_metrics = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date = today()", "Name.startsWith(`Memory-GC-`)"])
    .sort("Timestamp")
)
```

```groovy
// Memory metrics across all processes
memoryMetrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2")
    .where("Date = today()", "Name.startsWith(`Memory-`)")
    .view("Timestamp", "ProcessInfoId", "Name", "Last", "Max")
    .sort("Timestamp")

// GC metrics
gcMetrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2")
    .where("Date = today()", "Name.startsWith(`Memory-GC-`)")
    .sort("Timestamp")
```

### Persistent Query monitoring

```python
# Current PQ states
pq_states = (
    db.live_table("DbInternal", "PersistentQueryStateLog")
    .where("Date = today()")
    .last_by(["Owner", "Name"])
    .view(["Owner", "Name", "Status", "Timestamp", "ServerHost"])
)

# Failed or errored PQs
failed_pqs = pq_states.where("Status in `Failed`, `Error`, `Disconnected`")
```

```groovy
// Current PQ states
pqStates = db.liveTable("DbInternal", "PersistentQueryStateLog")
    .where("Date = today()")
    .lastBy("Owner", "Name")
    .view("Owner", "Name", "Status", "Timestamp", "ServerHost")

// Failed or errored PQs
failedPqs = pqStates.where("Status in `Failed`, `Error`, `Disconnected`")
```

### Resource utilization

```python
# Dispatcher resource usage
resource_util = (
    db.live_table("DbInternal", "ResourceUtilization")
    .where("Date = today()")
    .view(
        [
            "Timestamp",
            "ResourceProcessName",
            "HeapUsageMB",
            "HeapAvailableMB",
            "WorkerCount",
            "Comment",
        ]
    )
    .sort("Timestamp")
)
```

```groovy
// Dispatcher resource usage
resourceUtil = db.liveTable("DbInternal", "ResourceUtilization")
    .where("Date = today()")
    .view("Timestamp", "ResourceProcessName", "HeapUsageMB", "HeapAvailableMB", "WorkerCount", "Comment")
    .sort("Timestamp")
```

## Status Dashboard (Prometheus)

The [Status Dashboard](../status-dashboard.md) provides a Prometheus-compatible metrics endpoint for external monitoring.

### Default configuration

| Property                               | Default   | Description            |
| -------------------------------------- | --------- | ---------------------- |
| `StatusDashboard.prometheus.port`      | 8112      | Prometheus scrape port |
| `StatusDashboard.prometheus.namespace` | Deephaven | Metrics namespace      |
| `StatusDashboard.useSsl`               | true      | Enable HTTPS           |
| `StatusDashboard.useAuthentication`    | true      | Require authentication |

### Available metrics

The Status Dashboard exports:

- **Process health**: Controller, dispatchers, PQ status
- **Certificate expiration**: Days until SSL certificates expire
- **Data lag**: Latency of internal table updates
- **Custom PQ metrics**: User-defined monitoring

### Prometheus integration

Configure Prometheus to scrape the Status Dashboard:

```yaml
scrape_configs:
  - job_name: "deephaven"
    scheme: https
    basic_auth:
      username: dashboard_user
      password_file: /path/to/password
    static_configs:
      - targets: ["deephaven-server:8112"]
```

See [Status Dashboard](../status-dashboard.md) for complete configuration.

## Key metrics to monitor

### Process health

These processes must be running for the cluster to function:

- **Controller**: Manages PQ lifecycle and cluster state
- **Dispatcher**: Starts and manages workers
- **TDCP**: Caches intraday data for workers
- **DIS**: Ingests and serves intraday data

Use `dh_monit summary` to check process status.

### Resource utilization

Appropriate resource thresholds are system-dependent. Establish baselines during normal operations and monitor for deviation.

**How to monitor:**

```python
# Heap and GC metrics
memory_metrics = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date = today()", "Name.startsWith(`Memory-`)"])
    .view(["Timestamp", "ProcessInfoId", "Name", "Last", "Max"])
)
```

```groovy
// Heap and GC metrics
memoryMetrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2")
    .where("Date = today()", "Name.startsWith(`Memory-`)")
    .view("Timestamp", "ProcessInfoId", "Name", "Last", "Max")
```

**What to look for:**

- **Heap usage after GC**: Query `Memory-Heap.Used` / `Memory-Heap.Max`. Rising trends may indicate memory pressure.
- **GC pause time**: Query `Memory-GC-*` metrics. Increasing pauses affect responsiveness.
- **CPU utilization**: Use `htop` or query CPU metrics from `ProcessMetricsLogCoreV2`.
- **Disk utilization**: Use `df -h` and `iostat -xz 5` to check capacity and I/O.

### Data pipeline

**Metrics to track:**

- **DIS write throughput**: Significant drops indicate ingestion issues.
- **Merge completion**: Delays affect historical data availability.
- **Internal table lag**: Affects monitoring and audit data freshness.

### Persistent Queries

**Metrics to track:**

- **PQ status**: Error/Failed states require investigation.
- **PQ restart frequency**: High restart rates indicate instability.
- **Worker acquisition time**: Slow acquisition indicates dispatcher capacity issues.

## Alerting strategies

### Recommended alerts

**Critical (immediate action):**

- Controller or dispatcher down
- DIS not running
- Disk space critically low
- Certificate expiring soon

**Warning (investigate soon):**

- PQ in Failed/Error state
- Resource utilization consistently above baseline
- GC pauses increasing
- Merge jobs behind schedule

**Informational:**

- PQ restarts
- Worker acquisitions
- Configuration changes

### Alert configuration

Using Grafana with Prometheus:

1. Import the example dashboard from `/usr/illumon/latest/etc/grafanaDashboard.json`.
2. Configure alert rules based on your established baselines.
3. Set up notification channels (email, Slack, PagerDuty).

## Monitoring commands

### Process status

```bash
# Check all Deephaven processes
dh_monit summary

# Check specific process
dh_monit status iris_controller
dh_monit status db_tdcp
dh_monit status db_dis
```

### System metrics

```bash
# CPU and memory overview
top -b -n 1 | head -20

# I/O statistics
iostat -xz 5

# Network connections
netstat -an | grep -E ':(8123|22014|9092)' | wc -l
```

### Log analysis

```bash
# Recent errors in controller log
grep -i error /var/log/deephaven/iris_controller/PersistentQueryController.log.current | tail -20

# PQ failures
grep -i "Failed\|Error" /var/log/deephaven/iris_controller/PersistentQueryController.log.current
```

## Quick reference

### Internal table quick access

```python
# All key monitoring tables for today (sorted by Timestamp)
pel = (
    db.live_table("DbInternal", "ProcessEventLog")
    .where("Date=today()")
    .sort("Timestamp")
)
pml = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where("Date=today()")
    .sort("Timestamp")
)
pqsl = (
    db.live_table("DbInternal", "PersistentQueryStateLog")
    .where("Date=today()")
    .sort("Timestamp")
)
rul = (
    db.live_table("DbInternal", "ResourceUtilization")
    .where("Date=today()")
    .sort("Timestamp")
)
ssl = (
    db.live_table("DbInternal", "ServerStateLogCoreV2")
    .where("Date=today()")
    .sort("Timestamp")
)
```

```groovy
// All key monitoring tables for today (sorted by Timestamp)
pel = db.liveTable("DbInternal", "ProcessEventLog").where("Date=today()").sort("Timestamp")
pml = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2").where("Date=today()").sort("Timestamp")
pqsl = db.liveTable("DbInternal", "PersistentQueryStateLog").where("Date=today()").sort("Timestamp")
rul = db.liveTable("DbInternal", "ResourceUtilization").where("Date=today()").sort("Timestamp")
ssl = db.liveTable("DbInternal", "ServerStateLogCoreV2").where("Date=today()").sort("Timestamp")
```

### Monitoring checklist

1. ✅ Status Dashboard configured and accessible
2. ✅ Prometheus scraping Deephaven metrics
3. ✅ Node Exporter running on all hosts
4. ✅ Grafana dashboards configured
5. ✅ Alert rules defined for critical metrics
6. ✅ Notification channels configured
7. ✅ Internal table queries available for investigation

## Related documentation

- [Status Dashboard](../status-dashboard.md)
- [Internal tables reference](../internal-tables/internal-tables.md)
- [Process metrics](../internal-tables/process-metrics.md)
- [Persistent Query State Log](../internal-tables/persistent-query-state-log.md)
- [Monitor queries (user guide)](../../performance/monitor-queries.md)
- [Performance tuning overview](./overview.md)