---
title: Troubleshooting performance
sidebar_label: Troubleshooting
---

This guide helps administrators diagnose and resolve cluster-wide performance issues.

> [!NOTE]
> For troubleshooting individual slow queries, see [Why is my query slow?](../../resources/faq/query-slow.md).

## Diagnostic approach

1. **Identify the symptom**: What is slow or failing?
2. **Isolate the component**: Which process or server is affected?
3. **Gather metrics**: Collect relevant data from logs and internal tables.
4. **Identify the bottleneck**: CPU, memory, I/O, or network?
5. **Apply the fix**: Tune the relevant component.

## Common issues

### Slow data ingestion

**Symptoms:**

- Tailer lag increasing
- DIS falling behind real-time data

**Diagnostic steps:**

```bash
# Check DIS process status
dh_monit status db_dis

# Check recent DIS logs for errors
tail -500 /var/log/deephaven/dis/*.log.current | grep -i "error\|exception"

# Check disk I/O on intraday storage
iostat -xz 5
```

**Common causes:**

| Cause               | Check                           | Solution                         |
| ------------------- | ------------------------------- | -------------------------------- |
| Disk I/O bottleneck | `iostat` shows high utilization | Upgrade to faster storage (NVMe) |
| Network congestion  | Check network throughput        | Increase bandwidth, check MTU    |
| DIS heap exhausted  | GC logs, heap usage             | Increase DIS heap                |
| High message rate   | Tailer metrics                  | Add DIS instances, tune batching |

### Slow merges

**Symptoms:**

- Merge jobs taking longer than expected
- Intraday data accumulating beyond retention

**Diagnostic steps:**

```bash
# Check recent merge logs for errors
tail -500 /var/log/deephaven/merge_server/*.log.current | grep -i "error\|exception"

# Check I/O during merge
iostat -xz 5
```

Query merge events in internal tables:

```python
merge_perf = (
    db.live_table("DbInternal", "ProcessEventLog")
    .where(["Date = today()", "Process.contains(`merge`)"])
    .sort_descending("Timestamp")
)
```

```groovy
mergePerf = db.liveTable("DbInternal", "ProcessEventLog")
    .where("Date = today()", "Process.contains(`merge`)")
    .sortDescending("Timestamp")
```

**Common causes:**

| Cause                    | Check                     | Solution                                     |
| ------------------------ | ------------------------- | -------------------------------------------- |
| Storage throughput       | `iostat` high utilization | Upgrade storage, optimize for sequential I/O |
| Insufficient parallelism | Merge config              | Increase `iris.concurrentWriteThreads`       |
| Memory pressure          | GC logs                   | Increase merge heap                          |
| Large partition sizes    | Data volume               | Adjust partitioning strategy                 |

### Worker out of memory

**Symptoms:**

- `OutOfMemoryError` in worker logs
- Workers crashing unexpectedly
- PQ status shows Failed/Error

**Diagnostic steps:**

```bash
# Check worker logs for OOM
grep -i "OutOfMemory\|heap" /var/log/deephaven/query_server/*.log.current

# Check heap dump at configured heapDump.path (default: /var/log/deephaven/query_server/)
ls -la /var/log/deephaven/query_server/*.hprof
```

Check dispatcher resource usage:

```python
resource_util = (
    db.live_table("DbInternal", "ResourceUtilization")
    .where("Date = today()")
    .view(["Timestamp", "ResourceProcessName", "HeapUsageMB", "HeapAvailableMB"])
    .sort_descending("HeapUsageMB")
)
```

```groovy
resourceUtil = db.liveTable("DbInternal", "ResourceUtilization")
    .where("Date = today()")
    .view("Timestamp", "ResourceProcessName", "HeapUsageMB", "HeapAvailableMB")
    .sortDescending("HeapUsageMB")
```

**Common causes:**

| Cause                | Check                                                      | Solution                              |
| -------------------- | ---------------------------------------------------------- | ------------------------------------- |
| Heap too small       | `HeapUsageMB` / `HeapAvailableMB` in `ResourceUtilization` | Increase worker heap size             |
| Memory leak          | Heap dump analysis                                         | Identify and fix leak, restart worker |
| Large result sets    | Query analysis                                             | Optimize queries, add filters         |
| Too many open tables | Table count                                                | Close unused tables, optimize code    |

See [Troubleshoot Java processes](../troubleshooting/troubleshooting-java.md) for heap dump analysis.

### High query latency

**Symptoms:**

- Queries timing out
- Slow table operations
- UI responsiveness issues

**Diagnostic steps:**

Find slow queries:

```python
slow_queries = (
    db.live_table("DbInternal", "QueryPerformanceLogCoreV2")
    .where(["Date = today()", "UsageNanos > 10_000_000_000"])  # > 10 sec
    .sort_descending("UsageNanos")
    .view(["EvaluationNumber", "UsageNanos", "Description", "PrimaryEffectiveUser"])
)
```

```groovy
slowQueries = db.liveTable("DbInternal", "QueryPerformanceLogCoreV2")
    .where("Date = today()", "UsageNanos > 10_000_000_000")  // > 10 sec
    .sortDescending("UsageNanos")
    .view("EvaluationNumber", "UsageNanos", "Description", "PrimaryEffectiveUser")
```

Check update cycle performance:

```python
slow_updates = (
    db.live_table("DbInternal", "UpdatePerformanceLogCoreV2")
    .where(["Date = today()", "UsageNanos > 1_000_000_000"])  # > 1 sec
    .sort_descending("UsageNanos")
)
```

```groovy
slowUpdates = db.liveTable("DbInternal", "UpdatePerformanceLogCoreV2")
    .where("Date = today()", "UsageNanos > 1_000_000_000")  // > 1 sec
    .sortDescending("UsageNanos")
```

**Common causes:**

| Cause             | Check          | Solution                        |
| ----------------- | -------------- | ------------------------------- |
| CPU bottleneck    | `top`, `htop`  | Add CPU, optimize queries       |
| Memory pressure   | GC logs        | Increase heap, reduce data size |
| I/O wait          | `iostat` await | Upgrade storage                 |
| Network latency   | `ping`, `mtr`  | Check network, TDCP placement   |
| Inefficient query | Query analysis | Rewrite query, add indexes      |

### GC thrashing

**Symptoms:**

- High CPU with little progress
- Frequent long GC pauses
- Unresponsive workers

**Diagnostic steps:**

```bash
# Check GC logs
grep -i "GC\|pause" /var/log/deephaven/query_server/*.log.current
```

Check GC metrics from internal tables:

```python
gc_metrics = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date = today()", "Name.startsWith(`Memory-GC-`)"])
    .view(["Timestamp", "ProcessInfoId", "Name", "Last"])
)
```

```groovy
gcMetrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2")
    .where("Date = today()", "Name.startsWith(`Memory-GC-`)")
    .view("Timestamp", "ProcessInfoId", "Name", "Last")
```

**Common causes:**

| Cause             | Check                   | Solution                 |
| ----------------- | ----------------------- | ------------------------ |
| Heap too small    | Frequent full GCs       | Increase heap            |
| Memory leak       | Heap growing constantly | Identify leak, restart   |
| Large allocations | GC log analysis         | Optimize data structures |

## Diagnostic tools

### System tools

| Tool           | Purpose                         | Example                           |
| -------------- | ------------------------------- | --------------------------------- |
| `top` / `htop` | CPU and memory overview         | `htop`                            |
| `iostat`       | Disk I/O statistics             | `iostat -xz 5`                    |
| `vmstat`       | Virtual memory stats            | `vmstat 5`                        |
| `netstat`      | Network connections             | `netstat -an \| grep ESTABLISHED` |
| `iftop`        | Network bandwidth by connection | `sudo iftop`                      |

### Java tools

| Tool     | Purpose           | Example                                             |
| -------- | ----------------- | --------------------------------------------------- |
| `jstack` | Thread dump       | `sudo jstack -F <pid>`                              |
| `jmap`   | Heap dump         | `sudo jmap -dump:format=b,file=/tmp/heap.bin <pid>` |
| `jstat`  | GC statistics     | `jstat -gc <pid> 1000`                              |
| `jinfo`  | JVM configuration | `sudo jinfo <pid>`                                  |

See [Troubleshoot Java processes](../troubleshooting/troubleshooting-java.md) for detailed usage.

### Deephaven internal tables

| Table                        | Use for                   |
| ---------------------------- | ------------------------- |
| `QueryPerformanceLogCoreV2`  | Slow query identification |
| `UpdatePerformanceLogCoreV2` | Slow update cycles        |
| `ProcessMetricsLogCoreV2`    | JVM and system metrics    |
| `ProcessEventLog`            | Process lifecycle events  |
| `ResourceUtilization`        | Dispatcher resource usage |

## Quick reference

### Bottleneck identification

| Symptom                | Likely bottleneck | Check with         |
| ---------------------- | ----------------- | ------------------ |
| High CPU, slow queries | CPU-bound         | `top`, thread dump |
| Frequent GC, OOM       | Memory-bound      | GC logs, heap dump |
| High I/O wait          | Storage-bound     | `iostat`           |
| Network timeouts       | Network-bound     | `ping`, `iftop`    |

### First response checklist

1. ✅ Check process status: `dh_monit summary`
2. ✅ Check recent logs: `tail -100 /var/log/deephaven/<process>/*.log.current`
3. ✅ Check system resources: `top`, `iostat -xz 5`
4. ✅ Query internal tables for errors
5. ✅ Capture thread dump if process is unresponsive

## Related documentation

- [Troubleshoot Java processes](../troubleshooting/troubleshooting-java.md)
- [Process startup troubleshooting](../troubleshooting/process-startup-troubleshooting.md)
- [Why is my query slow?](../../resources/faq/query-slow.md)
- [Memory management](./memory-management.md)
- [CPU optimization](./cpu-optimization.md)
- [Performance monitoring](./monitoring.md)
- [Performance tuning overview](./overview.md)