Troubleshooting performance

This guide helps administrators diagnose and resolve cluster-wide performance issues.

Note

For troubleshooting individual slow queries, see Why is my query slow?.

Diagnostic approach

Identify the symptom: What is slow or failing?
Isolate the component: Which process or server is affected?
Gather metrics: Collect relevant data from logs and internal tables.
Identify the bottleneck: CPU, memory, I/O, or network?
Apply the fix: Tune the relevant component.

Common issues

Slow data ingestion

Symptoms:

Tailer lag increasing
DIS falling behind real-time data

Diagnostic steps:

# Check DIS process status
dh_monit status db_dis

# Check recent DIS logs for errors
tail -500 /var/log/deephaven/dis/*.log.current | grep -i "error\|exception"

# Check disk I/O on intraday storage
iostat -xz 5

Common causes:

Cause	Check	Solution
Disk I/O bottleneck	`iostat` shows high utilization	Upgrade to faster storage (NVMe)
Network congestion	Check network throughput	Increase bandwidth, check MTU
DIS heap exhausted	GC logs, heap usage	Increase DIS heap
High message rate	Tailer metrics	Add DIS instances, tune batching

Slow merges

Symptoms:

Merge jobs taking longer than expected
Intraday data accumulating beyond retention

Diagnostic steps:

# Check recent merge logs for errors
tail -500 /var/log/deephaven/merge_server/*.log.current | grep -i "error\|exception"

# Check I/O during merge
iostat -xz 5

Query merge events in internal tables:

mergePerf = db.liveTable("DbInternal", "ProcessEventLog")
    .where("Date = today()", "Process.contains(`merge`)")
    .sortDescending("Timestamp")

merge_perf = (
    db.live_table("DbInternal", "ProcessEventLog")
    .where(["Date = today()", "Process.contains(`merge`)"])
    .sort_descending("Timestamp")
)

Common causes:

Cause	Check	Solution
Storage throughput	`iostat` high utilization	Upgrade storage, optimize for sequential I/O
Insufficient parallelism	Merge config	Increase `iris.concurrentWriteThreads`
Memory pressure	GC logs	Increase merge heap
Large partition sizes	Data volume	Adjust partitioning strategy

Worker out of memory

Symptoms:

OutOfMemoryError in worker logs
Workers crashing unexpectedly
PQ status shows Failed/Error

Diagnostic steps:

# Check worker logs for OOM
grep -i "OutOfMemory\|heap" /var/log/deephaven/query_server/*.log.current

# Check heap dump at configured heapDump.path (default: /var/log/deephaven/query_server/)
ls -la /var/log/deephaven/query_server/*.hprof

Check dispatcher resource usage:

resourceUtil = db.liveTable("DbInternal", "ResourceUtilization")
    .where("Date = today()")
    .view("Timestamp", "ResourceProcessName", "HeapUsageMB", "HeapAvailableMB")
    .sortDescending("HeapUsageMB")

resource_util = (
    db.live_table("DbInternal", "ResourceUtilization")
    .where("Date = today()")
    .view(["Timestamp", "ResourceProcessName", "HeapUsageMB", "HeapAvailableMB"])
    .sort_descending("HeapUsageMB")
)

Common causes:

Cause	Check	Solution
Heap too small	`HeapUsageMB` / `HeapAvailableMB` in `ResourceUtilization`	Increase worker heap size
Memory leak	Heap dump analysis	Identify and fix leak, restart worker
Large result sets	Query analysis	Optimize queries, add filters
Too many open tables	Table count	Close unused tables, optimize code

See Troubleshoot Java processes for heap dump analysis.

High query latency

Symptoms:

Queries timing out
Slow table operations
UI responsiveness issues

Diagnostic steps:

Find slow queries:

slowQueries = db.liveTable("DbInternal", "QueryPerformanceLogCoreV2")
    .where("Date = today()", "UsageNanos > 10_000_000_000")  // > 10 sec
    .sortDescending("UsageNanos")
    .view("EvaluationNumber", "UsageNanos", "Description", "PrimaryEffectiveUser")

slow_queries = (
    db.live_table("DbInternal", "QueryPerformanceLogCoreV2")
    .where(["Date = today()", "UsageNanos > 10_000_000_000"])  # > 10 sec
    .sort_descending("UsageNanos")
    .view(["EvaluationNumber", "UsageNanos", "Description", "PrimaryEffectiveUser"])
)

Check update cycle performance:

slowUpdates = db.liveTable("DbInternal", "UpdatePerformanceLogCoreV2")
    .where("Date = today()", "UsageNanos > 1_000_000_000")  // > 1 sec
    .sortDescending("UsageNanos")

slow_updates = (
    db.live_table("DbInternal", "UpdatePerformanceLogCoreV2")
    .where(["Date = today()", "UsageNanos > 1_000_000_000"])  # > 1 sec
    .sort_descending("UsageNanos")
)

Common causes:

Cause	Check	Solution
CPU bottleneck	`top`, `htop`	Add CPU, optimize queries
Memory pressure	GC logs	Increase heap, reduce data size
I/O wait	`iostat` await	Upgrade storage
Network latency	`ping`, `mtr`	Check network, TDCP placement
Inefficient query	Query analysis	Rewrite query, add indexes

GC thrashing

Symptoms:

High CPU with little progress
Frequent long GC pauses
Unresponsive workers

Diagnostic steps:

# Check GC logs
grep -i "GC\|pause" /var/log/deephaven/query_server/*.log.current

Check GC metrics from internal tables:

gcMetrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2")
    .where("Date = today()", "Name.startsWith(`Memory-GC-`)")
    .view("Timestamp", "ProcessInfoId", "Name", "Last")

gc_metrics = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date = today()", "Name.startsWith(`Memory-GC-`)"])
    .view(["Timestamp", "ProcessInfoId", "Name", "Last"])
)

Common causes:

Cause	Check	Solution
Heap too small	Frequent full GCs	Increase heap
Memory leak	Heap growing constantly	Identify leak, restart
Large allocations	GC log analysis	Optimize data structures

Diagnostic tools

System tools

Tool	Purpose	Example
`top` / `htop`	CPU and memory overview	`htop`
`iostat`	Disk I/O statistics	`iostat -xz 5`
`vmstat`	Virtual memory stats	`vmstat 5`
`netstat`	Network connections	`netstat -an \| grep ESTABLISHED`
`iftop`	Network bandwidth by connection	`sudo iftop`

Java tools

Tool	Purpose	Example
`jstack`	Thread dump	`sudo jstack -F <pid>`
`jmap`	Heap dump	`sudo jmap -dump:format=b,file=/tmp/heap.bin <pid>`
`jstat`	GC statistics	`jstat -gc <pid> 1000`
`jinfo`	JVM configuration	`sudo jinfo <pid>`

See Troubleshoot Java processes for detailed usage.

Deephaven internal tables

Table	Use for
`QueryPerformanceLogCoreV2`	Slow query identification
`UpdatePerformanceLogCoreV2`	Slow update cycles
`ProcessMetricsLogCoreV2`	JVM and system metrics
`ProcessEventLog`	Process lifecycle events
`ResourceUtilization`	Dispatcher resource usage

Quick reference

Bottleneck identification

Symptom	Likely bottleneck	Check with
High CPU, slow queries	CPU-bound	`top`, thread dump
Frequent GC, OOM	Memory-bound	GC logs, heap dump
High I/O wait	Storage-bound	`iostat`
Network timeouts	Network-bound	`ping`, `iftop`

First response checklist

✅ Check process status: dh_monit summary
✅ Check recent logs: tail -100 /var/log/deephaven/<process>/*.log.current
✅ Check system resources: top, iostat -xz 5
✅ Query internal tables for errors
✅ Capture thread dump if process is unresponsive

Troubleshooting performance

Diagnostic approach

Common issues

Slow data ingestion

Slow merges

Worker out of memory

High query latency

GC thrashing

Diagnostic tools

System tools

Java tools

Deephaven internal tables

Quick reference

Bottleneck identification

First response checklist

Related documentation