JVM tuning

This guide covers JVM configuration for Deephaven services and workers, including garbage collection, heap sizing, and performance-related flags.

Note

For adjusting heap size for individual PQs, see Memory tuning FAQ. For the Deephaven memory model, data buffer pool configuration, and understanding what drives memory usage, see Memory management.

Overview

Deephaven runs multiple Java processes with different configuration methods:

Process type	Examples	Configuration method
Services	DIS, merge, controller, TDCP	Hostconfig file
Workers	PQ workers, consoles	Processing profiles

Heap sizing

Service heap configuration

Configure service heap sizes in the hostconfig file (/etc/sysconfig/illumon) using EXTRA_ARGS:

db_dis)
    EXTRA_ARGS="$EXTRA_ARGS -j -Xms8g -j -Xmx8g"
    ;;
db_merge)
    EXTRA_ARGS="$EXTRA_ARGS -j -Xms16g -j -Xmx16g"
    ;;

See Java process launch configuration for details on hostconfig syntax.

Worker heap configuration

Worker -Xmx is determined by the heap size configured per Persistent Query or console. Remote processing profiles can configure -Xms (initial heap) — see Setting -Xms for workers.

Garbage collection

GC profiles

Deephaven provides built-in GC profiles via remote processing profiles:

Profile	GC type	Best for	Java version
`G1 GC`	Garbage First	Most workloads	Java 11+ (recommended)
`CMS GC`	Concurrent Mark Sweep	Legacy	Java 8 only
`G1 MarkStackSize 128M`	G1 with larger mark stack	Complex queries with allocation failures	Java 11+

Set default profile:

RemoteQueryDispatcher.defaultJVMProfile=G1 GC

G1 GC tuning

G1 GC is recommended for Java 11+ deployments.

Key parameters:

Parameter	Purpose	Notes
`-XX:MaxGCPauseMillis`	Target max pause time	Default: 200ms; tune based on workload
`-XX:ParallelGCThreads`	Threads for STW phases	Tune based on available cores
`-XX:ConcGCThreads`	Concurrent marking threads	Tune based on GC behavior
`-XX:G1HeapRegionSize`	Heap region size	Auto-selected; rarely needs tuning
`-XX:InitiatingHeapOccupancyPercent`	When to start marking	Default: 45

Custom G1 profile example:

RemoteProcessingRequestProfile.custom.G1Tuned.include.1=G1 GC
RemoteProcessingRequestProfile.custom.G1Tuned.jvmParameter.maxPause=-XX:MaxGCPauseMillis=200
RemoteProcessingRequestProfile.custom.G1Tuned.jvmParameter.gcThreads=-XX:ParallelGCThreads=4
RemoteProcessingRequestProfile.custom.G1Tuned.jvmParameter.concThreads=-XX:ConcGCThreads=2

GC logging

Enable GC logging to diagnose performance issues.

Java 11+ (Unified Logging):

RemoteProcessingRequestProfile.custom.GCLogging.jvmParameter.gcLog=-Xlog:gc*:file=/var/log/deephaven/gc_%p.log:time,uptime,level,tags:filecount=5,filesize=100m

The -Xlog format is what:output:decorators:output-options:

gc* — Log all GC-related messages
file=...gc_%p.log — Write to file (%p = process ID)
time,uptime,level,tags — Include timestamps and log level
filecount=5,filesize=100m — Rotate across 5 files, 100 MB each

Analyze GC logs:

GCViewer: Open source tool for visualizing GC logs
GCEasy: Online GC log analyzer
Look for: full GC frequency, pause times, heap after GC

Common JVM flags

Memory flags

Flag	Purpose	Example
`-Xmx`	Maximum heap	`-Xmx32g`
`-Xms`	Initial heap	`-Xms32g`
`-XX:MaxDirectMemorySize`	Direct memory limit	`-XX:MaxDirectMemorySize=16g`
`-XX:MaxMetaspaceSize`	Metaspace limit	`-XX:MaxMetaspaceSize=512m`

Performance flags

Flag	Purpose	Example
`-XX:+AlwaysPreTouch`	Pre-touch heap pages	Reduces startup variance
`-XX:+UseStringDeduplication`	Deduplicate strings	Saves memory
`-XX:CICompilerCount`	JIT compiler threads	2-4

Diagnostic flags

Flag	Purpose	Example
`-XX:+HeapDumpOnOutOfMemoryError`	Dump heap on OOM	Always enable
`-XX:HeapDumpPath`	Heap dump location	`/tmp/heapdump.hprof`
`-XX:+PrintFlagsFinal`	Print all JVM flags	For debugging

Example diagnostic profile:

RemoteProcessingRequestProfile.custom.Debug.include.1=G1 GC
RemoteProcessingRequestProfile.custom.Debug.jvmParameter.heapDump=-XX:+HeapDumpOnOutOfMemoryError
RemoteProcessingRequestProfile.custom.Debug.jvmParameter.heapDumpPath=-XX:HeapDumpPath=/tmp

Troubleshooting

Out of memory errors

Heap space:

java.lang.OutOfMemoryError: Java heap space

Causes and solutions:

Cause	Check	Solution
Heap too small	Heap usage near max	Increase `-Xmx`
Memory leak	Heap dump analysis	Fix leak, restart
Large data sets	Query analysis	Optimize queries

Direct buffer memory

java.lang.OutOfMemoryError: Direct buffer memory

Solution: Increase -XX:MaxDirectMemorySize.

Long GC pauses

Symptoms:

UI freezes
Query timeouts
Log shows long pause times

Solutions:

Cause	Check	Solution
Heap too large	Pause times in logs	Reduce heap or tune G1
Full GCs	GC log analysis	Increase heap, reduce allocation rate
Fragmentation	G1 evacuation failures	Increase heap regions

G1 allocation failures

[Full GC (Allocation Failure) ...]

Solution: Use the G1 MarkStackSize 128M profile or add:

RemoteProcessingRequestProfile.custom.G1Large.jvmParameter.markStack=-XX:MarkStackSize=128M

Quick reference

JVM flags by purpose

Purpose	Flags
Heap sizing	`-Xms`, `-Xmx`, `-XX:MaxDirectMemorySize`
G1 tuning	`-XX:MaxGCPauseMillis`, `-XX:ParallelGCThreads`, `-XX:ConcGCThreads`
Diagnostics	`-XX:+HeapDumpOnOutOfMemoryError`, `-Xlog:gc*`

Profile selection guide

Workload	Profile	Notes
General purpose	`G1 GC`	Default for Java 11+
High allocation rate	`G1 MarkStackSize 128M`	If seeing allocation failures
Legacy Java 8	`CMS GC`	Deprecated
Custom requirements	Create custom profile	Include base profile

Heap sizing considerations

Worker heap depends on:

Query complexity: Simple queries vs. complex joins/aggregations
Data volume: Size of tables accessed during operations
Concurrent operations: Number of simultaneous operations

Start with the sizing guidelines in Memory management and adjust based on monitoring.