Memory management

This guide explains Deephaven's memory model and provides strategies for optimizing memory configuration to achieve better performance, stability, and resource utilization.

Understanding the Deephaven memory model

Deephaven uses a hybrid memory architecture that leverages both JVM heap memory and off-heap direct memory.

JVM heap memory (on-heap)

The JVM heap stores Java objects created by the application, including:

• Query state and metadata: Table references, operation graphs, and query execution state
• Application objects: Controller, dispatcher, and worker process objects
• Temporary computation results: Intermediate values during query evaluation
• Data buffer pool (optional): When DataBufferConfiguration.useDirectMemory=false

Heap size is controlled by the -Xmx JVM parameter. For example, -Xmx16g allocates a maximum of 16 GB of heap memory.

Direct memory (off-heap)

Direct memory is native (non-JVM) memory used for:

• Table data storage: When DataBufferConfiguration.useDirectMemory=true, columnar table data is stored in direct memory buffers.
• Network I/O buffers: Data transfer between workers and Data Import Servers.
• Binary log data: Tailer and Data Import Server buffers for streaming data.

Direct memory size is controlled by the -XX:MaxDirectMemorySize JVM parameter.

Key difference:

• Heap memory is managed by the JVM garbage collector.
• Direct memory is manually managed by the application (not subject to GC pauses).

The data buffer pool

Deephaven stores columnar table data in a data buffer pool, which can reside in either heap or direct memory.

Configuration:

Direct vs. heap memory for table data:

See Data buffer pool configuration for complete details.

Default pool size limits:

• Heap-based buffers: Pool size limited to 60% of heap.
• Direct memory buffers: Pool size can exceed heap size (controlled by DataBufferConfiguration.directMaxPoolToHeapSizeRatio).

Garbage collection and memory

GC reclaims heap memory occupied by objects that are no longer referenced.

GC impact on performance:

• Stop-the-world pauses: Most GC phases pause all application threads.
• CPU overhead: GC threads consume CPU while scanning and compacting.
• Frequency vs. duration trade-off: More frequent GCs mean shorter pauses but higher overhead.

Deephaven GC profiles via remote processing profiles:

• G1 GC (Garbage First): Recommended for Java 11+ and large heaps (> 4 GB).
• CMS GC (Concurrent Mark Sweep): Legacy collector for Java 8.

Monitoring memory

Monitoring heap and GC metrics

Enable process metrics to track memory trends:

Query memory metrics from internal tables:

Key metrics:

• Memory-Heap.Used — Track typical usage to validate heap sizing
• Memory-GC-G1.* — Monitor GC frequency and pause times

See Process metrics for complete metric list.

Memory tuning strategies

Sizing heap and direct memory

Heap sizing guidelines

Worker heap requirements vary widely based on query complexity, data volumes, and concurrent operations. Start conservatively and increase based on monitoring.

Workers request heap size at startup via Advanced Settings → Heap Size in the Web IDE.

Direct memory sizing

For workers with direct memory data buffers:

Leave headroom for network buffers and other direct memory allocations beyond the buffer pool.

For tailers and Data Import Servers: See Tailer memory properties for sizing details.

Choosing and tuning garbage collectors

G1 GC (Garbage First)

Recommended for: Most Deephaven deployments on Java 11+.

Tuning G1 GC:

• MaxGCPauseMillis — Target maximum pause time
• ParallelGCThreads — Threads for stop-the-world phases
• ConcGCThreads — Threads for concurrent marking

CMS GC (Concurrent Mark Sweep)

For: Legacy Java 8 deployments only. Deprecated in Java 9+.

Quick reference

G1 GC key parameters

Memory monitoring approach

Appropriate memory thresholds are system-dependent.

How to establish your baseline:

1. Query heap metrics during normal operations:
   GroovyPython
2. Note typical heap usage after GC cycles
3. Set alerts when usage consistently exceeds your observed normal range

What to look for:

• Heap usage after GC: If usage trends upward over time, investigate for memory pressure or leaks
• GC pause time: Query Memory-GC-* metrics. Increasing pauses affect query responsiveness.
• Full GC frequency: Look for Full GC entries in GC logs. Frequent full GCs indicate heap sizing issues.
• Direct memory usage: If using direct memory for data buffers, monitor with jcmd <pid> VM.native_memory

Common memory issues

Related documentation

• Performance tuning overview
• CPU optimization
• JVM tuning
• Remote processing profiles
• Troubleshoot Java processes
• Process metrics