Process memory configuration

Deephaven processes use several distinct kinds of memory. Understanding which kind a service uses — and how to adjust it — prevents over-allocation, avoids out-of-memory failures, and provides a baseline for capacity planning.

This guide covers system service memory defaults and tuning for all deployment types (bare-metal, Podman, and Kubernetes). For worker/Persistent Query heap sizing, see the worker heap size guide.

Memory types

Per-service memory defaults

The defaults below come from the hostconfig templates in configs/src/main/resources/templates/ and the bin/start_tailer* scripts. These are the values installed by default; you can override them per service without modifying installer-owned files.

Signs of memory pressure

Before tuning, confirm that memory is actually the problem. Search the Process Event Log for these patterns:

• JVM heap exhausted: java.lang.OutOfMemoryError: Java heap space — the service needs more -Xmx, or a query running inside it is over-allocating.
• Direct memory exhausted: java.lang.OutOfMemoryError: Direct buffer memory — increase -XX:MaxDirectMemorySize for the service, or reduce concurrent operations that hold direct buffers.
• GC overhead: java.lang.OutOfMemoryError: GC overhead limit exceeded — the JVM is spending more than 98% of CPU time on garbage collection. Increase heap significantly or investigate object retention in queries.
• Data buffer pool pressure: AutoReclaimingObjectPool-DataBufferPool: Failed to take() — see Key log lines for the full progression from pressure to fatal exhaustion.
• Kubernetes OOMKilled: a pod exits with Reason: OOMKilled in kubectl describe pod output — the container exceeded its memory limit. Both the JVM heap and the container limit must be increased together; see Kubernetes.

Tuning system process memory

Bare-metal and Podman

Override memory settings in the customer-owned hostconfig file — do not modify installer-owned files so that overrides survive upgrades.

Edit /etc/sysconfig/deephaven/illumon.iris.hostconfig and add a case clause for the target process. Always append to EXTRA_ARGS using the "$EXTRA_ARGS <args>" pattern, and prefix each JVM argument with -j.

To increase db_dis heap from 16 GB to 32 GB:

To increase direct memory for a tailer:

For more examples, including how to handle Java heap and direct memory OutOfMemoryError, see the out-of-memory FAQ.

Kubernetes

For Kubernetes deployments, memory for non-worker services is set via Helm values. Two settings must stay in sync: the JVM max heap (-Xmx) and the container memory limit.

To change a service's JVM heap, set userProc.<service>.jvmArgsMemory in your my-values.yaml:

Then raise the container memory limit to match (include headroom for off-heap usage):

Run helm upgrade with your values file to apply changes. For troubleshooting OOM-killed pods, see Kubernetes troubleshooting.

Worker and PQ memory

Each Persistent Query or Code Studio worker runs in its own JVM, separate from the dispatcher. Heap size is set in the Settings panel of the PQ or Code Studio UI. The default displayed is RemoteProcessingRequest.defaultQueryHeapMB (4096 MB).

For Python-heavy workloads, the Additional Memory field reserves extra memory beyond the JVM heap for Python native allocations and Java direct memory. On Kubernetes, this value is added to the worker pod's memory request and limit.

For detailed guidance on dispatcher-level limits, per-worker overhead, and multi-dispatcher tuning, see the worker heap size guide.

Data buffer pool

The data buffer pool is an internal pool of fixed-size binary buffers (64 KB each by default) used by db_dis, db_ltds, db_tdcp, and query worker processes to read, write, and cache Deephaven-format binary data. It is the most significant driver of merge performance and a critical resource for the Data Import Server.

How sizing works

The pool size is clamped automatically between minimum and maximum ratios of the JVM heap:

• Minimum: 10% of heap (DataBufferConfiguration.minPoolToHeapSizeRatio=0.1)
• Maximum (heap buffers): 60% of heap (DataBufferConfiguration.heapMaxPoolToHeapSizeRatio=0.6)
• Maximum (direct buffers): 200% of heap (DataBufferConfiguration.directMaxPoolToHeapSizeRatio=2.0)

The configured pool size is rounded down to the nearest multiple of the buffer block size (64 KB). If the configured size falls outside the ratio bounds, it is clamped to the nearest bound.

Default pool sizes

The per-service defaults (from hostconfig) are set via the DataBufferPool.sizeInBytes property:

Configuring pool size

To change the pool size for a system process, override the property in hostconfig:

For Persistent Queries, set Data Memory Ratio in the PQ editor's advanced options. This ratio is applied to the PQ's heap at startup. The Controller enforces minimum and maximum ratio bounds defined in the Controller's reloadable configuration.

DIS sizing guidance

The Data Import Server must hold one buffer per column file per open partition (each partition with an active tailer connection). A practical rule of thumb is:

pool size ≥ nPartitions × nColumns × 1.2 × 64 KB

If the DIS exhausts the pool and cannot free buffers through synchronous cleanup, the process will terminate.

LTDS sizing guidance

The LocalTableDataServer (db_ltds) serves historical table data to query workers. Its buffer pool must hold enough buffers to satisfy concurrent read requests from all connected workers. On deployments with many simultaneous queries reading large historical tables, the default 4 GB pool can become a bottleneck.

A practical starting point is to size the pool to match the peak number of concurrent worker read streams, then double the DIS sizing rule-of-thumb:

pool size ≥ nConcurrentWorkers × nColumns × averageReadDepth × 64 KB

If you observe repeated Failed to take() log lines from db_ltds specifically, increase DataBufferConfiguration.poolSize for that process and raise -Xmx proportionally to stay within the 60% heap ratio bound.

Merge worker sizing guidance

For merge workers, the pool is the primary driver of heap usage. The recommended sizing formula is:

buffer pool size = max_column_file_size × nWritingThreads / nOutputPartitions / 0.85

heap size = 2 × buffer pool size

For more context on merge memory planning, see Merge optimization.

Configuration properties

The following properties influence data buffer pool behavior. Typically only DataBufferConfiguration.poolSize and iris.concurrentWriteThreads need to be changed from their defaults.

Key log lines

Monitor these log messages to identify pool pressure before it becomes a problem:

• AutoReclaimingObjectPool-DataBufferPool: Failed to take() an item for thread=<thread>, initiating synchronous cleanup — A thread could not acquire a buffer lock-free and is blocking to reclaim space. The pool is under pressure; consider increasing pool size.
• AutoReclaimingObjectPool-DataBufferPool: Unable to take() an item for thread=<thread>, yielding (<n>/<n>) — All buffers are actively in use; the thread is yielding the CPU. Increase pool size.
• AutoReclaimingObjectPool-DataBufferPool: Unable to take() an item for thread=<thread> after <n> yields — Fatal: the pool is exhausted after multiple yield attempts. This is followed by an ObjectPoolExhaustedError. The process will terminate.

Related documentation

• Worker heap size
• Out-of-memory errors
• Memory tuning for persistent queries
• Merge optimization
• Metrics and monitoring
• Kubernetes service resource configuration
• Tailer memory properties
• Heap dump on assertion failure