--- title: Process memory configuration --- Deephaven processes use several distinct kinds of memory. Understanding which kind a service uses — and how to adjust it — prevents over-allocation, avoids out-of-memory failures, and provides a baseline for capacity planning. This guide covers system service memory defaults and tuning for all deployment types (bare-metal, Podman, and Kubernetes). For worker/Persistent Query heap sizing, see the [worker heap size guide](../pq-controller/worker-heap-size.md). ## Memory types | Type | What it is | Where it is configured | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | **JVM heap** | Objects allocated by Java code; managed by the garbage collector. Set with `-Xmx` (max) and `-Xms` (initial). | `hostconfig` / Helm values per service; PQ settings for workers | | **JVM direct memory** | `ByteBuffer.allocateDirect` allocations outside the heap; used by NIO and data pipeline buffers. Set with `-XX:MaxDirectMemorySize`. | `hostconfig` / Helm values; defaults on query/merge/tailer processes | | **Data buffer pool** | A bounded pool of fixed-size (64 KB) buffers shared within a process for reading, writing, and caching Deephaven-format binary data. Backed by heap memory by default; can use direct memory via `DataBufferConfiguration.useDirectMemory`. | `DataBufferConfiguration.poolSize` or `DataBufferPool.sizeInBytes` (deprecated); see [Data buffer pool](#data-buffer-pool) | | **Python native memory** | Memory allocated by the Python runtime in workers running user-defined functions (UDFs). Not tracked by JVM settings. | **Additional Memory** field in PQ / Code Studio settings | ## Per-service memory defaults The defaults below come from the hostconfig templates in `configs/src/main/resources/templates/` and the `bin/start_tailer*` scripts. These are the values installed by default; you can override them per service without modifying installer-owned files. | Service | JVM heap (`-Xms`/`-Xmx`) | Direct memory | Data buffer pool | | -------------------------------------- | ------------------------ | ------------- | ---------------- | | `iris_controller` | 4 GB / 4 GB | — | — | | `configuration_server` | 4 GB / 4 GB | — | — | | `authentication_server` | 1 GB / 1 GB | — | — | | `db_acl_write_server` | 1 GB / 1 GB | — | — | | `log_aggregator_service` | 4 GB / 4 GB | — | — | | `web_api_service` | 4 GB / 4 GB | — | — | | `db_query_server` (dispatcher process) | 4 GB / 4 GB | 2 GB | — | | `db_merge_server` (dispatcher process) | 4 GB / 4 GB | 2 GB | — | | `db_dis` | 16 GB / 16 GB | — | 10 GB | | `db_tdcp` | 4 GB / 4 GB | — | 4 GB | | `db_ltds` | 8 GB / 8 GB | — | 4 GB | | `tailer` (1/2/n) | 2 GB / 2 GB | 256 MB | — | | `status_dashboard` | 2 GB / 2 GB | — | — | > [!NOTE] > The `db_query_server` and `db_merge_server` rows describe the dispatcher process itself. Each worker the dispatcher spawns runs in a separate JVM with its own heap; see [Worker and PQ memory](#worker-and-pq-memory). > [!TIP] > If all services run on a single host, the combined default JVM heap across all system services is approximately 60 GB. The three services with data buffer pools (`db_dis`, `db_tdcp`, `db_ltds`) add another 18 GB of committed pool memory. Worker heap is additional and depends on the number of concurrent queries and their configured sizes. ## Signs of memory pressure Before tuning, confirm that memory is actually the problem. Search the [Process Event Log](../internal-tables/process-event-log.md) for these patterns: - **JVM heap exhausted**: `java.lang.OutOfMemoryError: Java heap space` — the service needs more `-Xmx`, or a query running inside it is over-allocating. - **Direct memory exhausted**: `java.lang.OutOfMemoryError: Direct buffer memory` — increase `-XX:MaxDirectMemorySize` for the service, or reduce concurrent operations that hold direct buffers. - **GC overhead**: `java.lang.OutOfMemoryError: GC overhead limit exceeded` — the JVM is spending more than 98% of CPU time on garbage collection. Increase heap significantly or investigate object retention in queries. - **Data buffer pool pressure**: `AutoReclaimingObjectPool-DataBufferPool: Failed to take()` — see [Key log lines](#key-log-lines) for the full progression from pressure to fatal exhaustion. - **Kubernetes OOMKilled**: a pod exits with `Reason: OOMKilled` in `kubectl describe pod` output — the container exceeded its memory limit. Both the JVM heap and the container limit must be increased together; see [Kubernetes](#kubernetes). ## Tuning system process memory ### Bare-metal and Podman Override memory settings in the customer-owned hostconfig file — do not modify installer-owned files so that overrides survive upgrades. Edit `/etc/sysconfig/deephaven/illumon.iris.hostconfig` and add a `case` clause for the target process. Always append to `EXTRA_ARGS` using the `"$EXTRA_ARGS "` pattern, and prefix each JVM argument with `-j`. To increase `db_dis` heap from 16 GB to 32 GB: ```bash db_dis) EXTRA_ARGS="$EXTRA_ARGS -j -Xms32g -j -Xmx32g" ;; ``` To increase direct memory for a tailer: ```bash tailer) EXTRA_ARGS="$EXTRA_ARGS -j -XX:MaxDirectMemorySize=1g" ;; ``` > [!IMPORTANT] > The tailer's default heap and direct memory are set in `bin/start_tailer` (not in a hostconfig template), so no base `EXTRA_ARGS` exists to append to. Use the hostconfig override to add JVM arguments on top of the defaults in the start script. For more examples, including how to handle Java heap and direct memory `OutOfMemoryError`, see the [out-of-memory FAQ](../../resources/faq/out-of-memory.md). ### Kubernetes For Kubernetes deployments, memory for non-worker services is set via Helm values. Two settings must stay in sync: the JVM max heap (`-Xmx`) and the container memory limit. To change a service's JVM heap, set `userProc..jvmArgsMemory` in your `my-values.yaml`: ```yaml userProc: controller: jvmArgsMemory: "-Xmx6g -Xms6g" ``` Then raise the container memory limit to match (include headroom for off-heap usage): ```yaml resources: controller: limits: memory: 6.5Gi requests: memory: 6.5Gi ``` Run `helm upgrade` with your values file to apply changes. For troubleshooting OOM-killed pods, see [Kubernetes troubleshooting](../kubernetes/troubleshooting-kubernetes.md#how-do-i-troubleshoot-deephaven-service-pods-that-get-killed-by-kubernetes). ## Worker and PQ memory Each Persistent Query or Code Studio worker runs in its own JVM, separate from the dispatcher. Heap size is set in the **Settings** panel of the PQ or Code Studio UI. The default displayed is `RemoteProcessingRequest.defaultQueryHeapMB` (4096 MB). For Python-heavy workloads, the **Additional Memory** field reserves extra memory beyond the JVM heap for Python native allocations and Java direct memory. On Kubernetes, this value is added to the worker pod's memory request and limit. For detailed guidance on dispatcher-level limits, per-worker overhead, and multi-dispatcher tuning, see the [worker heap size guide](../pq-controller/worker-heap-size.md). ## Data buffer pool The data buffer pool is an internal pool of fixed-size binary buffers (64 KB each by default) used by `db_dis`, `db_ltds`, `db_tdcp`, and query worker processes to read, write, and cache Deephaven-format binary data. It is the most significant driver of merge performance and a critical resource for the Data Import Server. ### How sizing works The pool size is clamped automatically between minimum and maximum ratios of the JVM heap: - **Minimum**: 10% of heap (`DataBufferConfiguration.minPoolToHeapSizeRatio=0.1`) - **Maximum (heap buffers)**: 60% of heap (`DataBufferConfiguration.heapMaxPoolToHeapSizeRatio=0.6`) - **Maximum (direct buffers)**: 200% of heap (`DataBufferConfiguration.directMaxPoolToHeapSizeRatio=2.0`) The configured pool size is rounded down to the nearest multiple of the buffer block size (64 KB). If the configured size falls outside the ratio bounds, it is clamped to the nearest bound. ### Default pool sizes The per-service defaults (from hostconfig) are set via the `DataBufferPool.sizeInBytes` property: | Service | Default pool size | | ------------------ | --------------------------------------------------------------- | | `db_dis` | 10 GB (`10737418240` bytes) | | `db_tdcp` | 4 GB (`4294967296` bytes) | | `db_ltds` | 4 GB (`4294967296` bytes) | | Workers (PQ/merge) | Set by the `dataBufferPoolToHeapSizeRatio` field in PQ settings | ### Configuring pool size To change the pool size for a system process, override the property in `hostconfig`: ```bash db_dis) EXTRA_ARGS="$EXTRA_ARGS -j -DDataBufferConfiguration.poolSize=16g" ;; ``` For Persistent Queries, set **Data Memory Ratio** in the PQ editor's advanced options. This ratio is applied to the PQ's heap at startup. The Controller enforces minimum and maximum ratio bounds defined in the Controller's reloadable configuration. > [!NOTE] > `DataBufferPool.sizeInBytes` is deprecated as of Deephaven v1.20200331. Use `DataBufferConfiguration.poolSize` for new configuration. ### DIS sizing guidance The Data Import Server must hold one buffer per column file per open partition (each partition with an active tailer connection). A practical rule of thumb is: > `pool size ≥ nPartitions × nColumns × 1.2 × 64 KB` If the DIS exhausts the pool and cannot free buffers through synchronous cleanup, the process will terminate. ### LTDS sizing guidance The LocalTableDataServer (`db_ltds`) serves historical table data to query workers. Its buffer pool must hold enough buffers to satisfy concurrent read requests from all connected workers. On deployments with many simultaneous queries reading large historical tables, the default 4 GB pool can become a bottleneck. A practical starting point is to size the pool to match the peak number of concurrent worker read streams, then double the DIS sizing rule-of-thumb: > `pool size ≥ nConcurrentWorkers × nColumns × averageReadDepth × 64 KB` If you observe repeated `Failed to take()` log lines from `db_ltds` specifically, increase `DataBufferConfiguration.poolSize` for that process and raise `-Xmx` proportionally to stay within the 60% heap ratio bound. ### Merge worker sizing guidance For merge workers, the pool is the primary driver of heap usage. The recommended sizing formula is: > `buffer pool size = max_column_file_size × nWritingThreads / nOutputPartitions / 0.85` > > `heap size = 2 × buffer pool size` For more context on merge memory planning, see [Merge optimization](../../data-guide/merge-optimization.md). ### Configuration properties The following properties influence data buffer pool behavior. Typically only `DataBufferConfiguration.poolSize` and `iris.concurrentWriteThreads` need to be changed from their defaults. | Property | Default | Description | | ------------------------------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | `DataBufferConfiguration.bufferSize` | `65536` (64 KB) | Size of each buffer. Supports XmX-style units (`64k`, `1m`). Must be a positive integer less than 2^30 bytes. | | `DataBufferConfiguration.useDirectMemory` | `false` | Allocate buffers in direct (off-heap) memory instead of heap memory. | | `DataBufferConfiguration.poolEnabled` | `true` | Whether to use a pool. Disabling the pool is not advisable. | | `DataBufferConfiguration.poolSize` | ~5.6 GB (via `DataBufferPool.sizeInBytes`; overridden per service in hostconfig) | Total pool size. Supports XmX-style units (`10g`). Clamped between min/max ratio bounds. | | `DataBufferConfiguration.minPoolToHeapSizeRatio` | `0.1` | Minimum pool-to-heap ratio. | | `DataBufferConfiguration.heapMaxPoolToHeapSizeRatio` | `0.6` | Maximum pool-to-heap ratio when using heap buffers. | | `DataBufferConfiguration.directMaxPoolToHeapSizeRatio` | `2.0` | Maximum pool-to-heap ratio when using direct buffers. | | `DataBufferConfiguration.poolCleanupThresholdRatioUsed` | `0.9` | Pool occupancy ratio that triggers concurrent cleanup. | | `DataBufferConfiguration.poolCleanupTargetRatioUsed` | `0.6` | Target occupancy ratio after cleanup. | | `DataBufferConfiguration.poolCleanupIntervalMillis` | `60000` | Interval between scheduled cleanup checks (ms). | | `DataBufferConfiguration.poolClockIntervalMillis` | `10000` | Interval between logical clock ticks used for buffer timestamping (ms). | ### Key log lines Monitor these log messages to identify pool pressure before it becomes a problem: - `AutoReclaimingObjectPool-DataBufferPool: Failed to take() an item for thread=, initiating synchronous cleanup` — A thread could not acquire a buffer lock-free and is blocking to reclaim space. The pool is under pressure; consider increasing pool size. - `AutoReclaimingObjectPool-DataBufferPool: Unable to take() an item for thread=, yielding (/)` — All buffers are actively in use; the thread is yielding the CPU. Increase pool size. - `AutoReclaimingObjectPool-DataBufferPool: Unable to take() an item for thread= after yields` — Fatal: the pool is exhausted after multiple yield attempts. This is followed by an `ObjectPoolExhaustedError`. The process will terminate. ## Related documentation - [Worker heap size](../pq-controller/worker-heap-size.md) - [Out-of-memory errors](../../resources/faq/out-of-memory.md) - [Memory tuning for persistent queries](../../resources/faq/memory-tuning.md) - [Merge optimization](../../data-guide/merge-optimization.md) - [Metrics and monitoring](./metrics-and-monitoring.md) - [Kubernetes service resource configuration](../kubernetes/kubernetes-configuration-settings.md#configuring-worker-process-resources) - [Tailer memory properties](../configuration/data-tailer.md#memory-properties) - [Heap dump on assertion failure](../optional-settings/heap-dump.md)