---
title: Memory management
---

This guide explains Deephaven's memory model and provides strategies for optimizing memory configuration to achieve better performance, stability, and resource utilization.

> [!NOTE]
> For troubleshooting memory-related problems (`OutOfMemoryError`s, memory leaks, heap dumps), see [Troubleshoot Java processes](../troubleshooting/troubleshooting-java.md). For the mechanics of applying JVM flags to services and workers (Remote Processing Profiles, hostconfig), see [JVM tuning](./jvm-tuning.md).

## Understanding the Deephaven memory model

Deephaven uses a hybrid memory architecture that leverages both JVM heap memory and off-heap direct memory.

### JVM heap memory (on-heap)

The JVM heap stores Java objects created by the application, including:

- **Query state and metadata**: Table references, operation graphs, and query execution state
- **Application objects**: Controller, dispatcher, and worker process objects
- **Temporary computation results**: Intermediate values during query evaluation
- **Data buffer pool (optional)**: When `DataBufferConfiguration.useDirectMemory=false`

Heap size is controlled by the `-Xmx` JVM parameter. For example, `-Xmx16g` allocates a maximum of 16 GB of heap memory.

### Direct memory (off-heap)

Direct memory is native (non-JVM) memory used for:

- **Table data storage**: When `DataBufferConfiguration.useDirectMemory=true`, columnar table data is stored in direct memory buffers.
- **Network I/O buffers**: Data transfer between workers and Data Import Servers.
- **Binary log data**: Tailer and Data Import Server buffers for streaming data.

Direct memory size is controlled by the `-XX:MaxDirectMemorySize` JVM parameter.

**Key difference:**

- **Heap memory** is managed by the JVM garbage collector.
- **Direct memory** is manually managed by the application (not subject to GC pauses).

### The data buffer pool

Deephaven stores columnar table data in a data buffer pool, which can reside in either heap or direct memory.

**Configuration:**

```properties
DataBufferConfiguration.useDirectMemory=true   # Use direct memory (recommended for large data)
DataBufferConfiguration.poolSize=32g           # Total size of the buffer pool
```

**Direct vs. heap memory for table data:**

| Mode   | Max pool size | Trade-off                                                             |
| ------ | ------------- | --------------------------------------------------------------------- |
| Heap   | 60% of heap   | Simpler configuration, but limited capacity                           |
| Direct | 200% of heap  | Larger capacity, but requires `-XX:MaxDirectMemorySize` configuration |

See [Data buffer pool configuration](../ops-guide/process-memory.md#configuration-properties) for complete details.

**Default pool size limits:**

- **Heap-based buffers**: Pool size limited to 60% of heap.
- **Direct memory buffers**: Pool size can exceed heap size (controlled by `DataBufferConfiguration.directMaxPoolToHeapSizeRatio`).

### Garbage collection and memory

GC reclaims heap memory occupied by objects that are no longer referenced.

**GC impact on performance:**

- **Stop-the-world pauses**: Most GC phases pause all application threads.
- **CPU overhead**: GC threads consume CPU while scanning and compacting.
- **Frequency vs. duration trade-off**: More frequent GCs mean shorter pauses but higher overhead.

**Deephaven GC profiles** via [remote processing profiles](../pq-controller/remote-processing-profiles.md):

- **G1 GC** (Garbage First): Recommended for Java 11+ and large heaps (> 4 GB).
- **CMS GC** (Concurrent Mark Sweep): Legacy collector for Java 8.

## Monitoring memory

### Monitoring heap and GC metrics

Enable process metrics to track memory trends:

```properties
IrisLogDefaults.writeDatabaseProcessMetrics=true
```

Query memory metrics from internal tables:

```python
memory_metrics = (
    db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
    .where(["Date = today()", "Name.startsWith(`Memory-`)"])
    .view(["Timestamp", "Name", "Last", "Min", "Max", "Avg"])
)
```

```groovy
memoryMetrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2")
    .where("Date = today()", "Name.startsWith(`Memory-`)")
    .view("Timestamp", "Name", "Last", "Min", "Max", "Avg")
```

**Key metrics:**

- `Memory-Heap.Used` — Track typical usage to validate heap sizing
- `Memory-GC-G1.*` — Monitor GC frequency and pause times

See [Process metrics](../internal-tables/process-metrics.md) for complete metric list.

## Memory tuning strategies

### Sizing heap and direct memory

#### Heap sizing guidelines

Worker heap requirements vary widely based on query complexity, data volumes, and concurrent operations. Start conservatively and increase based on monitoring.

Workers request heap size at startup via **Advanced Settings** → **Heap Size** in the [Web IDE](../../interfaces/web/code-studio.md).

#### Direct memory sizing

For workers with direct memory data buffers:

```properties
-XX:MaxDirectMemorySize=32g
DataBufferConfiguration.useDirectMemory=true
DataBufferConfiguration.poolSize=28g
```

Leave headroom for network buffers and other direct memory allocations beyond the buffer pool.

**For tailers and Data Import Servers:** See [Tailer memory properties](../configuration/data-tailer.md) for sizing details.

### Choosing and tuning garbage collectors

#### G1 GC (Garbage First)

**Recommended for:** Most Deephaven deployments on Java 11+.

```properties
RemoteQueryDispatcher.defaultJVMProfile=G1 GC
```

**Tuning G1 GC:**

```properties
RemoteProcessingRequestProfile.custom.G1Tuned.include.1=G1 GC
RemoteProcessingRequestProfile.custom.G1Tuned.jvmParameter.maxGCPause=-XX:MaxGCPauseMillis=200
RemoteProcessingRequestProfile.custom.G1Tuned.jvmParameter.gcThreads=-XX:ParallelGCThreads=4
RemoteProcessingRequestProfile.custom.G1Tuned.jvmParameter.concGcThreads=-XX:ConcGCThreads=2
```

- `MaxGCPauseMillis` — Target maximum pause time
- `ParallelGCThreads` — Threads for stop-the-world phases
- `ConcGCThreads` — Threads for concurrent marking

#### CMS GC (Concurrent Mark Sweep)

**For:** Legacy Java 8 deployments only. Deprecated in Java 9+.

```properties
RemoteQueryDispatcher.defaultJVMProfile=CMS GC
```

## Quick reference

### G1 GC key parameters

| Parameter                 | Default | Notes                                          |
| ------------------------- | ------- | ---------------------------------------------- |
| `-Xms` / `-Xmx`           | N/A     | Setting `-Xms` = `-Xmx` avoids resize overhead |
| `-XX:MaxDirectMemorySize` | N/A     | Size for data buffers                          |
| `-XX:MaxGCPauseMillis`    | 200ms   | Tune based on workload                         |
| `-XX:G1HeapRegionSize`    | Auto    | Auto-selected; rarely needs tuning             |

### Memory monitoring approach

Appropriate memory thresholds are system-dependent.

**How to establish your baseline:**

1. Query heap metrics during normal operations:
   ```python
   heap_metrics = (
       db.live_table("DbInternal", "ProcessMetricsLogCoreV2")
       .where(["Date = today()", "Name in `Memory-Heap.Used`, `Memory-Heap.Max`"])
       .view(["Timestamp", "ProcessInfoId", "Name", "Last"])
   )
   ```
   ```groovy
   heapMetrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2")
       .where("Date = today()", "Name in `Memory-Heap.Used`, `Memory-Heap.Max`")
       .view("Timestamp", "ProcessInfoId", "Name", "Last")
   ```
2. Note typical heap usage after GC cycles
3. Set alerts when usage consistently exceeds your observed normal range

**What to look for:**

- **Heap usage after GC**: If usage trends upward over time, investigate for memory pressure or leaks
- **GC pause time**: Query `Memory-GC-*` metrics. Increasing pauses affect query responsiveness.
- **Full GC frequency**: Look for `Full GC` entries in GC logs. Frequent full GCs indicate heap sizing issues.
- **Direct memory usage**: If using direct memory for data buffers, monitor with `jcmd <pid> VM.native_memory`

### Common memory issues

| Symptom                                  | Likely cause                 | Solution                           |
| ---------------------------------------- | ---------------------------- | ---------------------------------- |
| `OutOfMemoryError: Java heap space`      | Heap exhausted               | Increase `-Xmx`                    |
| `OutOfMemoryError: Direct buffer memory` | Direct memory exhausted      | Increase `-XX:MaxDirectMemorySize` |
| Long GC pauses                           | Heap too large or fragmented | Reduce heap or tune G1             |
| Frequent full GCs                        | Heap too small               | Increase heap size                 |

## Related documentation

- [Performance tuning overview](./overview.md)
- [CPU optimization](./cpu-optimization.md)
- [JVM tuning](./jvm-tuning.md)
- [Remote processing profiles](../pq-controller/remote-processing-profiles.md)
- [Troubleshoot Java processes](../troubleshooting/troubleshooting-java.md)
- [Process metrics](../internal-tables/process-metrics.md)