---
title: JVM tuning
sidebar_label: JVM tuning
---

This guide covers JVM configuration for Deephaven services and workers, including garbage collection, heap sizing, and performance-related flags.

> [!NOTE]
> For adjusting heap size for individual PQs, see [Memory tuning FAQ](../../resources/faq/memory-tuning.md). For the Deephaven memory model, data buffer pool configuration, and understanding what drives memory usage, see [Memory management](./memory-management.md).

## Overview

Deephaven runs multiple Java processes with different configuration methods:

| Process type | Examples                     | Configuration method |
| ------------ | ---------------------------- | -------------------- |
| Services     | DIS, merge, controller, TDCP | Hostconfig file      |
| Workers      | PQ workers, consoles         | Processing profiles  |

## Heap sizing

### Service heap configuration

Configure service heap sizes in the [hostconfig file](../configuration/configuration-file-locations-overview.md#hostconfig) (`/etc/sysconfig/illumon`) using `EXTRA_ARGS`:

```bash
db_dis)
    EXTRA_ARGS="$EXTRA_ARGS -j -Xms8g -j -Xmx8g"
    ;;
db_merge)
    EXTRA_ARGS="$EXTRA_ARGS -j -Xms16g -j -Xmx16g"
    ;;
```

See [Java process launch configuration](../configuration/java-process-launch.md) for details on hostconfig syntax.

### Worker heap configuration

Worker `-Xmx` is determined by the heap size configured per Persistent Query or console. Remote processing profiles can configure `-Xms` (initial heap) — see [Setting -Xms for workers](../pq-controller/remote-processing-profiles.md#setting--xms-for-workers).

## Garbage collection

### GC profiles

Deephaven provides built-in GC profiles via [remote processing profiles](../pq-controller/remote-processing-profiles.md):

| Profile                 | GC type                   | Best for                                 | Java version           |
| ----------------------- | ------------------------- | ---------------------------------------- | ---------------------- |
| `G1 GC`                 | Garbage First             | Most workloads                           | Java 11+ (recommended) |
| `CMS GC`                | Concurrent Mark Sweep     | Legacy                                   | Java 8 only            |
| `G1 MarkStackSize 128M` | G1 with larger mark stack | Complex queries with allocation failures | Java 11+               |

**Set default profile:**

```properties
RemoteQueryDispatcher.defaultJVMProfile=G1 GC
```

### G1 GC tuning

G1 GC is recommended for Java 11+ deployments.

**Key parameters:**

| Parameter                            | Purpose                    | Notes                                  |
| ------------------------------------ | -------------------------- | -------------------------------------- |
| `-XX:MaxGCPauseMillis`               | Target max pause time      | Default: 200ms; tune based on workload |
| `-XX:ParallelGCThreads`              | Threads for STW phases     | Tune based on available cores          |
| `-XX:ConcGCThreads`                  | Concurrent marking threads | Tune based on GC behavior              |
| `-XX:G1HeapRegionSize`               | Heap region size           | Auto-selected; rarely needs tuning     |
| `-XX:InitiatingHeapOccupancyPercent` | When to start marking      | Default: 45                            |

**Custom G1 profile example:**

```properties
RemoteProcessingRequestProfile.custom.G1Tuned.include.1=G1 GC
RemoteProcessingRequestProfile.custom.G1Tuned.jvmParameter.maxPause=-XX:MaxGCPauseMillis=200
RemoteProcessingRequestProfile.custom.G1Tuned.jvmParameter.gcThreads=-XX:ParallelGCThreads=4
RemoteProcessingRequestProfile.custom.G1Tuned.jvmParameter.concThreads=-XX:ConcGCThreads=2
```

### GC logging

Enable GC logging to diagnose performance issues.

**Java 11+ (Unified Logging):**

```properties
RemoteProcessingRequestProfile.custom.GCLogging.jvmParameter.gcLog=-Xlog:gc*:file=/var/log/deephaven/gc_%p.log:time,uptime,level,tags:filecount=5,filesize=100m
```

The `-Xlog` format is `what:output:decorators:output-options`:

- **`gc*`** — Log all GC-related messages
- **`file=...gc_%p.log`** — Write to file (`%p` = process ID)
- **`time,uptime,level,tags`** — Include timestamps and log level
- **`filecount=5,filesize=100m`** — Rotate across 5 files, 100 MB each

**Analyze GC logs:**

- **GCViewer**: Open source tool for visualizing GC logs
- **GCEasy**: Online GC log analyzer
- Look for: full GC frequency, pause times, heap after GC

## Common JVM flags

### Memory flags

| Flag                      | Purpose             | Example                       |
| ------------------------- | ------------------- | ----------------------------- |
| `-Xmx`                    | Maximum heap        | `-Xmx32g`                     |
| `-Xms`                    | Initial heap        | `-Xms32g`                     |
| `-XX:MaxDirectMemorySize` | Direct memory limit | `-XX:MaxDirectMemorySize=16g` |
| `-XX:MaxMetaspaceSize`    | Metaspace limit     | `-XX:MaxMetaspaceSize=512m`   |

### Performance flags

| Flag                          | Purpose              | Example                  |
| ----------------------------- | -------------------- | ------------------------ |
| `-XX:+AlwaysPreTouch`         | Pre-touch heap pages | Reduces startup variance |
| `-XX:+UseStringDeduplication` | Deduplicate strings  | Saves memory             |
| `-XX:CICompilerCount`         | JIT compiler threads | 2-4                      |

### Diagnostic flags

| Flag                              | Purpose             | Example               |
| --------------------------------- | ------------------- | --------------------- |
| `-XX:+HeapDumpOnOutOfMemoryError` | Dump heap on OOM    | Always enable         |
| `-XX:HeapDumpPath`                | Heap dump location  | `/tmp/heapdump.hprof` |
| `-XX:+PrintFlagsFinal`            | Print all JVM flags | For debugging         |

**Example diagnostic profile:**

```properties
RemoteProcessingRequestProfile.custom.Debug.include.1=G1 GC
RemoteProcessingRequestProfile.custom.Debug.jvmParameter.heapDump=-XX:+HeapDumpOnOutOfMemoryError
RemoteProcessingRequestProfile.custom.Debug.jvmParameter.heapDumpPath=-XX:HeapDumpPath=/tmp
```

## Troubleshooting

### Out of memory errors

**Heap space:**

```
java.lang.OutOfMemoryError: Java heap space
```

**Causes and solutions:**

| Cause           | Check                                                   | Solution          |
| --------------- | ------------------------------------------------------- | ----------------- |
| Heap too small  | Heap usage near max                                     | Increase `-Xmx`   |
| Memory leak     | [Heap dump](../optional-settings/heap-dump.md) analysis | Fix leak, restart |
| Large data sets | Query analysis                                          | Optimize queries  |

### Direct buffer memory

```
java.lang.OutOfMemoryError: Direct buffer memory
```

**Solution:** Increase `-XX:MaxDirectMemorySize`.

### Long GC pauses

**Symptoms:**

- UI freezes
- Query timeouts
- Log shows long pause times

**Solutions:**

| Cause          | Check                  | Solution                              |
| -------------- | ---------------------- | ------------------------------------- |
| Heap too large | Pause times in logs    | Reduce heap or tune G1                |
| Full GCs       | GC log analysis        | Increase heap, reduce allocation rate |
| Fragmentation  | G1 evacuation failures | Increase heap regions                 |

### G1 allocation failures

```
[Full GC (Allocation Failure) ...]
```

**Solution:** Use the `G1 MarkStackSize 128M` profile or add:

```properties
RemoteProcessingRequestProfile.custom.G1Large.jvmParameter.markStack=-XX:MarkStackSize=128M
```

## Quick reference

### JVM flags by purpose

| Purpose     | Flags                                                                |
| ----------- | -------------------------------------------------------------------- |
| Heap sizing | `-Xms`, `-Xmx`, `-XX:MaxDirectMemorySize`                            |
| G1 tuning   | `-XX:MaxGCPauseMillis`, `-XX:ParallelGCThreads`, `-XX:ConcGCThreads` |
| Diagnostics | `-XX:+HeapDumpOnOutOfMemoryError`, `-Xlog:gc*`                       |

### Profile selection guide

| Workload             | Profile                 | Notes                         |
| -------------------- | ----------------------- | ----------------------------- |
| General purpose      | `G1 GC`                 | Default for Java 11+          |
| High allocation rate | `G1 MarkStackSize 128M` | If seeing allocation failures |
| Legacy Java 8        | `CMS GC`                | Deprecated                    |
| Custom requirements  | Create custom profile   | Include base profile          |

### Heap sizing considerations

Worker heap depends on:

- **Query complexity**: Simple queries vs. complex joins/aggregations
- **Data volume**: Size of tables accessed during operations
- **Concurrent operations**: Number of simultaneous operations

Start with the sizing guidelines in [Memory management](./memory-management.md) and adjust based on monitoring.

## Related documentation

- [Remote processing profiles](../pq-controller/remote-processing-profiles.md)
- [Memory management](./memory-management.md)
- [CPU optimization](./cpu-optimization.md)
- [Troubleshoot Java processes](../troubleshooting/troubleshooting-java.md)
- [Memory tuning FAQ](../../resources/faq/memory-tuning.md)
- [Performance tuning overview](./overview.md)