---
title: Resource considerations for scaling
sidebar_label: Resource considerations
---

When scaling Deephaven across multiple servers, careful resource planning ensures optimal performance and prevents bottlenecks. This guide covers memory, CPU, storage, and network considerations for multi-server bare metal installations, with a focus on query servers and data import servers — the most resource-intensive components in a Deephaven deployment.

For an overview of multi-server architecture and server types, see [Scaling to multiple servers](./scaling.md).

## Memory planning

Memory is typically the most critical resource for Deephaven deployments. Workers consume heap memory for query operations, and the JVM uses additional off-heap memory for internal structures.

### JVM heap allocation

Each dispatcher controls how much memory its workers can consume through these properties:

| Property                                             | Description                                             |
| ---------------------------------------------------- | ------------------------------------------------------- |
| `RemoteQueryDispatcher.maxTotalQueryProcessorHeapMB` | Total heap available for all workers on this dispatcher |
| `RemoteQueryDispatcher.maxPerWorkerHeapMB`           | Maximum heap for any single worker                      |

For example, a query server with 512 GB of RAM might allocate 400 GB total for workers:

```properties
RemoteQueryDispatcher.maxTotalQueryProcessorHeapMB=409600
RemoteQueryDispatcher.maxPerWorkerHeapMB=131072
```

For detailed configuration guidance, including more examples and related properties, see [Controlling query worker heap size](../pq-controller/worker-heap-size.md).

### Accounting for JVM overhead

The JVM uses memory beyond the heap for garbage collection metadata, metaspace, direct buffers, and compiled code. Deephaven accounts for this with overhead properties:

| Property                                         | Description                                              | Default |
| ------------------------------------------------ | -------------------------------------------------------- | ------- |
| `RemoteQueryDispatcher.memoryOverheadMB`         | Fixed overhead added to each worker's memory calculation | 500     |
| `RemoteQueryDispatcher.memoryOverheadMultiplier` | Percentage of heap added as overhead                     | 0.05    |

The dispatcher calculates a worker's memory footprint as:

```
memory = heap + ceil(heap × multiplier) + overheadMB
```

For a 32 GB worker with default settings:

- Heap: 32,768 MB
- Multiplier overhead: ceil(32,768 × 0.05) = ceil(1,638.4) = 1,639 MB
- Fixed overhead: 500 MB
- **Total: 34,907 MB**

If workers are unexpectedly killed by the OS out-of-memory killer, increase these overhead values. Start with small increments and observe the impact.

### Reserved memory for the operating system

The dispatcher can check available system memory before starting workers:

| Property                                               | Description                         | Default |
| ------------------------------------------------------ | ----------------------------------- | ------- |
| `RemoteQueryDispatcher.reservedAvailableMemoryMB`      | Memory reserved for non-admin users | 2048    |
| `RemoteQueryDispatcher.adminReservedAvailableMemoryMB` | Memory reserved for admin users     | 1024    |

When set to a non-negative value, the dispatcher reads `/proc/meminfo` and refuses to start workers if doing so would leave less than the reserved amount available.

### Memory planning guidelines

When planning memory allocation for a server:

1. **Account for Deephaven service processes** — See [Installation planning](../installation/installation-planning.md#considerations-for-infrastructure-nodes) for documented memory requirements per service.
2. **Plan for concurrent workers** — Each worker and Persistent Query consumes at least 8 GB of heap. See the [capacity formula in Installation planning](../installation/installation-planning.md#considerations-for-query-and-merge-servers).
3. **Reserve memory for the operating system** — File system caches, kernel operations, and other system processes need memory.
4. **Consider peak usage** — Size for expected peak load, not average usage.

## CPU planning

CPU resources affect query execution speed, garbage collection efficiency, and overall system responsiveness.

### CPU allocation

The Deephaven engine parallelizes many operations across available cores. Workers with larger heaps typically benefit from more cores to handle garbage collection and parallel query execution.

[Installation planning](../installation/installation-planning.md#considerations-for-query-and-merge-servers) recommends allocating at least one CPU core per expected Persistent Query. Monitor CPU utilization to determine if your workload requires additional cores.

### NUMA considerations

Modern multi-socket servers use Non-Uniform Memory Access (NUMA), where each CPU socket has faster access to its local memory than to memory attached to other sockets. For large-memory Deephaven servers:

- **Memory access patterns matter** — A worker whose heap spans multiple NUMA nodes may experience inconsistent memory latency.
- **Garbage collection can be impacted** — GC pause times can increase when the collector must access remote memory.
- **Operating system policies control memory allocation** — Linux NUMA policies determine how memory is distributed across nodes.

For servers with multiple CPU sockets, consider:

- Sizing workers so their heap fits within a single NUMA node.
- Using NUMA-aware memory policies for Deephaven processes.
- Monitoring memory access patterns with tools like `numastat`.

See [Linux NUMA memory policy documentation](https://www.kernel.org/doc/html/latest/admin-guide/mm/numa_memory_policy.html) for background on NUMA architecture.

### Process isolation and CPU affinity

For performance-critical deployments, you may want to isolate Deephaven processes to specific CPU cores, preventing them from competing with other workloads and reducing context switching. This is particularly useful when:

- Running multiple services on the same physical server.
- Achieving consistent, predictable latency for real-time workloads.
- Preventing noisy-neighbor effects between workers.

Linux provides several tools for CPU affinity and isolation:

- **`taskset`** — Binds a process to specific CPU cores at startup or runtime.
- **`numactl`** — Controls NUMA memory policy and CPU binding.
- **`cset`** (cpuset) — Creates isolated CPU sets for groups of processes.

For detailed guidance on these tools, see:

- [Red Hat: Configuring CPU affinity and NUMA policies](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/managing_monitoring_and_updating_the_kernel/assembly_configuring-cpu-affinity-and-numa-policies-using-systemd_managing-monitoring-and-updating-the-kernel)
- [cpuset (cset) documentation](https://github.com/lpechacek/cpuset)

### Garbage collection tuning

Large-heap workers benefit from garbage collection tuning:

- **G1 GC** is the default collector for Java 11 and later; CMS GC is the default for Java 8.
- **GC thread count** scales with available cores; very large heaps may benefit from explicit tuning.
- Use [JVM profiles](../pq-controller/remote-processing-profiles.md) to apply consistent GC settings across workers.

## Storage I/O planning

Storage performance directly affects query speed and data ingestion throughput.

### Storage by server type

| Server type  | Storage requirement              | Recommendation                            |
| ------------ | -------------------------------- | ----------------------------------------- |
| Query server | Historical data (`/db/Systems/`) | NFS-mounted shared storage                |
| Query server | Temporary files, caches          | Local SSD for `/db` mount                 |
| DIS          | Intraday data (`/db/Intraday/`)  | Low-latency direct-attached SSD           |
| DIS          | Binary logs                      | Local storage, sized for at least one day |

For detailed sizing guidance, see [Installation planning: Storage](../installation/installation-planning.md#storage).

## Network planning

Network bandwidth affects data access speed and inter-server communication. Larger installations use 10 GbE networks for NFS access to storage (see [Scaling to multiple servers](./scaling.md#the-storage-layer)).

### Key recommendations

- Place Deephaven servers and storage on the same network segment.
- Minimize network hops between query servers and storage.
- If your network supports jumbo frames, ensure consistent MTU configuration across all servers, NICs, and switches.

## Query server sizing

Query servers run the dispatcher and workers that execute user queries and Persistent Queries.

### Sizing guidelines

Query server memory requirements depend on the number of concurrent workers and their heap sizes. [Installation planning](../installation/installation-planning.md#considerations-for-query-and-merge-servers) provides a formula for calculating memory needs:

- **TDCP:** 4 GB
- **Per worker/PQ:** At least 8 GB each
- **CPU:** At least 1 core per Persistent Query

For minimum cluster requirements (8 cores, 128 GB RAM), see [Installation planning](../installation/installation-planning.md#cpu--ram).

Scale horizontally by adding query servers rather than vertically scaling a single server. The [automated server selection](../pq-controller/automated-server-selection.md) feature distributes workers across available dispatchers.

### Dispatcher configuration for multiple query servers

When running multiple query servers, configure the controller's server selection provider to balance load:

```properties
PersistentQueryController.ServerSelectionProvider=com.illumon.iris.controller.SimpleServerSelectionProvider
SimpleServerSelectionProvider.ActiveGroups=AutoQuery
SimpleServerSelectionProvider.Group.AutoQuery.ServerClass=Query
SimpleServerSelectionProvider.Group.AutoQuery.DefaultHeapMBPerServer=262144
```

See [Automated server selection](../pq-controller/automated-server-selection.md) for complete configuration details.

## Data Import Server sizing

DIS servers handle real-time data ingestion and serve intraday data to workers. The DIS process requires **16 GB** of memory according to [Installation planning](../installation/installation-planning.md#considerations-for-infrastructure-nodes).

### Scaling DIS capacity

For high-volume data ingestion:

- Add additional DIS servers to distribute load.
- Use [data routing](../configuration/yaml.md) to direct different data streams to different DIS instances.

See [Add a dedicated DIS for data ingestion](./add-dis-server.md) for instructions.

## Administrative server sizing

Administrative servers run orchestration and authentication processes. They have lower resource requirements than query or DIS servers and can often be co-located with other services.

### Memory requirements

The following services typically run on an administrative server:

| Service                     | Memory    |
| --------------------------- | --------- |
| Persistent Query Controller | 4 GB      |
| Configuration Server        | 4 GB      |
| Web API Service             | 4 GB      |
| Authentication Server       | 1 GB      |
| ACL Write Server            | 1 GB      |
| Status Dashboard            | 2 GB      |
| **Subtotal**                | **16 GB** |

> [!NOTE]
> This excludes DIS (16 GB), Log Aggregator (4 GB), and Tailer (4 GB), which run on infrastructure or DIS nodes. See [Installation planning](../installation/installation-planning.md#considerations-for-infrastructure-nodes) for the complete breakdown totaling 40 GB.

Administrative servers do not require access to historical data mounts or high-speed storage.

## Monitoring resource usage

After deployment, monitor resource usage to identify bottlenecks:

- **Memory:** Watch for OOM kills in system logs; monitor dispatcher heap allocation.
- **CPU:** Monitor for sustained high CPU usage or GC pressure.
- **Storage I/O:** Track read/write latency and throughput.
- **Network:** Monitor bandwidth utilization between servers and storage.

The [Status Dashboard](../status-dashboard.md) and [metrics and monitoring](../ops-guide/metrics-and-monitoring.md) documentation describe available monitoring options.

## Related documentation

- [Automated server selection](../pq-controller/automated-server-selection.md)
- [Controlling query worker heap size](../pq-controller/worker-heap-size.md)
- [Dispatcher configuration](../pq-controller/dispatcher.md)
- [Installation planning](../installation/installation-planning.md)
- [Metrics and monitoring](../ops-guide/metrics-and-monitoring.md)
- [Scaling to multiple servers](./scaling.md)