Resource considerations for scaling

When scaling Deephaven across multiple servers, careful resource planning ensures optimal performance and prevents bottlenecks. This guide covers memory, CPU, storage, and network considerations for multi-server bare metal installations, with a focus on query servers and data import servers — the most resource-intensive components in a Deephaven deployment.

For an overview of multi-server architecture and server types, see Scaling to multiple servers.

Memory planning

Memory is typically the most critical resource for Deephaven deployments. Workers consume heap memory for query operations, and the JVM uses additional off-heap memory for internal structures.

JVM heap allocation

Each dispatcher controls how much memory its workers can consume through these properties:

For example, a query server with 512 GB of RAM might allocate 400 GB total for workers:

For detailed configuration guidance, including more examples and related properties, see Controlling query worker heap size.

Accounting for JVM overhead

The JVM uses memory beyond the heap for garbage collection metadata, metaspace, direct buffers, and compiled code. Deephaven accounts for this with overhead properties:

The dispatcher calculates a worker's memory footprint as:

For a 32 GB worker with default settings:

• Heap: 32,768 MB
• Multiplier overhead: ceil(32,768 × 0.05) = ceil(1,638.4) = 1,639 MB
• Fixed overhead: 500 MB
• Total: 34,907 MB

If workers are unexpectedly killed by the OS out-of-memory killer, increase these overhead values. Start with small increments and observe the impact.

Reserved memory for the operating system

The dispatcher can check available system memory before starting workers:

When set to a non-negative value, the dispatcher reads /proc/meminfo and refuses to start workers if doing so would leave less than the reserved amount available.

Memory planning guidelines

When planning memory allocation for a server:

1. Account for Deephaven service processes — See Installation planning for documented memory requirements per service.
2. Plan for concurrent workers — Each worker and Persistent Query consumes at least 8 GB of heap. See the capacity formula in Installation planning.
3. Reserve memory for the operating system — File system caches, kernel operations, and other system processes need memory.
4. Consider peak usage — Size for expected peak load, not average usage.

CPU planning

CPU resources affect query execution speed, garbage collection efficiency, and overall system responsiveness.

CPU allocation

The Deephaven engine parallelizes many operations across available cores. Workers with larger heaps typically benefit from more cores to handle garbage collection and parallel query execution.

Installation planning recommends allocating at least one CPU core per expected Persistent Query. Monitor CPU utilization to determine if your workload requires additional cores.

NUMA considerations

Modern multi-socket servers use Non-Uniform Memory Access (NUMA), where each CPU socket has faster access to its local memory than to memory attached to other sockets. For large-memory Deephaven servers:

• Memory access patterns matter — A worker whose heap spans multiple NUMA nodes may experience inconsistent memory latency.
• Garbage collection can be impacted — GC pause times can increase when the collector must access remote memory.
• Operating system policies control memory allocation — Linux NUMA policies determine how memory is distributed across nodes.

For servers with multiple CPU sockets, consider:

• Sizing workers so their heap fits within a single NUMA node.
• Using NUMA-aware memory policies for Deephaven processes.
• Monitoring memory access patterns with tools like numastat.

See Linux NUMA memory policy documentation for background on NUMA architecture.

Process isolation and CPU affinity

For performance-critical deployments, you may want to isolate Deephaven processes to specific CPU cores, preventing them from competing with other workloads and reducing context switching. This is particularly useful when:

• Running multiple services on the same physical server.
• Achieving consistent, predictable latency for real-time workloads.
• Preventing noisy-neighbor effects between workers.

Linux provides several tools for CPU affinity and isolation:

• taskset — Binds a process to specific CPU cores at startup or runtime.
• numactl — Controls NUMA memory policy and CPU binding.
• cset (cpuset) — Creates isolated CPU sets for groups of processes.

For detailed guidance on these tools, see:

• Red Hat: Configuring CPU affinity and NUMA policies
• cpuset (cset) documentation

Garbage collection tuning

Large-heap workers benefit from garbage collection tuning:

• G1 GC is the default collector for Java 11 and later; CMS GC is the default for Java 8.
• GC thread count scales with available cores; very large heaps may benefit from explicit tuning.
• Use JVM profiles to apply consistent GC settings across workers.

Storage I/O planning

Storage performance directly affects query speed and data ingestion throughput.

Storage by server type

For detailed sizing guidance, see Installation planning: Storage.

Network planning

Network bandwidth affects data access speed and inter-server communication. Larger installations use 10 GbE networks for NFS access to storage (see Scaling to multiple servers).

Key recommendations

• Place Deephaven servers and storage on the same network segment.
• Minimize network hops between query servers and storage.
• If your network supports jumbo frames, ensure consistent MTU configuration across all servers, NICs, and switches.

Query server sizing

Query servers run the dispatcher and workers that execute user queries and Persistent Queries.

Sizing guidelines

Query server memory requirements depend on the number of concurrent workers and their heap sizes. Installation planning provides a formula for calculating memory needs:

• TDCP: 4 GB
• Per worker/PQ: At least 8 GB each
• CPU: At least 1 core per Persistent Query

For minimum cluster requirements (8 cores, 128 GB RAM), see Installation planning.

Scale horizontally by adding query servers rather than vertically scaling a single server. The automated server selection feature distributes workers across available dispatchers.

Dispatcher configuration for multiple query servers

When running multiple query servers, configure the controller's server selection provider to balance load:

See Automated server selection for complete configuration details.

Data Import Server sizing

DIS servers handle real-time data ingestion and serve intraday data to workers. The DIS process requires 16 GB of memory according to Installation planning.

Scaling DIS capacity

For high-volume data ingestion:

• Add additional DIS servers to distribute load.
• Use data routing to direct different data streams to different DIS instances.

See Add a dedicated DIS for data ingestion for instructions.

Administrative server sizing

Administrative servers run orchestration and authentication processes. They have lower resource requirements than query or DIS servers and can often be co-located with other services.

Memory requirements

The following services typically run on an administrative server:

Administrative servers do not require access to historical data mounts or high-speed storage.

Monitoring resource usage

After deployment, monitor resource usage to identify bottlenecks:

• Memory: Watch for OOM kills in system logs; monitor dispatcher heap allocation.
• CPU: Monitor for sustained high CPU usage or GC pressure.
• Storage I/O: Track read/write latency and throughput.
• Network: Monitor bandwidth utilization between servers and storage.

The Status Dashboard and metrics and monitoring documentation describe available monitoring options.

Related documentation

• Automated server selection
• Controlling query worker heap size
• Dispatcher configuration
• Installation planning
• Metrics and monitoring
• Scaling to multiple servers