Capacity planning

This guide helps you size Deephaven deployments for optimal performance based on workload characteristics, data volumes, and user requirements. Proper capacity planning ensures you provision adequate resources without over-provisioning, balancing performance and cost.

Capacity planning methodology

Effective capacity planning follows a structured approach:

1. Define workload characteristics: Data volumes, query patterns, user concurrency
2. Estimate resource requirements: CPU, memory, storage, network
3. Validate with testing: Load test with representative workloads
4. Plan for growth: Monitor trends and set scaling triggers

Define workload characteristics

Characterize your workload across these dimensions:

Data characteristics (aggregate across all tables):

• Ingest rate: Total rows per second across all tables
• Row size: Average bytes per row (varies by table)
• Table count: Number of actively ingested tables
• Data types: Proportion of numeric vs. string vs. complex types
• Retention: Days of intraday data before merging to historical

Query characteristics:

• Concurrent users: Users running queries and users viewing/subscribing to query results (each consumer adds load)
• Query complexity: Simple aggregations vs. complex joins vs. custom operations
• Active workers: Concurrent PQs and Code Studios per user
• Tracked rows: Rows tracked per active query
• Interactive vs. batch: Real-time dashboards vs. scheduled reports

Access patterns:

• Historical data access: Percentage of queries accessing historical vs. intraday data
• Data scanning: Typical date ranges queried (1 day, 1 week, 1 year)
• Update frequency: How often tables update (tick-by-tick, 1-second snapshots, etc.)

Estimate resource requirements

Use the following formulas as starting points, then validate with testing. The sections below cover CPU, memory, storage, and network sizing.

CPU capacity planning

Query host CPU sizing

Workers use all available CPUs on the host. When sizing a query host, estimate total cores needed:

Formula:

Components:

• Update parallelism: Varies by table complexity and update frequency
• Concurrent workers: Peak number of active workers on the host
• Background threads: Additional cores for updates, GC, and maintenance per worker
• Headroom: Reserve capacity for burst loads

See CPU optimization for tuning strategies.

DIS CPU sizing

Data Import Server (DIS):

DIS CPU requirements depend heavily on row complexity, column count, data transformations, and disk throughput.

To estimate your requirements:

1. Deploy a test DIS with representative data schemas and ingestion rates.
2. Measure CPU utilization under expected load.
3. Scale based on observed utilization, leaving headroom for burst capacity.

Memory capacity planning

Heap sizing

Heap sizing depends on query complexity, table sizes, and concurrent operations. Key factors include:

• Query working set: Tables actively being processed
• Table caches: Cached metadata and frequently accessed data
• Overhead: JVM internals, garbage collection headroom

Starting point:

The default heap for new Code Studios and Persistent Queries is controlled by RemoteProcessingRequest.defaultQueryHeapMB. Check your environment's configured default and adjust based on workload.

How to adjust:

1. Start a PQ or Code Studio with the default heap size.
2. Run your typical workload and monitor heap usage using:
   • The heap indicator in the Code Studio toolbar (shows current usage, free, and max).
   • Or query ProcessMetricsLogCoreV2 for heap metrics:
     GroovyPython
3. Increase heap if you see OOM errors or heap usage consistently near max after GC.
4. Decrease heap if usage after GC is consistently below 50% (frees resources for other workers).

See Controlling query worker heap size for configuration details.

Direct memory sizing: When using direct memory for data buffers, the data buffer pool can be configured up to 2× heap size. Size -XX:MaxDirectMemorySize to accommodate the data buffer pool plus overhead for network buffers.

System memory sizing

Formula:

Any RAM not used by processes becomes available to the OS page cache, improving I/O performance. Leave headroom beyond process requirements to benefit from caching.

Example:

• Heap: 32 GB
• Direct memory: 16 GB
• OS overhead: 8 GB
• Minimum system RAM: 56 GB → provision 64 GB or more

The extra ~8 GB becomes available for OS page cache.

See Memory management for tuning strategies.

Storage capacity planning

Storage capacity sizing

Intraday storage:

Example (illustrative): 100 GB/day × 3 days × 1.5 (growth factor) = 450 GB per namespace

Historical storage:

Compression ratios vary widely based on data characteristics (cardinality, repetition, data types). As a rough guide:

• Deephaven native: ~1.0× (uncompressed)
• Parquet SNAPPY: 0.3× to 0.5× typical, but can range from 0.1× (highly compressible) to 0.8× (low cardinality strings, random data)

Example (illustrative): 100 GB/day × 365 days × 0.4 (compression) × 1.2 (overhead) = 17.5 TB

Storage performance sizing

IOPS and throughput requirements vary significantly based on:

• Data ingestion rate: Higher row rates require more write IOPS/throughput.
• Query patterns: Concurrent queries increase read demands.
• Column count and types: Wide tables with strings require more I/O than narrow numeric tables.
• Merge workloads: Merge operations require sustained read+write throughput.

To estimate your requirements:

1. Measure baseline I/O with representative test data using tools like iostat or iotop.
2. Scale estimates based on expected production data volumes.
3. Add headroom (20-30%) for burst capacity.

See Storage tuning for optimization strategies.

Network capacity planning

Network traffic sources

Network bandwidth requirements depend on the sum of all traffic sources:

• Data ingestion: Inbound data from external sources (Kafka, binary logs, etc.)
• TDCP traffic: Data served to workers via TDCP from DIS
• NFS/S3 storage: Historical data access over network-attached storage
• etcd: Cluster coordination traffic (frequent small packets)
• Inter-process communication: Controller, dispatchers, workers, web services

To estimate requirements:

1. Measure baseline traffic with tools like iftop, nethogs, or switch port statistics.
2. Identify peak periods (market open, batch jobs, etc.).
3. Add headroom (2-3×) for burst capacity.

Validation and testing

Capacity testing procedure

1. Load test with synthetic data:

   • Generate representative data volumes and schemas.
   • Simulate peak concurrent user load.
   • Measure resource utilization (CPU, memory, I/O, network).

2. Identify bottlenecks:

   • Which resource reaches capacity first?
   • What is headroom at peak load?
   • Are there queuing delays or timeouts?

3. Measure key metrics:

   • Query response time (p50, p95, p99) — see QueryPerformanceLogCoreV2
   • Data ingestion lag
   • GC pause times and frequency
   • Storage I/O wait
   • Network utilization

4. Adjust and retest:

   • Increase constrained resources.
   • Validate improvement.
   • Test edge cases (burst load, data spikes).

Capacity headroom guidelines

Maintain headroom for growth and burst capacity.

How to establish headroom:

1. Measure baseline utilization during normal and peak operations:
   • CPU: htop or top
   • Memory: free -h and heap metrics from ProcessMetricsLogCoreV2
   • Storage: iostat -xz 5
   • Network: iftop or switch port statistics
2. Note your typical peak — this becomes your baseline.
3. Plan to scale when sustained utilization consistently exceeds your baseline with limited headroom for bursts.

Growth planning

Scaling triggers

Plan to scale when you observe:

• CPU: Sustained high utilization during peak hours relative to your baseline
• Memory: Frequent full GC cycles, growing GC pause times, or OOM errors
• Storage: Sustained high I/O utilization or nearing capacity
• Network: Sustained high utilization or increasing latency

Scaling strategies

Quick reference checklist

1. ✅ Define workload: Data ingestion rate, concurrent users, query patterns
2. ✅ Estimate CPU: Host cores based on query parallelism and concurrency
3. ✅ Estimate memory: Heap and direct memory based on working set size
4. ✅ Estimate storage: Capacity (intraday + historical) and performance
5. ✅ Estimate network: Bandwidth for ingestion, queries, and storage access
6. ✅ Validate with testing: Load test with representative workload
7. ✅ Measure headroom: Ensure 20-40% capacity above peak load
8. ✅ Plan for growth: Monitor trends and set scaling triggers

Related documentation

• Performance tuning overview
• CPU optimization
• Memory management
• Storage tuning
• Scaling to multiple servers
• Resilience planning