Storage tuning

This guide covers strategies for optimizing storage I/O performance in your Deephaven deployment through proper storage architecture, hardware selection, and filesystem configuration.

Understanding Deephaven's storage architecture

Storage access patterns

For detailed architecture, see Data Lifecycle.

Merge operations and I/O impact

Merge operations are the most I/O-intensive operations in Deephaven:

• Read phase: Sequential read of intraday partitions
• Processing: In-memory sorting and grouping
• Write phase: Sequential write to historical partitions
• I/O pattern: Sustained sequential reads + sustained sequential writes

See Merging data for complete details.

Tuning merge performance

The data buffer pool size and number of concurrent write threads drive merge performance more than raw I/O throughput.

Data buffer pool

The data buffer pool caches column data as it is read from disk. It is the most important driver of merge performance. In heap mode, Deephaven automatically clamps the pool between 10% and 60% of the merge process heap size. To size it explicitly:

Concurrent write threads

Increasing write threads allows merge to write multiple output partitions in parallel:

For best results, the number of writing threads should be a multiple of the number of output partitions.

Merge heap sizing

A reliable rule of thumb:

See Merge optimization for full details, including symbol table and ordering memory guidance.

Tuning DIS and tailer performance

Use the following levers when tailer lag increases or the DIS falls behind real-time data.

Scale horizontally with DIS sharding

A single DIS instance processes all intraday writes for its assigned tables. For high-throughput deployments, shard across multiple DIS instances by table or partition. Sharding is configured in the routing configuration. See Add a DIS server and dhconfig routing for details.

Tailer watch service

Controls how the tailer detects new binary log data. Use JavaWatchService for local disk (more efficient); use PollWatchService for NFS-mounted log directories:

Poll interval

Lower values reduce ingestion latency at the cost of higher CPU usage. Default is 100ms:

Concurrent table location pool

Controls how many table locations the tailer processes concurrently. Increase if many tables are lagging:

See Data tailer configuration for the full property reference.

Storage capacity planning

IOPS and throughput requirements

IOPS and throughput requirements vary based on data volume, column count, query patterns, and hardware. Measure baseline I/O with representative test data using tools like iostat or iotop, then scale estimates for production volumes.

Key factors affecting storage performance:

• Intraday (DIS): Write-heavy during ingestion; read-heavy when serving queries
• Historical: Read-heavy during queries; sustained read+write throughput during merge operations

Storage capacity sizing

Intraday retention:

Historical storage:

Hardware optimization

Storage media comparison

RAID configuration

Local vs. network storage

Filesystem selection and tuning

Filesystem comparison

Mount options for intraday (XFS)

Mount options for historical (XFS)

Block device tuning

NFS optimization

NFS mount options

See NFS configuration for complete details.

S3 optimization

For S3-based historical storage:

• Regional placement: Deploy in same AWS region as S3 buckets.
• VPC endpoints: Use S3 VPC endpoints to reduce latency and cost.
• Performance: S3 performance is significantly slower than NFS. See the S3 performance section for benchmark details.

See S3 table storage for details.

Monitoring I/O performance

Key metrics

Appropriate storage performance targets are system-dependent.

How to establish your baseline:

1. Run iostat -xz 5 during normal operations and note typical values for:
   • %util — device utilization
   • r_await / w_await — read/write latency in ms
   • rMB/s / wMB/s — throughput
2. Set alerts when metrics consistently exceed your observed normal range

What to look for:

• Throughput (rMB/s, wMB/s): Sustained drops indicate bottlenecks.
• Queue depth (avgqu-sz): Rising values indicate saturation.
• Latency (await): Increasing latency affects query responsiveness.
• Utilization (%util): Sustained high utilization leaves no headroom for bursts.

Monitoring commands

Common storage issues

Related documentation

• Performance tuning overview
• Capacity planning
• Data Lifecycle
• Merging data
• Merge optimization
• Add a DIS server
• Data tailer configuration
• NFS configuration
• S3 table storage
• Table storage overview