---
title: Storage tuning
---

This guide covers strategies for optimizing storage I/O performance in your Deephaven deployment through proper storage architecture, hardware selection, and filesystem configuration.

> [!NOTE]
> For data organization and partitioning strategies, see [Table storage](../table-storage/table-storage-overview.md).

## Understanding Deephaven's storage architecture

### Storage access patterns

| Data type  | Default Location | Access pattern                       | Characteristics                          |
| ---------- | ---------------- | ------------------------------------ | ---------------------------------------- |
| Intraday   | `/db/Intraday/`  | Sequential writes, random reads      | Write-intensive, latency-sensitive       |
| Historical | `/db/Systems/`   | Large sequential reads, batch writes | Read-intensive, tolerates higher latency |

For detailed architecture, see [Data Lifecycle](../architecture/data-lifecycle.md).

### Merge operations and I/O impact

Merge operations are the most I/O-intensive operations in Deephaven:

- **Read phase**: Sequential read of intraday partitions
- **Processing**: In-memory sorting and grouping
- **Write phase**: Sequential write to historical partitions
- **I/O pattern**: Sustained sequential reads + sustained sequential writes

See [Merging data](../../data-guide/merging.md) for complete details.

### Tuning merge performance

The data buffer pool size and number of concurrent write threads drive merge performance more than raw I/O throughput.

**Data buffer pool**

The data buffer pool caches column data as it is read from disk. It is the most important driver of merge performance. In heap mode, Deephaven automatically clamps the pool between 10% and 60% of the merge process heap size. To size it explicitly:

```properties
DataBufferConfiguration.poolSize=<total bytes>
```

**Concurrent write threads**

Increasing write threads allows merge to write multiple output partitions in parallel:

```properties
iris.concurrentWriteThreads=<number of threads>
```

For best results, the number of writing threads should be a multiple of the number of output partitions.

**Merge heap sizing**

A reliable rule of thumb:

```
buffer pool size = max column file size × nWriteThreads / nOutputPartitions / 0.85
heap size = 2 × buffer pool size
```

See [Merge optimization](../../data-guide/merge-optimization.md) for full details, including symbol table and ordering memory guidance.

### Tuning DIS and tailer performance

Use the following levers when tailer lag increases or the DIS falls behind real-time data.

**Scale horizontally with DIS sharding**

A single DIS instance processes all intraday writes for its assigned tables. For high-throughput deployments, shard across multiple DIS instances by table or partition. Sharding is configured in the routing configuration. See [Add a DIS server](../architecture/add-dis-server.md) and [dhconfig routing](../configuration/dhconfig/routing.md) for details.

**Tailer watch service**

Controls how the tailer detects new binary log data. Use `JavaWatchService` for local disk (more efficient); use `PollWatchService` for NFS-mounted log directories:

```properties
log.tailer.watchServiceType=JavaWatchService
```

**Poll interval**

Lower values reduce ingestion latency at the cost of higher CPU usage. Default is 100ms:

```properties
log.tailer.poll.pause=100
```

**Concurrent table location pool**

Controls how many table locations the tailer processes concurrently. Increase if many tables are lagging:

```properties
DataContent.userPoolCapacity=128
DataContent.systemPoolCapacity=128
```

See [Data tailer configuration](../configuration/data-tailer.md) for the full property reference.

## Storage capacity planning

### IOPS and throughput requirements

IOPS and throughput requirements vary based on data volume, column count, query patterns, and hardware. Measure baseline I/O with representative test data using tools like `iostat` or `iotop`, then scale estimates for production volumes.

Key factors affecting storage performance:

- **Intraday (DIS)**: Write-heavy during ingestion; read-heavy when serving queries
- **Historical**: Read-heavy during queries; sustained read+write throughput during merge operations

### Storage capacity sizing

**Intraday retention:**

```
Intraday capacity = Daily data volume × Retention days × Growth factor (1.5×)
```

**Historical storage:**

```
Historical capacity = Daily volume × Days retained × Compression ratio × Overhead (1.2×)

Compression ratios (typical, where 1.0× = uncompressed):
- Deephaven native: 1.0×
- Parquet SNAPPY (default): 0.3× to 0.5×
```

> [!NOTE]
> Actual compression ratios vary significantly based on data characteristics (cardinality, repetition, data types). Test with representative data to determine accurate sizing.

## Hardware optimization

> [!NOTE]
> Hardware recommendations in this section are general guidelines. Validate sizing against your specific workload before purchasing.

### Storage media comparison

| Media    | Throughput   | IOPS   | Latency | Use case              | Cost |
| -------- | ------------ | ------ | ------- | --------------------- | ---- |
| NVMe SSD | 3-7 GB/s     | 500k+  | < 100μs | Intraday              | $$$  |
| SATA SSD | 500-600 MB/s | 90k+   | ~100μs  | Historical (moderate) | $$   |
| HDD      | 100-200 MB/s | 75-150 | 5-10ms  | Cold archive only     | $    |

### RAID configuration

| RAID    | Read perf | Write perf | Capacity     | Use case               |
| ------- | --------- | ---------- | ------------ | ---------------------- |
| RAID 10 | Excellent | Good       | 50%          | Intraday (recommended) |
| RAID 6  | Good      | Moderate   | (N-2) drives | Historical storage     |

### Local vs. network storage

| Storage type   | Best for             | Advantages                   |
| -------------- | -------------------- | ---------------------------- |
| Local NVMe/SSD | Intraday             | Lowest latency, highest IOPS |
| SAN            | Both (if configured) | Shared, enterprise features  |
| NFS            | Historical           | Simple, shared, flexible     |
| S3             | Cold historical      | Unlimited capacity, low cost |

## Filesystem selection and tuning

### Filesystem comparison

| Filesystem | Best for            | Mount options                        | Strengths                 |
| ---------- | ------------------- | ------------------------------------ | ------------------------- |
| XFS        | Intraday/Historical | `noatime,nodiratime,largeio,swalloc` | Large files, parallel I/O |
| ext4       | Alternative         | `noatime,data=ordered,commit=60`     | Fast metadata             |
| ZFS        | Advanced            | Tune ARC, recordsize                 | Compression, snapshots    |

### Mount options for intraday (XFS)

```bash
mount -t xfs -o noatime,nodiratime,logbufs=8,logbsize=256k /dev/sdb1 /db/Intraday
```

### Mount options for historical (XFS)

```bash
mount -t xfs -o noatime,nodiratime,largeio,swalloc /dev/sdd1 /db/Systems
```

### Block device tuning

```bash
# For SSDs: Use none or mq-deadline scheduler
echo none > /sys/block/sda/queue/scheduler

# Set read-ahead for sequential workloads (historical)
blockdev --setra 8192 /dev/sdd  # 4MB read-ahead

# For random workloads (intraday), keep read-ahead low
blockdev --setra 256 /dev/sdc
```

## NFS optimization

### NFS mount options

```bash
mount -t nfs -o rsize=1048576,wsize=1048576,tcp,hard,timeo=600,retrans=2,noatime \
  nfs-server:/export/deephaven /db/Systems
```

| Option            | Value         | Purpose            |
| ----------------- | ------------- | ------------------ |
| `rsize` / `wsize` | 1048576 (1MB) | Match jumbo frames |
| `tcp`             | -             | Reliability        |
| `hard`            | -             | Retry indefinitely |
| `timeo`           | 600           | 60-second timeout  |
| `noatime`         | -             | Reduce write I/O   |

See [NFS configuration](../table-storage/nfs.md) for complete details.

## S3 optimization

For S3-based historical storage:

- **Regional placement**: Deploy in same AWS region as S3 buckets.
- **VPC endpoints**: Use S3 VPC endpoints to reduce latency and cost.
- **Performance**: S3 performance is significantly slower than NFS. See the [S3 performance section](../table-storage/table-storage-s3.md#performance) for benchmark details.

See [S3 table storage](../table-storage/table-storage-s3.md) for details.

## Monitoring I/O performance

### Key metrics

Appropriate storage performance targets are system-dependent.

**How to establish your baseline:**

1. Run `iostat -xz 5` during normal operations and note typical values for:
   - `%util` — device utilization
   - `r_await` / `w_await` — read/write latency in ms
   - `rMB/s` / `wMB/s` — throughput
2. Set alerts when metrics consistently exceed your observed normal range

**What to look for:**

- **Throughput** (`rMB/s`, `wMB/s`): Sustained drops indicate bottlenecks.
- **Queue depth** (`avgqu-sz`): Rising values indicate saturation.
- **Latency** (`await`): Increasing latency affects query responsiveness.
- **Utilization** (`%util`): Sustained high utilization leaves no headroom for bursts.

### Monitoring commands

```bash
# Monitor I/O stats every 5 seconds
iostat -xz 5

# Real-time I/O by process
sudo iotop -o

# Benchmark sequential read
fio --name=seqread --rw=read --bs=1M --size=10G --directory=/db/Systems
```

### Common storage issues

| Symptom                 | Likely cause                  | Solution                         |
| ----------------------- | ----------------------------- | -------------------------------- |
| Slow historical queries | Throughput bottleneck         | Upgrade storage, distribute load |
| Data ingestion lag      | Write throughput insufficient | Upgrade to NVMe                  |
| Long merge durations    | Low sequential throughput     | Optimize for sequential I/O      |
| High I/O wait           | Storage saturation            | Check health, upgrade media      |

## Related documentation

- [Performance tuning overview](./overview.md)
- [Capacity planning](./capacity-planning.md)
- [Data Lifecycle](../architecture/data-lifecycle.md)
- [Merging data](../../data-guide/merging.md)
- [Merge optimization](../../data-guide/merge-optimization.md)
- [Add a DIS server](../architecture/add-dis-server.md)
- [Data tailer configuration](../configuration/data-tailer.md)
- [NFS configuration](../table-storage/nfs.md)
- [S3 table storage](../table-storage/table-storage-s3.md)
- [Table storage overview](../table-storage/table-storage-overview.md)