Methodologies for mounting historical NFS partitions

Deephaven can use NFS (Network File System) mounts to store historical data. NFS is a distributed file system protocol allowing a user on a client computer to access files over a computer network much like local storage is accessed. This document outlines three primary methodologies used by Deephaven for mounting NFS partitions: Direct Mounting, Partial Indirect Mounting, and Full Indirect Mounting. The choice of methodology depends on the scale of your deployment and desired flexibility. By default, Deephaven uses Partial Indirect Mounting. However, large deployments may benefit from the flexibility of Full Indirect Mounting, as discussed later.

Key directory structures and conventions:

  • Historical data is typically stored in /db/Systems/<NAMESPACE>, where <NAMESPACE> is a placeholder for your specific database namespace (e.g., DbInternal).
  • Read operations for a namespace use file systems mounted under /db/Systems/<NAMESPACE>/Partitions/0..N.
  • Write operations for a namespace use file systems mounted under /db/Systems/<NAMESPACE>/WritablePartitions/0..N.
  • These directories can reside on local disk, shared filesystems, NFS volumes, or be soft links.
  • WritablePartitions can be a subset of Partitions if less write I/O bandwidth is needed.

Historical Data Partition Mounting Strategies

There are three options for creating historical data directories and mounts.

Direct Mounting

NFS volumes are mounted directly to the Partitions and WritablePartitions sub-directories for each namespace.

# Replace <NAMESPACE> with your specific namespace (e.g., DbInternal)
# Mount the NFS volume read-only for partitions
sudo mount -t nfs -o ro <NFSSERVER_ADDRESS>:/namespace-0 /db/Systems/<NAMESPACE>/Partitions/0

# Mount the NFS volume read-write for writable partitions
sudo mount -t nfs <NFSSERVER_ADDRESS>:/namespace-0 /db/Systems/<NAMESPACE>/WritablePartitions/0

As an example, the namespace-0 NFS volume is mounted directly. The Partitions/0 directory is mounted read-only (-o ro) and the WritablePartitions/0 directory is mounted read-write to allow new data to be ingested.

If additional storage beyond this is required, a second partition namespace-1, and two more mount points would be created, and the new NFS volume would be mounted under each:

sudo mkdir -p /db/Systems/<NAMESPACE>/Partitions/1
sudo mkdir -p /db/Systems/<NAMESPACE>/WritablePartitions/1

sudo mount -t nfs -o ro <NFSSERVER_ADDRESS>:/namespace-1 /db/Systems/<NAMESPACE>/Partitions/1
sudo mount -t nfs <NFSSERVER_ADDRESS>:/namespace-1 /db/Systems/<NAMESPACE>/WritablePartitions/1

Direct Mounting should only be used in small data installations where only a handful of mounts are required.

Partial Indirect Mounting

Each NFS volume is mounted once under /db/Systems/<NAMESPACE>/Partitions/0..N. The /db/Systems/<NAMESPACE>/WritablePartitions/0..N directories are then created as soft-links to their counterparts under /db/Systems/<NAMESPACE>/Partitions/.

# Replace <NAMESPACE> with your specific namespace (e.g., DbInternal)
# Mount the NFS volume once under the Partitions directory
sudo mount -t nfs <NFSSERVER_ADDRESS>:/namespace-0 /db/Systems/<NAMESPACE>/Partitions/0

# Create a symbolic link in WritablePartitions pointing to the Partitions directory
sudo ln -s /db/Systems/<NAMESPACE>/Partitions/0 /db/Systems/<NAMESPACE>/WritablePartitions/0

Additional file systems would be added under /db/Systems/<NAMESPACE>/Partitions/ and then soft links would be created under the WritablePartitions directory.

Note that this approach cannot be used if underlying NFS volumes for read partitions need to change independently of the write partitions.

Full Indirect Mounting

The most flexible methodology, Full Indirect Mounting, allows each individual mount to be customized, or moved with minimal impact to Deephaven. Both /db/Systems/<NAMESPACE>/Partitions/0..N and /db/Systems/<NAMESPACE>/WritablePartitions/0..N are soft-linked to NFS volumes mounted elsewhere on the server (e.g., under /srv/<NAMESPACE>). Therefore, the underlying NFS file systems can be switched out by changing a link, or removed from service by deleting the link altogether.

# Create directories under /srv/<NAMESPACE> for the primary and read-only mounts
sudo mkdir /srv/<NAMESPACE>/0
sudo mkdir /srv/<NAMESPACE>/read-only/0

# Mount the NFS volume to the designated directories with appropriate read/write options
sudo mount -t nfs <NFSSERVER_ADDRESS>:/namespace-0 /srv/<NAMESPACE>/0
sudo mount -t nfs -o ro <NFSSERVER_ADDRESS>:/namespace-0 /srv/<NAMESPACE>/read-only/0

# Create symbolic links in Partitions and WritablePartitions pointing to these mount targets
sudo ln -s /srv/<NAMESPACE>/0 /db/Systems/<NAMESPACE>/WritablePartitions/0
sudo ln -s /srv/<NAMESPACE>/read-only/0 /db/Systems/<NAMESPACE>/Partitions/0

The namespace-0 NFS volume is mounted elsewhere on the server, and soft links are created to both the Partitions and WritablePartitions directory. Adding additional storage or IO bandwidth is mounted under /srv/<NAMESPACE> directories, and new soft links are created for Partitions or WritablePartitions.