File system backup and restoration (Podman)

This guide covers file system backup and restoration for Deephaven Podman deployments. Unlike traditional installations, Podman deployments store configuration and data in mounted volumes managed through the VOLUME_BASE_DIR directory structure.

For traditional deployments, see File system backup and restoration. For Kubernetes deployments, see File system backup and restoration (Kubernetes).

Key directories and volumes

The following directories under VOLUME_BASE_DIR contain data and configuration that should be backed up:

DirectoryPurposeShared across pods?
deephaven-etcd/etcd data and server configurationNo (infra only)
deephaven-shared-config/Shared configuration files and private keysYes
deephaven-tls/TLS certificates and keys for the hostNo
db-systems/Historical dataYes

Back up the following directories if you configured their optional volumes. Data whose volume is not configured is stored inside the pod: it is not in this directory tree, and it is deleted when the pod is recreated. Export it as described in Export data not on persistent volumes.

DirectoryPurposeHow to configure
db-intraday/Intraday data, including Web UI dashboards and notebooks--intraday-data-volume (-I)
var-log-deephaven/Logs--logs-volume (-L)
db-users/Direct user table data--users-volume
db-intradayuser/Centrally managed user table data--intradayuser-volume
java_lib/Custom JARs--java-lib-volume
calendars/Custom calendar files--calendars-volume

What happens on upgrade

When you run start_command.sh --upgrade, the script moves regenerable shared configuration into a timestamped backup directory:

  • Moved to upgrade-backup-<timestamp>/: auth, trust, dh-config, etcd, installation, cluster_status (where present)
  • Preserved in place, and copied into the backup: dh_etcd_config.tgz (so the existing etcd database is reused), priv-keys (because the public halves are registered in the auth database), etcd-overrides.yaml

Caution

Files you add to /etc/sysconfig/deephaven/auth inside the container, such as SAML or Active Directory certificates, are lost whenever the pod is recreated. Store them in a host directory mounted with --additional-volume. See Persisting authentication certificates.

Persisting authentication certificates

The auth directory is inside the container and is not preserved when a pod is recreated. To keep SAML and Active Directory certificates and keys across pod recreation and upgrades, store them in a host directory mounted in the infra pod with --additional-volume, and keep that directory under VOLUME_BASE_DIR so it is backed up with the rest. See Persisting authentication certificates for the mount and the properties to update.

Tailer configuration files

The tailer process scans only JAR files in /etc/sysconfig/illumon.d/java_lib. To persist custom tailer configuration XML files:

  1. Package the XML files into a JAR:

  2. Place the JAR in the java_lib volume (configured with --java-lib-volume).

Backup procedures

Back up the VOLUME_BASE_DIR directory tree

The primary backup target is the entire VOLUME_BASE_DIR directory structure. This captures etcd data, shared configuration, TLS certificates, and any data stored on persistent volumes whose paths are under VOLUME_BASE_DIR.

Important

If you configured volumes outside VOLUME_BASE_DIR using start_command.sh options like -H, -I, -U, -W, --java-lib-volume, --calendars-volume, or --additional-volume, you must back up those paths separately. Review your start_command.sh invocation to identify all volume paths. tar stores a symbolic link as a link, not the files it points to: if VOLUME_BASE_DIR, or a directory in it, is a symbolic link (for example to an NFS mount), back up the directory the link points to as well.

Caution

Stop all Deephaven pods before creating a filesystem backup. A tar of live etcd or database files can capture inconsistent state and produce an unusable recovery point.

Note

If db-systems/ or other directories are hosted on network storage (e.g., NFS), you may want to exclude them from this backup if the network storage system already backs them up.

Export data not on persistent volumes

If you have not configured persistent volumes for certain paths, use podman cp to export that data from the pod. Stop the pod first, so that no file is copied while Deephaven is writing it, and export before the pod is recreated, which deletes this data:

Note

In a multi-user deployment, podman cp does not keep the owners of the files, in either direction, so Deephaven's accounts may not be able to write data imported this way. To keep the data from now on, move it onto host volumes instead.

Back up etcd

For etcd-specific backup procedures, see etcd and MySQL backup and restoration. The snapshot is taken from the running infra container: finish any exports that need the pod stopped first, then start it (podman pod start dh-infra-pod). In a Podman deployment, run dh_etcdctl.sh from the extracted deephaven-podman-<version> directory on the infra host. It runs etcdctl in the infra container with the etcd root credentials:

Automating backups

Schedule regular backups using cron or a similar scheduler.

Important

Rootless Podman pods are owned by the user who ran start_command.sh. Schedule backup scripts from that user's crontab (crontab -e), not from /etc/cron.daily which runs as root and cannot see another user's Podman storage. For rootless Podman to work from cron while that user is not logged in, enable lingering for the user: sudo loginctl enable-linger <user>.

This example stops the pods, backs up the VOLUME_BASE_DIR daily, restarts the pods it stopped on success or failure, and retains backups for 7 days. Set PODS to every pod on this host; the script stops without backing anything up if one of them does not exist. The example is for a deployment whose pods all run on one host. In a multi-node cluster, independent copies on each host do not make a consistent backup: stop the pods on every host before any host starts copying, and start them again only after every host has finished, for example by running the stop, copy and start steps for all hosts from one machine over ssh.

Note

You can also use the Deephaven backup script within a Podman deployment by exec'ing into the pod, though the VOLUME_BASE_DIR backup approach is typically simpler for containerized deployments.

Tip

For zero-downtime backups of etcd, use the etcd snapshot procedure instead of stopping pods. Filesystem-level backups of live etcd data directories can capture inconsistent state.

Restore procedures

Restore the VOLUME_BASE_DIR directory structure

  1. Stop all Deephaven pods. Specify a timeout of at least 90 seconds so that all Deephaven processes, monit, and etcd exit cleanly:

  2. Extract the backup to the VOLUME_BASE_DIR. For multi-user rootless deployments, use podman unshare to restore files with correct mapped ownership:

  3. If SELinux is in use, reset the labels:

    --range=s0 removes the categories that limit a file to the pod that created it, so every pod, including query pods, can read the restored shared configuration.

    For more information, see Troubleshooting SELinux in Podman.

  4. Start the pods with podman pod start. If the pods no longer exist, create them with start_command.sh and the same options used originally.

Import data with podman cp

To restore data that was exported with podman cp, stop the pod, copy the contents (not the directory itself) to avoid creating nested paths, then start the pod:

Restore authentication certificates after upgrade

An upgrade from Grizzly moves Grizzly's shared auth directory, with any SAML or Active Directory certificates you added to it, into upgrade-backup-<timestamp>/auth. Copy the certificates into your deephaven-saml directory and point the properties at them, as described in Persisting authentication certificates: