File system backup and restoration (Podman)
This guide covers file system backup and restoration for Deephaven Podman deployments. Unlike traditional installations, Podman deployments store configuration and data in mounted volumes managed through the VOLUME_BASE_DIR directory structure.
For traditional deployments, see File system backup and restoration. For Kubernetes deployments, see File system backup and restoration (Kubernetes).
Key directories and volumes
The following directories under VOLUME_BASE_DIR contain data and configuration that should be backed up:
| Directory | Purpose | Shared across pods? |
|---|---|---|
deephaven-etcd/ | etcd data and server configuration | No (infra only) |
deephaven-shared-config/ | Shared configuration files and private keys | Yes |
deephaven-tls/ | TLS certificates and keys for the host | No |
db-systems/ | Historical data | Yes |
Back up the following directories if you configured their optional volumes. Data whose volume is not configured is stored inside the pod: it is not in this directory tree, and it is deleted when the pod is recreated. Export it as described in Export data not on persistent volumes.
| Directory | Purpose | How to configure |
|---|---|---|
db-intraday/ | Intraday data, including Web UI dashboards and notebooks | --intraday-data-volume (-I) |
var-log-deephaven/ | Logs | --logs-volume (-L) |
db-users/ | Direct user table data | --users-volume |
db-intradayuser/ | Centrally managed user table data | --intradayuser-volume |
java_lib/ | Custom JARs | --java-lib-volume |
calendars/ | Custom calendar files | --calendars-volume |
What happens on upgrade
When you run start_command.sh --upgrade, the script moves regenerable shared configuration into a timestamped backup directory:
- Moved to
upgrade-backup-<timestamp>/:auth,trust,dh-config,etcd,installation,cluster_status(where present) - Preserved in place, and copied into the backup:
dh_etcd_config.tgz(so the existing etcd database is reused),priv-keys(because the public halves are registered in the auth database),etcd-overrides.yaml
Caution
Files you add to /etc/sysconfig/deephaven/auth inside the container, such as SAML or Active Directory certificates, are lost whenever the pod is recreated. Store them in a host directory mounted with --additional-volume. See Persisting authentication certificates.
Persisting authentication certificates
The auth directory is inside the container and is not preserved when a pod is recreated. To keep SAML and Active Directory certificates and keys across pod recreation and upgrades, store them in a host directory mounted in the infra pod with --additional-volume, and keep that directory under VOLUME_BASE_DIR so it is backed up with the rest. See Persisting authentication certificates for the mount and the properties to update.
Tailer configuration files
The tailer process scans only JAR files in /etc/sysconfig/illumon.d/java_lib. To persist custom tailer configuration XML files:
-
Package the XML files into a JAR:
-
Place the JAR in the
java_libvolume (configured with--java-lib-volume).
Backup procedures
Back up the VOLUME_BASE_DIR directory tree
The primary backup target is the entire VOLUME_BASE_DIR directory structure. This captures etcd data, shared configuration, TLS certificates, and any data stored on persistent volumes whose paths are under VOLUME_BASE_DIR.
Important
If you configured volumes outside VOLUME_BASE_DIR using start_command.sh options like -H, -I, -U, -W, --java-lib-volume, --calendars-volume, or --additional-volume, you must back up those paths separately. Review your start_command.sh invocation to identify all volume paths. tar stores a symbolic link as a link, not the files it points to: if VOLUME_BASE_DIR, or a directory in it, is a symbolic link (for example to an NFS mount), back up the directory the link points to as well.
Caution
Stop all Deephaven pods before creating a filesystem backup. A tar of live etcd or database files can capture inconsistent state and produce an unusable recovery point.
Note
If db-systems/ or other directories are hosted on network storage (e.g., NFS), you may want to exclude them from this backup if the network storage system already backs them up.
Export data not on persistent volumes
If you have not configured persistent volumes for certain paths, use podman cp to export that data from the pod. Stop the pod first, so that no file is copied while Deephaven is writing it, and export before the pod is recreated, which deletes this data:
Note
In a multi-user deployment, podman cp does not keep the owners of the files, in either direction, so Deephaven's accounts may not be able to write data imported this way. To keep the data from now on, move it onto host volumes instead.
Back up etcd
For etcd-specific backup procedures, see etcd and MySQL backup and restoration. The snapshot is taken from the running infra container: finish any exports that need the pod stopped first, then start it (podman pod start dh-infra-pod). In a Podman deployment, run dh_etcdctl.sh from the extracted deephaven-podman-<version> directory on the infra host. It runs etcdctl in the infra container with the etcd root credentials:
Automating backups
Schedule regular backups using cron or a similar scheduler.
Important
Rootless Podman pods are owned by the user who ran start_command.sh. Schedule backup scripts from that user's crontab (crontab -e), not from /etc/cron.daily which runs as root and cannot see another user's Podman storage. For rootless Podman to work from cron while that user is not logged in, enable lingering for the user: sudo loginctl enable-linger <user>.
This example stops the pods, backs up the VOLUME_BASE_DIR daily, restarts the pods it stopped on success or failure, and retains backups for 7 days. Set PODS to every pod on this host; the script stops without backing anything up if one of them does not exist. The example is for a deployment whose pods all run on one host. In a multi-node cluster, independent copies on each host do not make a consistent backup: stop the pods on every host before any host starts copying, and start them again only after every host has finished, for example by running the stop, copy and start steps for all hosts from one machine over ssh.
Note
You can also use the Deephaven backup script within a Podman deployment by exec'ing into the pod, though the VOLUME_BASE_DIR backup approach is typically simpler for containerized deployments.
Tip
For zero-downtime backups of etcd, use the etcd snapshot procedure instead of stopping pods. Filesystem-level backups of live etcd data directories can capture inconsistent state.
Restore procedures
Restore the VOLUME_BASE_DIR directory structure
-
Stop all Deephaven pods. Specify a timeout of at least 90 seconds so that all Deephaven processes,
monit, andetcdexit cleanly: -
Extract the backup to the
VOLUME_BASE_DIR. For multi-user rootless deployments, usepodman unshareto restore files with correct mapped ownership: -
If SELinux is in use, reset the labels:
--range=s0removes the categories that limit a file to the pod that created it, so every pod, including query pods, can read the restored shared configuration.For more information, see Troubleshooting SELinux in Podman.
-
Start the pods with
podman pod start. If the pods no longer exist, create them withstart_command.shand the same options used originally.
Import data with podman cp
To restore data that was exported with podman cp, stop the pod, copy the contents (not the directory itself) to avoid creating nested paths, then start the pod:
Restore authentication certificates after upgrade
An upgrade from Grizzly moves Grizzly's shared auth directory, with any SAML or Active Directory certificates you added to it, into upgrade-backup-<timestamp>/auth. Copy the certificates into your deephaven-saml directory and point the properties at them, as described in Persisting authentication certificates: