File system backup and restoration (Podman)

This guide covers file system backup and restoration for Deephaven Podman deployments. Unlike traditional installations, Podman deployments store configuration and data in mounted volumes managed through the VOLUME_BASE_DIR directory structure.

For traditional deployments, see File system backup and restoration. For Kubernetes deployments, see File system backup and restoration (Kubernetes).

Key directories and volumes

The following directories under VOLUME_BASE_DIR contain data and configuration that should be backed up:

Directory	Purpose	Shared across pods?
`deephaven-etcd/`	etcd data and server configuration	No (infra only)
`deephaven-shared-config/`	Shared configuration files (auth, trust, dh-config, etc.)	Yes
`deephaven-tls/`	TLS certificates and keys for the host	No
`db-intraday/`	Intraday data	No (infra only)
`db-systems/`	Historical data	Yes

If configured with optional volumes:

Directory	Purpose	How to configure
`db-users/`	Direct user table data	`--users-volume`
`db-intradayuser/`	Centrally managed user table data	`--intradayuser-volume`
`java_lib/`	Custom JARs and authentication certificates	`--java-lib-volume`
`calendars/`	Custom calendar files	`--calendars-volume`

What happens on upgrade

When you run start_command.sh --upgrade, the script moves regenerable shared configuration into a timestamped backup directory:

Moved to upgrade-backup-<timestamp>/: auth, trust, dh-config, etcd, installation, cluster_status
Preserved in place: dh_etcd_config.tgz (so the existing etcd database is reused), priv-keys (because the public halves are registered in the auth database)

Caution

If you store SAML or Active Directory certificates in the auth directory, they are moved to the backup directory during upgrade. You must either restore them manually after the upgrade or store them in a persistent location such as the java_lib volume. See Persisting authentication certificates.

Persisting authentication certificates

Upgrades move the /deephaven-shared-config/auth directory to a backup location. To persist authentication certificates across restarts and upgrades, store them in the java_lib volume:

Mount a persistent volume for java_lib using the --java-lib-volume option:

./start_command.sh \
  ... \
  --java-lib-volume "${VOLUME_BASE_DIR:?}/custom-java-lib/"

Place certificate files in the host directory. They appear in the container at /etc/sysconfig/illumon.d/java_lib/.

Update the relevant properties to reference the new path:

Certificate type	Property	Example path
SAML IdP certificate	`authentication.samlauth.idp.x509cert.file`	`/etc/sysconfig/illumon.d/java_lib/idp-cert.pem`
Active Directory trust store	`authentication.server.ldap.truststore.ad.path`	`/etc/sysconfig/illumon.d/java_lib/ad-truststore.jks`

For more details, see Persisting authentication certificates.

Tailer configuration files

The tailer process scans only JAR files in /etc/sysconfig/illumon.d/java_lib. To persist custom tailer configuration XML files:

Package the XML files into a JAR:

jar cf tailer-config.jar tailer-config.xml

Place the JAR in the java_lib volume (configured with --java-lib-volume).

Backup procedures

Back up the `VOLUME_BASE_DIR` directory tree

The primary backup target is the entire VOLUME_BASE_DIR directory structure. This captures etcd data, shared configuration, TLS certificates, and any data stored on persistent volumes whose paths are under VOLUME_BASE_DIR.

Important

If you configured volumes outside VOLUME_BASE_DIR using start_command.sh options like -H, -I, -U, -W, --java-lib-volume, --calendars-volume, or --additional-volume, you must back up those paths separately. Review your start_command.sh invocation to identify all volume paths.

Caution

Stop all Deephaven pods before creating a filesystem backup. A tar of live etcd or database files can capture inconsistent state and produce an unusable recovery point.

# Stop pods cleanly (90-second timeout for etcd shutdown)
podman pod stop dh-infra-pod -t 90
podman pod stop dh-query1-pod -t 90  # if running query pods

# Create a timestamped backup of the entire VOLUME_BASE_DIR
# For multi-user rootless deployments, use 'podman unshare tar' to read files
# owned by translated container UIDs
umask 077  # Restrict permissions — backup contains secrets
BACKUP_DATE=$(date +%Y-%m-%d_%H-%M-%S)

# Single-user deployment
tar -czvf "deephaven-backup-${BACKUP_DATE}.tar.gz" "${VOLUME_BASE_DIR:?}"

# Multi-user rootless deployment (can read files owned by mapped UIDs)
# podman unshare tar -czvf "deephaven-backup-${BACKUP_DATE}.tar.gz" "${VOLUME_BASE_DIR:?}"

# Restart pods
./start_command.sh ...

Note

If db-systems/ or other directories are hosted on network storage (e.g., NFS), you may want to exclude them from this backup if the network storage system already backs them up.

Export data not on persistent volumes

If you have not configured persistent volumes for certain paths, use podman cp to export data from a running or stopped pod:

# Export direct user table data
podman cp dh-infra:/db/Users/ ./BackupUsers/

# Export centrally managed user table data
podman cp dh-infra:/db/IntradayUser/ ./BackupIntradayUser/

# Export custom JARs (if not using --java-lib-volume)
podman cp dh-infra:/etc/sysconfig/illumon.d/java_lib/ ./BackupJavaLib/

# Export custom calendars (if not using --calendars-volume)
podman cp dh-infra:/etc/sysconfig/illumon.d/calendars/ ./BackupCalendars/

Back up etcd

For etcd-specific backup procedures, see etcd and MySQL backup and restoration. In a Podman deployment, you can exec into the pod to run etcd backup commands:

podman exec dh-infra /usr/illumon/latest/bin/etcdctl.sh snapshot save /tmp/etcd-snapshot.db
podman cp dh-infra:/tmp/etcd-snapshot.db ./etcd-snapshot.db

Automating backups

Schedule regular backups using cron or a similar scheduler.

Important

Rootless Podman pods are owned by the user who ran start_command.sh. Schedule backup scripts from that user's crontab (crontab -e), not from /etc/cron.daily which runs as root and cannot see another user's Podman storage.

This example stops the pods, backs up the VOLUME_BASE_DIR daily, restarts the pods on success or failure, and retains backups for 7 days:

#!/bin/bash
# deephaven-backup.sh
# Schedule via the Podman user's crontab: crontab -e
# Example: 0 2 * * * /path/to/deephaven-backup.sh

set -euo pipefail
umask 077  # Restrict permissions — backup contains secrets

VOLUME_BASE_DIR="/container-test"
BACKUP_DIR="/backups/deephaven"
RETENTION_DAYS=7
START_COMMAND="/path/to/start_command.sh"  # Adjust to your deployment
START_ARGS="..."  # Your start_command.sh arguments

BACKUP_DATE=$(date +%Y-%m-%d_%H-%M-%S)
BACKUP_FILE="${BACKUP_DIR}/deephaven-backup-${BACKUP_DATE}.tar.gz"

# Track which pods were running so we can restart them on exit
PODS_TO_RESTART=()

cleanup() {
    echo "Restarting pods..."
    for pod_args in "${PODS_TO_RESTART[@]}"; do
        ${START_COMMAND} ${pod_args} || echo "Warning: failed to restart pod"
    done
}
trap cleanup EXIT

# Create backup directory if it does not exist
mkdir -p "${BACKUP_DIR}"

# Stop pods cleanly before backup (90-second timeout for etcd shutdown)
if podman pod exists dh-infra-pod && podman pod stop dh-infra-pod -t 90; then
    PODS_TO_RESTART+=("${START_ARGS}")
else
    echo "Failed to stop dh-infra-pod or pod does not exist"; exit 1
fi

# Stop query pod if it exists (treat failure as fatal to avoid live-data backup)
if podman pod exists dh-query1-pod 2>/dev/null; then
    if podman pod stop dh-query1-pod -t 90; then
        PODS_TO_RESTART+=("--query-pod-args")  # Add your query pod start args
    else
        echo "Failed to stop dh-query1-pod"; exit 1
    fi
fi

# Create backup
# For multi-user rootless deployments, use 'podman unshare tar' to read mapped UIDs
tar -czvf "${BACKUP_FILE}" "${VOLUME_BASE_DIR}"
# podman unshare tar -czvf "${BACKUP_FILE}" "${VOLUME_BASE_DIR}"  # Multi-user variant

# Remove old backups
find "${BACKUP_DIR}" -name "deephaven-backup-*.tar.gz" -mtime +${RETENTION_DAYS} -delete

echo "Backup completed: ${BACKUP_FILE}"
# EXIT trap restarts pods automatically

Note

You can also use the Deephaven backup script within a Podman deployment by exec'ing into the pod, though the VOLUME_BASE_DIR backup approach is typically simpler for containerized deployments.

Tip

For zero-downtime backups of etcd, use the etcd snapshot procedure instead of stopping pods. Filesystem-level backups of live etcd data directories can capture inconsistent state.

Restore procedures

Restore the VOLUME_BASE_DIR directory structure

Stop all Deephaven pods. Specify a timeout of at least 90 seconds so that all Deephaven processes, monit, and etcd exit cleanly:
```
podman pod stop dh-infra-pod -t 90
# If running query pods:
podman pod stop dh-query1-pod -t 90
```

Extract the backup to the VOLUME_BASE_DIR. For multi-user rootless deployments, use podman unshare to restore files with correct mapped ownership:

# Single-user deployment
tar -xzvf deephaven-backup-YYYY-MM-DD_HH-MM-SS.tar.gz -C /

# Multi-user rootless deployment (preserves UID/GID mapping)
podman unshare tar -xzvf deephaven-backup-YYYY-MM-DD_HH-MM-SS.tar.gz -C /

If SELinux is in use, reset the labels:
```
chcon -vR -u system_u -r object_r -t container_file_t "${VOLUME_BASE_DIR:?}"
```
For more information, see Troubleshooting SELinux in Podman.
Restart the pods using start_command.sh with the same options used originally.

Import data with podman cp

To restore data that was exported with podman cp, copy the contents (not the directory itself) to avoid creating nested paths:

# Import user table data (use /. to copy contents, not the directory)
podman cp ./BackupUsers/. dh-infra:/db/Users/

# Import centrally managed user table data
podman cp ./BackupIntradayUser/. dh-infra:/db/IntradayUser/

# Import custom JARs
podman cp ./BackupJavaLib/. dh-infra:/etc/sysconfig/illumon.d/java_lib/

# Import custom calendars
podman cp ./BackupCalendars/. dh-infra:/etc/sysconfig/illumon.d/calendars/

Restore authentication certificates after upgrade

If authentication certificates were moved to upgrade-backup-<timestamp>/ during an upgrade:

# Find the backup directory
ls "${VOLUME_BASE_DIR}/deephaven-shared-config/" | grep upgrade-backup

# Copy certificates back to the auth directory
cp "${VOLUME_BASE_DIR}/deephaven-shared-config/upgrade-backup-<timestamp>/auth/idp-cert.pem" \
   "${VOLUME_BASE_DIR}/deephaven-shared-config/auth/"

Alternatively, move the certificates to the java_lib volume to prevent this issue in future upgrades. See Persisting authentication certificates.