---
title: File system backup and restoration (Podman)
sidebar_label: File system (Podman)
---

This guide covers file system backup and restoration for Deephaven Podman deployments. Unlike traditional installations, Podman deployments store configuration and data in mounted volumes managed through the [`VOLUME_BASE_DIR`](../installation/podman-installation.md#set-up-the-volume_base_dir) directory structure.

For traditional deployments, see [File system backup and restoration](./configuration-files-backup.md). For Kubernetes deployments, see [File system backup and restoration (Kubernetes)](./configuration-files-backup-kubernetes.md).

## Key directories and volumes

The following directories under `VOLUME_BASE_DIR` contain data and configuration that should be backed up:

| Directory                  | Purpose                                                   | Shared across pods? |
| -------------------------- | --------------------------------------------------------- | ------------------- |
| `deephaven-etcd/`          | etcd data and server configuration                        | No (infra only)     |
| `deephaven-shared-config/` | Shared configuration files (auth, trust, dh-config, etc.) | Yes                 |
| `deephaven-tls/`           | TLS certificates and keys for the host                    | No                  |
| `db-intraday/`             | Intraday data                                             | No (infra only)     |
| `db-systems/`              | Historical data                                           | Yes                 |

If configured with optional volumes:

| Directory          | Purpose                                     | How to configure        |
| ------------------ | ------------------------------------------- | ----------------------- |
| `db-users/`        | Direct user table data                      | `--users-volume`        |
| `db-intradayuser/` | Centrally managed user table data           | `--intradayuser-volume` |
| `java_lib/`        | Custom JARs and authentication certificates | `--java-lib-volume`     |
| `calendars/`       | Custom calendar files                       | `--calendars-volume`    |

## What happens on upgrade

When you run `start_command.sh --upgrade`, the script moves regenerable shared configuration into a timestamped backup directory:

- **Moved to `upgrade-backup-<timestamp>/`**: `auth`, `trust`, `dh-config`, `etcd`, `installation`, `cluster_status`
- **Preserved in place**: `dh_etcd_config.tgz` (so the existing etcd database is reused), `priv-keys` (because the public halves are registered in the auth database)

> [!CAUTION]
> If you store SAML or Active Directory certificates in the `auth` directory, they are moved to the backup directory during upgrade. You must either restore them manually after the upgrade or store them in a persistent location such as the `java_lib` volume. See [Persisting authentication certificates](#persisting-authentication-certificates).

## Persisting authentication certificates

Upgrades move the `/deephaven-shared-config/auth` directory to a backup location. To persist authentication certificates across restarts and upgrades, store them in the `java_lib` volume:

1. Mount a persistent volume for `java_lib` using the `--java-lib-volume` option:

   ```bash
   ./start_command.sh \
     ... \
     --java-lib-volume "${VOLUME_BASE_DIR:?}/custom-java-lib/"
   ```

2. Place certificate files in the host directory. They appear in the container at `/etc/sysconfig/illumon.d/java_lib/`.

3. Update the relevant properties to reference the new path:

   | Certificate type             | Property                                        | Example path                                          |
   | ---------------------------- | ----------------------------------------------- | ----------------------------------------------------- |
   | SAML IdP certificate         | `authentication.samlauth.idp.x509cert.file`     | `/etc/sysconfig/illumon.d/java_lib/idp-cert.pem`      |
   | Active Directory trust store | `authentication.server.ldap.truststore.ad.path` | `/etc/sysconfig/illumon.d/java_lib/ad-truststore.jks` |

For more details, see [Persisting authentication certificates](../installation/podman-installation.md#persisting-authentication-certificates).

## Tailer configuration files

The tailer process scans only JAR files in `/etc/sysconfig/illumon.d/java_lib`. To persist custom [tailer configuration](../configuration/data-tailer.md#tailer-configuration) XML files:

1. Package the XML files into a JAR:

   ```bash
   jar cf tailer-config.jar tailer-config.xml
   ```

2. Place the JAR in the `java_lib` volume (configured with `--java-lib-volume`).

## Backup procedures

### Back up the `VOLUME_BASE_DIR` directory tree

The primary backup target is the entire `VOLUME_BASE_DIR` directory structure. This captures etcd data, shared configuration, TLS certificates, and any data stored on persistent volumes whose paths are under `VOLUME_BASE_DIR`.

> [!IMPORTANT]
> If you configured volumes outside `VOLUME_BASE_DIR` using `start_command.sh` options like `-H`, `-I`, `-U`, `-W`, `--java-lib-volume`, `--calendars-volume`, or `--additional-volume`, you must back up those paths separately. Review your `start_command.sh` invocation to identify all volume paths.

> [!CAUTION]
> Stop all Deephaven pods before creating a filesystem backup. A tar of live etcd or database files can capture inconsistent state and produce an unusable recovery point.

```bash
# Stop pods cleanly (90-second timeout for etcd shutdown)
podman pod stop dh-infra-pod -t 90
podman pod stop dh-query1-pod -t 90  # if running query pods

# Create a timestamped backup of the entire VOLUME_BASE_DIR
# For multi-user rootless deployments, use 'podman unshare tar' to read files
# owned by translated container UIDs
umask 077  # Restrict permissions — backup contains secrets
BACKUP_DATE=$(date +%Y-%m-%d_%H-%M-%S)

# Single-user deployment
tar -czvf "deephaven-backup-${BACKUP_DATE}.tar.gz" "${VOLUME_BASE_DIR:?}"

# Multi-user rootless deployment (can read files owned by mapped UIDs)
# podman unshare tar -czvf "deephaven-backup-${BACKUP_DATE}.tar.gz" "${VOLUME_BASE_DIR:?}"

# Restart pods
./start_command.sh ...
```

> [!NOTE]
> If `db-systems/` or other directories are hosted on network storage (e.g., NFS), you may want to exclude them from this backup if the network storage system already backs them up.

### Export data not on persistent volumes

If you have not configured persistent volumes for certain paths, use `podman cp` to export data from a running or stopped pod:

```bash
# Export direct user table data
podman cp dh-infra:/db/Users/ ./BackupUsers/

# Export centrally managed user table data
podman cp dh-infra:/db/IntradayUser/ ./BackupIntradayUser/

# Export custom JARs (if not using --java-lib-volume)
podman cp dh-infra:/etc/sysconfig/illumon.d/java_lib/ ./BackupJavaLib/

# Export custom calendars (if not using --calendars-volume)
podman cp dh-infra:/etc/sysconfig/illumon.d/calendars/ ./BackupCalendars/
```

### Back up etcd

For etcd-specific backup procedures, see [etcd and MySQL backup and restoration](./etcd-and-acls-backup.md). In a Podman deployment, you can exec into the pod to run etcd backup commands:

```bash
podman exec dh-infra /usr/illumon/latest/bin/etcdctl.sh snapshot save /tmp/etcd-snapshot.db
podman cp dh-infra:/tmp/etcd-snapshot.db ./etcd-snapshot.db
```

## Automating backups

Schedule regular backups using cron or a similar scheduler.

> [!IMPORTANT]
> Rootless Podman pods are owned by the user who ran `start_command.sh`. Schedule backup scripts from that user's crontab (`crontab -e`), not from `/etc/cron.daily` which runs as root and cannot see another user's Podman storage.

This example stops the pods, backs up the `VOLUME_BASE_DIR` daily, restarts the pods on success or failure, and retains backups for 7 days:

```bash
#!/bin/bash
# deephaven-backup.sh
# Schedule via the Podman user's crontab: crontab -e
# Example: 0 2 * * * /path/to/deephaven-backup.sh

set -euo pipefail
umask 077  # Restrict permissions — backup contains secrets

VOLUME_BASE_DIR="/container-test"
BACKUP_DIR="/backups/deephaven"
RETENTION_DAYS=7
START_COMMAND="/path/to/start_command.sh"  # Adjust to your deployment
START_ARGS="..."  # Your start_command.sh arguments

BACKUP_DATE=$(date +%Y-%m-%d_%H-%M-%S)
BACKUP_FILE="${BACKUP_DIR}/deephaven-backup-${BACKUP_DATE}.tar.gz"

# Track which pods were running so we can restart them on exit
PODS_TO_RESTART=()

cleanup() {
    echo "Restarting pods..."
    for pod_args in "${PODS_TO_RESTART[@]}"; do
        ${START_COMMAND} ${pod_args} || echo "Warning: failed to restart pod"
    done
}
trap cleanup EXIT

# Create backup directory if it does not exist
mkdir -p "${BACKUP_DIR}"

# Stop pods cleanly before backup (90-second timeout for etcd shutdown)
if podman pod exists dh-infra-pod && podman pod stop dh-infra-pod -t 90; then
    PODS_TO_RESTART+=("${START_ARGS}")
else
    echo "Failed to stop dh-infra-pod or pod does not exist"; exit 1
fi

# Stop query pod if it exists (treat failure as fatal to avoid live-data backup)
if podman pod exists dh-query1-pod 2>/dev/null; then
    if podman pod stop dh-query1-pod -t 90; then
        PODS_TO_RESTART+=("--query-pod-args")  # Add your query pod start args
    else
        echo "Failed to stop dh-query1-pod"; exit 1
    fi
fi

# Create backup
# For multi-user rootless deployments, use 'podman unshare tar' to read mapped UIDs
tar -czvf "${BACKUP_FILE}" "${VOLUME_BASE_DIR}"
# podman unshare tar -czvf "${BACKUP_FILE}" "${VOLUME_BASE_DIR}"  # Multi-user variant

# Remove old backups
find "${BACKUP_DIR}" -name "deephaven-backup-*.tar.gz" -mtime +${RETENTION_DAYS} -delete

echo "Backup completed: ${BACKUP_FILE}"
# EXIT trap restarts pods automatically
```

> [!NOTE]
> You can also use the [Deephaven backup script](./backup-and-restore.md#deephaven-backup-script) within a Podman deployment by exec'ing into the pod, though the `VOLUME_BASE_DIR` backup approach is typically simpler for containerized deployments.

> [!TIP]
> For zero-downtime backups of etcd, use the etcd snapshot procedure instead of stopping pods. Filesystem-level backups of live etcd data directories can capture inconsistent state.

## Restore procedures

### Restore the VOLUME_BASE_DIR directory structure

1. Stop all Deephaven pods. Specify a timeout of at least 90 seconds so that all Deephaven processes, `monit`, and `etcd` exit cleanly:

   ```bash
   podman pod stop dh-infra-pod -t 90
   # If running query pods:
   podman pod stop dh-query1-pod -t 90
   ```

2. Extract the backup to the `VOLUME_BASE_DIR`. For multi-user rootless deployments, use `podman unshare` to restore files with correct mapped ownership:

   ```bash
   # Single-user deployment
   tar -xzvf deephaven-backup-YYYY-MM-DD_HH-MM-SS.tar.gz -C /

   # Multi-user rootless deployment (preserves UID/GID mapping)
   podman unshare tar -xzvf deephaven-backup-YYYY-MM-DD_HH-MM-SS.tar.gz -C /
   ```

3. If SELinux is in use, reset the labels:

   ```bash
   chcon -vR -u system_u -r object_r -t container_file_t "${VOLUME_BASE_DIR:?}"
   ```

   For more information, see [Troubleshooting SELinux in Podman](../troubleshooting/podman-troubleshooting.md#troubleshooting-selinux-in-podman).

4. Restart the pods using `start_command.sh` with the same options used originally.

### Import data with podman cp

To restore data that was exported with `podman cp`, copy the contents (not the directory itself) to avoid creating nested paths:

```bash
# Import user table data (use /. to copy contents, not the directory)
podman cp ./BackupUsers/. dh-infra:/db/Users/

# Import centrally managed user table data
podman cp ./BackupIntradayUser/. dh-infra:/db/IntradayUser/

# Import custom JARs
podman cp ./BackupJavaLib/. dh-infra:/etc/sysconfig/illumon.d/java_lib/

# Import custom calendars
podman cp ./BackupCalendars/. dh-infra:/etc/sysconfig/illumon.d/calendars/
```

### Restore authentication certificates after upgrade

If authentication certificates were moved to `upgrade-backup-<timestamp>/` during an upgrade:

```bash
# Find the backup directory
ls "${VOLUME_BASE_DIR}/deephaven-shared-config/" | grep upgrade-backup

# Copy certificates back to the auth directory
cp "${VOLUME_BASE_DIR}/deephaven-shared-config/upgrade-backup-<timestamp>/auth/idp-cert.pem" \
   "${VOLUME_BASE_DIR}/deephaven-shared-config/auth/"
```

Alternatively, move the certificates to the `java_lib` volume to prevent this issue in future upgrades. See [Persisting authentication certificates](#persisting-authentication-certificates).

## Related documentation

- [Backup, Restore, and Migration overview](./backup-and-restore.md)
- [File system backup and restoration (traditional)](./configuration-files-backup.md)
- [File system backup and restoration (Kubernetes)](./configuration-files-backup-kubernetes.md)
- [etcd and MySQL backup and restore](./etcd-and-acls-backup.md)
- [Configuration backup and restore](./configuration-properties-backup.md)
- [Podman installation](../installation/podman-installation.md)
- [Restart and upgrade a Podman deployment](../installation/podman-upgrade.md)