File system backup and restoration (Kubernetes)

This guide covers file system backup and restoration for Deephaven Kubernetes deployments. Unlike traditional installations, Kubernetes deployments do not use the /etc/sysconfig/deephaven/illumon.d.latest/ directory structure. Instead, configuration and data are stored in etcd, Helm values, Kubernetes Secrets, ConfigMaps, and Persistent Volumes.

For traditional deployments, see File system backup and restoration. For Podman deployments, see File system backup and restoration (Podman).

Configuration storage locations

Deephaven Kubernetes deployments store configuration in several locations:

Storage type	What it contains	How to back up
etcd	ACLs, Persistent Queries, schemas, data routing YAML, property files	etcd snapshots (automatic via CronJob or manual)
Helm values	Deployment configuration (resources, tolerations, image tags, etc.)	`helm get values`
Secrets	TLS certificates (`deephaven-tls`), image pull credentials, etcd root password	`kubectl get secret -o yaml`
ConfigMaps	Worker pod templates, cluster config, Envoy config	`kubectl get configmap -o yaml`
Persistent Volumes	Historical data (`/db/Systems`), user data (`/db/Users`), intraday data, etcd data	Varies by storage class

What to back up

etcd snapshots

etcd stores most Deephaven configuration, including ACLs, Persistent Queries, schemas, routing, and property files. The deephaven-etcd Helm chart provides automatic backup via a Kubernetes CronJob when backup.enabled=true (the default).

Verify backups are enabled:

# Check for the backup CronJob
kubectl get cronjob -l app.kubernetes.io/component=backup

# Or inspect release values
helm get values <etcd-install-name>

List available snapshots:

ETCD_POD=$(kubectl get pod -l app.kubernetes.io/component=etcd \
    -o jsonpath='{.items[0].metadata.name}')
kubectl exec ${ETCD_POD} -- ls -lh /snapshots

Snapshots are named db-YYYY-MM-DD_HH-MM and written to the backup PVC mounted at /snapshots.

Trigger a manual backup:

BACKUP_CRONJOB=$(kubectl get cronjob -l app.kubernetes.io/component=backup \
    -o jsonpath='{.items[0].metadata.name}')
JOB_NAME="manual-backup-$(date +%s)"
kubectl create job --from=cronjob/${BACKUP_CRONJOB} ${JOB_NAME}

# Wait for the backup job to complete before proceeding
kubectl wait --for=condition=complete --timeout=300s job/${JOB_NAME}

Copy a snapshot off-cluster:

kubectl cp ${ETCD_POD}:/snapshots/db-2025-09-16_19-30 ./etcd-snapshot.db --retries=10

For detailed etcd backup and restore procedures, see Kubernetes etcd backup and recovery.

Helm values

Back up your Helm release values. These contain deployment configuration that etcd does not store:

# Back up Deephaven Helm values
helm get values <deephaven-release-name> > deephaven-values-backup.yaml

# Back up etcd Helm values
helm get values <etcd-release-name> > etcd-values-backup.yaml

# Back up NFS Helm values (if using deephaven-nfs)
helm get values <nfs-release-name> > nfs-values-backup.yaml

Caution

helm get values output includes sensitive values such as auth.rootPassword. Redact credentials before committing to version control, or store these backups in a secrets manager rather than in a repository.

Secrets

Back up critical Secrets. Secret names are configurable via Helm values (envoyTlsSecret, authTrustSecret, authUserSecret). Resolve the actual names from your release and strip cluster-assigned metadata so backups can be applied to a new cluster:

umask 077  # Restrict permissions — backups contain secrets

# Helper function to export a secret without cluster-assigned metadata
backup_secret() {
    local secret_name="$1"
    local output_file="$2"
    kubectl get secret "${secret_name}" -o json | \
        jq 'del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp, 
            .metadata.managedFields, .metadata.ownerReferences)' > "${output_file}"
}

# Get secret names from Helm values (handle both absent and empty as "use default")
VALUES_JSON=$(helm get values <deephaven-release-name> -o json)
TLS_SECRET=$(echo "${VALUES_JSON}" | jq -r '.envoyTlsSecret | if . == "" or . == null then "deephaven-tls" else . end')
AUTH_TRUST_SECRET=$(echo "${VALUES_JSON}" | jq -r '.authTrustSecret | if . == "" or . == null then "deephaven-auth-trust" else . end')
AUTH_USER_SECRET=$(echo "${VALUES_JSON}" | jq -r '.authUserSecret | if . == "" or . == null then "deephaven-auth-user" else . end')

# Back up TLS certificate
backup_secret "${TLS_SECRET}" "${TLS_SECRET}-backup.json"

# Back up authentication secrets
backup_secret "${AUTH_TRUST_SECRET}" "${AUTH_TRUST_SECRET}-backup.json"
backup_secret "${AUTH_USER_SECRET}" "${AUTH_USER_SECRET}-backup.json"

# Image pull secrets (check your Helm values for imagePullSecrets array)
# Default installations often use repo-deephaven-io-imgpull
backup_secret "repo-deephaven-io-imgpull" "imgpull-backup.json" 2>/dev/null || \
    echo "Note: repo-deephaven-io-imgpull not found; check imagePullSecrets in Helm values"

# etcd root password secret
ETCD_SECRET=$(kubectl get secrets -l app.kubernetes.io/component=etcd \
    -o jsonpath='{.items[0].metadata.name}')
backup_secret "${ETCD_SECRET}" "etcd-secret-backup.json"

Caution

Secret backups contain sensitive data (passwords, certificates, credentials). Store them securely and restrict access.

Persistent Volume data

Persistent Volume backup depends on your storage class and infrastructure:

Volume type	Typical backing	Backup approach
NFS shared volumes (`/db/Systems`, `/db/Users`)	NFS server	NFS server-level backup or PVC snapshots
etcd data volumes	Cloud block storage	Use etcd snapshots (see above), not volume snapshots
DIS intraday volumes	Local or block storage	Storage class snapshots

Caution

Do not rely on storage-class snapshots of etcd data volumes for backup. Independent block-volume snapshots of each etcd member can capture divergent Raft state and produce an inconsistent restore. Always use the etcd snapshot procedure described above.

Note

If using the deephaven-nfs Helm chart, the NFS server's backing PVC (dh-nfs-pvc) contains all shared data. Backing up this PVC captures /db/Systems, /db/Users, and etcd backup snapshots.

Custom JARs and plugins

In Kubernetes, you typically build custom JARs into container images rather than placing them on the file system:

Build into images: Use --customer-coreplus-jar or --customer-plugin flags with buildAllForK8s.sh. This is the recommended approach.
Mount via volumes: Use workerExtraVolumes in Helm values to mount PVCs containing JARs. See Mounting volumes and secrets to workers.

If you build custom JARs into images, ensure your source JARs and Dockerfiles are backed up in version control.

Automating backups

etcd backups (automatic)

When backup.enabled=true in the deephaven-etcd Helm chart, a CronJob automatically writes snapshots. The default schedule is every 30 minutes. Configure the schedule via Helm values:

backup:
  enabled: true
  schedule: "*/30 * * * *" # Every 30 minutes

Helm values and Secrets backup script

Create a script to back up Helm values and Secrets. Pass your Deephaven namespace as the first argument:

#!/bin/bash
# deephaven-k8s-backup.sh
# Usage: ./deephaven-k8s-backup.sh <namespace> <deephaven-release-name> [backup-dir]

set -euo pipefail
umask 077  # Restrict file permissions — backups contain secrets

NAMESPACE="${1:?Usage: $0 <namespace> <deephaven-release-name> [backup-dir]}"
DH_RELEASE="${2:?Usage: $0 <namespace> <deephaven-release-name> [backup-dir]}"
BACKUP_DIR="${3:-./k8s-backups}"
BACKUP_DATE=$(date +%Y-%m-%d_%H-%M-%S)
BACKUP_PATH="${BACKUP_DIR}/${BACKUP_DATE}"

mkdir -p "${BACKUP_PATH}"

# Helper function to export a secret without cluster-assigned metadata
backup_secret() {
    local ns="$1"
    local secret_name="$2"
    local output_file="$3"
    if kubectl get secret -n "${ns}" "${secret_name}" &>/dev/null; then
        kubectl get secret -n "${ns}" "${secret_name}" -o json | \
            jq 'del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp,
                .metadata.managedFields, .metadata.ownerReferences)' > "${output_file}"
        echo "Backed up: ${secret_name}"
    else
        echo "Warning: Secret '${secret_name}' not found"
    fi
}

# Back up Helm releases
for release in $(helm list -n ${NAMESPACE} -q); do
    helm get values -n ${NAMESPACE} "${release}" > "${BACKUP_PATH}/${release}-values.yaml"
done

# Resolve secret names from Helm values (handle both absent and empty as "use default")
VALUES_JSON=$(helm get values -n ${NAMESPACE} "${DH_RELEASE}" -o json 2>/dev/null || echo '{}')
TLS_SECRET=$(echo "${VALUES_JSON}" | jq -r '.envoyTlsSecret | if . == "" or . == null then "deephaven-tls" else . end')
AUTH_TRUST_SECRET=$(echo "${VALUES_JSON}" | jq -r '.authTrustSecret | if . == "" or . == null then "deephaven-auth-trust" else . end')
AUTH_USER_SECRET=$(echo "${VALUES_JSON}" | jq -r '.authUserSecret | if . == "" or . == null then "deephaven-auth-user" else . end')

# Back up TLS and auth secrets
backup_secret "${NAMESPACE}" "${TLS_SECRET}" "${BACKUP_PATH}/${TLS_SECRET}.json"
backup_secret "${NAMESPACE}" "${AUTH_TRUST_SECRET}" "${BACKUP_PATH}/${AUTH_TRUST_SECRET}.json"
backup_secret "${NAMESPACE}" "${AUTH_USER_SECRET}" "${BACKUP_PATH}/${AUTH_USER_SECRET}.json"

# Back up image pull secrets from Helm values
for secret in $(echo "${VALUES_JSON}" | jq -r '.imagePullSecrets[]?.name // empty'); do
    backup_secret "${NAMESPACE}" "${secret}" "${BACKUP_PATH}/${secret}.json"
done

# Back up etcd secret
ETCD_SECRET=$(kubectl get secrets -n ${NAMESPACE} -l app.kubernetes.io/component=etcd \
    -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
if [ -n "${ETCD_SECRET}" ]; then
    backup_secret "${NAMESPACE}" "${ETCD_SECRET}" "${BACKUP_PATH}/etcd-secret.json"
fi

echo "Backup completed: ${BACKUP_PATH}"

Copy etcd snapshots off-cluster

Add a step to copy etcd snapshots to external storage:

# Get the latest snapshot filename
ETCD_POD=$(kubectl get pod -l app.kubernetes.io/component=etcd \
    -o jsonpath='{.items[0].metadata.name}')
LATEST_SNAPSHOT=$(kubectl exec ${ETCD_POD} -- ls -t /snapshots | head -1)

# Copy to local storage
kubectl cp ${ETCD_POD}:/snapshots/${LATEST_SNAPSHOT} ./backups/${LATEST_SNAPSHOT} --retries=10

# Upload to external storage (example: S3)
# aws s3 cp ./backups/${LATEST_SNAPSHOT} s3://my-bucket/deephaven-backups/

Restore procedures

Restore etcd from snapshot

For detailed etcd restore procedures, see Kubernetes etcd backup and recovery. The basic process is:

Identify the snapshot to restore.
Delete etcd StatefulSet PVCs.
Upgrade with restore.enabled=true, restore.snapshotFilename, and backup.pvc.existingClaim (all three are required).
Disable restore after pods are healthy.

Reapply Helm values

If you need to redeploy with backed-up values:

# Scale down first (if the release still exists)
./setupTools/scaleAll.sh 0

# Upgrade or install with backed-up values (--install handles missing releases after cluster loss)
helm upgrade --install <deephaven-release-name> deephaven \
    -f deephaven-values-backup.yaml \
    --set image.tag="<version>"

Recreate Secrets

If Secrets were lost, recreate them from backups. The backup files have cluster metadata stripped, so they can be applied directly:

# Get secret names from your backed-up Helm values
VALUES_JSON=$(cat deephaven-values-backup.yaml | yq -o=json)
TLS_SECRET=$(echo "${VALUES_JSON}" | jq -r '.envoyTlsSecret | if . == "" or . == null then "deephaven-tls" else . end')
AUTH_TRUST_SECRET=$(echo "${VALUES_JSON}" | jq -r '.authTrustSecret | if . == "" or . == null then "deephaven-auth-trust" else . end')
AUTH_USER_SECRET=$(echo "${VALUES_JSON}" | jq -r '.authUserSecret | if . == "" or . == null then "deephaven-auth-user" else . end')

# Delete existing secrets (if corrupted) and apply backups
kubectl delete secret "${TLS_SECRET}" --ignore-not-found
kubectl apply -f "${TLS_SECRET}-backup.json"

kubectl delete secret "${AUTH_TRUST_SECRET}" --ignore-not-found
kubectl apply -f "${AUTH_TRUST_SECRET}-backup.json"

kubectl delete secret "${AUTH_USER_SECRET}" --ignore-not-found
kubectl apply -f "${AUTH_USER_SECRET}-backup.json"

For the TLS secret, you can also recreate it from the original certificate files:

kubectl create secret tls deephaven-tls \
    --cert=tls.crt \
    --key=tls.key

Verify PVC data

After restore, verify that PVC data is intact:

# Check PVC status
kubectl get pvc

# Verify data in the management shell
kubectl exec -it deploy/management-shell -- ls -la /db/Systems
kubectl exec -it deploy/management-shell -- ls -la /db/Users

Using dhconfig in Kubernetes

To use the dhconfig tool for configuration backup and restore in Kubernetes, connect to the management shell:

kubectl exec -it deploy/management-shell -- /bin/bash

# Export configuration
/usr/illumon/latest/bin/dhconfig properties export -f iris-environment.prop -d /tmp/
/usr/illumon/latest/bin/dhconfig acls export --file /tmp/acls.xml
/usr/illumon/latest/bin/dhconfig pq export --file /tmp/pqs.xml
exit

# Copy all exports out of the pod (get pod name with: kubectl get pods | grep management-shell)
kubectl cp <management-shell-pod>:/tmp/iris-environment.prop ./iris-environment.prop
kubectl cp <management-shell-pod>:/tmp/acls.xml ./acls.xml
kubectl cp <management-shell-pod>:/tmp/pqs.xml ./pqs.xml

For more details, see Configuration properties backup and restoration.

File system backup and restoration (Kubernetes)

Configuration storage locations

What to back up

etcd snapshots

Helm values

Secrets

Persistent Volume data

Custom JARs and plugins

Automating backups

etcd backups (automatic)

Helm values and Secrets backup script

Copy etcd snapshots off-cluster

Restore procedures

Restore etcd from snapshot

Reapply Helm values

Recreate Secrets

Verify PVC data

Using dhconfig in Kubernetes

Related documentation