--- title: File system backup and restoration (Kubernetes) sidebar_label: File system (Kubernetes) --- This guide covers file system backup and restoration for Deephaven Kubernetes deployments. Unlike traditional installations, Kubernetes deployments do not use the `/etc/sysconfig/deephaven/illumon.d.latest/` directory structure. Instead, configuration and data are stored in etcd, Helm values, Kubernetes Secrets, `ConfigMaps`, and Persistent Volumes. For traditional deployments, see [File system backup and restoration](./configuration-files-backup.md). For Podman deployments, see [File system backup and restoration (Podman)](./configuration-files-backup-podman.md). ## Configuration storage locations Deephaven Kubernetes deployments store configuration in several locations: | Storage type | What it contains | How to back up | | ---------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------ | | **etcd** | ACLs, Persistent Queries, schemas, data routing YAML, property files | etcd snapshots (automatic via CronJob or manual) | | **Helm values** | Deployment configuration (resources, tolerations, image tags, etc.) | `helm get values` | | **Secrets** | TLS certificates (`deephaven-tls`), image pull credentials, etcd root password | `kubectl get secret -o yaml` | | **ConfigMaps** | Worker pod templates, cluster config, Envoy config | `kubectl get configmap -o yaml` | | **Persistent Volumes** | Historical data (`/db/Systems`), user data (`/db/Users`), intraday data, etcd data | Varies by storage class | ## What to back up ### etcd snapshots etcd stores most Deephaven configuration, including ACLs, Persistent Queries, schemas, routing, and property files. The `deephaven-etcd` Helm chart provides automatic backup via a Kubernetes CronJob when `backup.enabled=true` (the default). **Verify backups are enabled:** ```bash # Check for the backup CronJob kubectl get cronjob -l app.kubernetes.io/component=backup # Or inspect release values helm get values ``` **List available snapshots:** ```bash ETCD_POD=$(kubectl get pod -l app.kubernetes.io/component=etcd \ -o jsonpath='{.items[0].metadata.name}') kubectl exec ${ETCD_POD} -- ls -lh /snapshots ``` Snapshots are named `db-YYYY-MM-DD_HH-MM` and written to the backup PVC mounted at `/snapshots`. **Trigger a manual backup:** ```bash BACKUP_CRONJOB=$(kubectl get cronjob -l app.kubernetes.io/component=backup \ -o jsonpath='{.items[0].metadata.name}') JOB_NAME="manual-backup-$(date +%s)" kubectl create job --from=cronjob/${BACKUP_CRONJOB} ${JOB_NAME} # Wait for the backup job to complete before proceeding kubectl wait --for=condition=complete --timeout=300s job/${JOB_NAME} ``` **Copy a snapshot off-cluster:** ```bash kubectl cp ${ETCD_POD}:/snapshots/db-2025-09-16_19-30 ./etcd-snapshot.db --retries=10 ``` For detailed etcd backup and restore procedures, see [Kubernetes etcd backup and recovery](../kubernetes/kubernetes-etcd-recovery.md). ### Helm values Back up your Helm release values. These contain deployment configuration that etcd does not store: ```bash # Back up Deephaven Helm values helm get values > deephaven-values-backup.yaml # Back up etcd Helm values helm get values > etcd-values-backup.yaml # Back up NFS Helm values (if using deephaven-nfs) helm get values > nfs-values-backup.yaml ``` > [!CAUTION] > `helm get values` output includes sensitive values such as `auth.rootPassword`. Redact credentials before committing to version control, or store these backups in a secrets manager rather than in a repository. ### Secrets Back up critical Secrets. Secret names are configurable via Helm values (`envoyTlsSecret`, `authTrustSecret`, `authUserSecret`). Resolve the actual names from your release and strip cluster-assigned metadata so backups can be applied to a new cluster: ```bash umask 077 # Restrict permissions — backups contain secrets # Helper function to export a secret without cluster-assigned metadata backup_secret() { local secret_name="$1" local output_file="$2" kubectl get secret "${secret_name}" -o json | \ jq 'del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp, .metadata.managedFields, .metadata.ownerReferences)' > "${output_file}" } # Get secret names from Helm values (handle both absent and empty as "use default") VALUES_JSON=$(helm get values -o json) TLS_SECRET=$(echo "${VALUES_JSON}" | jq -r '.envoyTlsSecret | if . == "" or . == null then "deephaven-tls" else . end') AUTH_TRUST_SECRET=$(echo "${VALUES_JSON}" | jq -r '.authTrustSecret | if . == "" or . == null then "deephaven-auth-trust" else . end') AUTH_USER_SECRET=$(echo "${VALUES_JSON}" | jq -r '.authUserSecret | if . == "" or . == null then "deephaven-auth-user" else . end') # Back up TLS certificate backup_secret "${TLS_SECRET}" "${TLS_SECRET}-backup.json" # Back up authentication secrets backup_secret "${AUTH_TRUST_SECRET}" "${AUTH_TRUST_SECRET}-backup.json" backup_secret "${AUTH_USER_SECRET}" "${AUTH_USER_SECRET}-backup.json" # Image pull secrets (check your Helm values for imagePullSecrets array) # Default installations often use repo-deephaven-io-imgpull backup_secret "repo-deephaven-io-imgpull" "imgpull-backup.json" 2>/dev/null || \ echo "Note: repo-deephaven-io-imgpull not found; check imagePullSecrets in Helm values" # etcd root password secret ETCD_SECRET=$(kubectl get secrets -l app.kubernetes.io/component=etcd \ -o jsonpath='{.items[0].metadata.name}') backup_secret "${ETCD_SECRET}" "etcd-secret-backup.json" ``` > [!CAUTION] > Secret backups contain sensitive data (passwords, certificates, credentials). Store them securely and restrict access. ### Persistent Volume data Persistent Volume backup depends on your storage class and infrastructure: | Volume type | Typical backing | Backup approach | | ----------------------------------------------- | ---------------------- | ---------------------------------------------------- | | NFS shared volumes (`/db/Systems`, `/db/Users`) | NFS server | NFS server-level backup or PVC snapshots | | etcd data volumes | Cloud block storage | Use etcd snapshots (see above), not volume snapshots | | DIS intraday volumes | Local or block storage | Storage class snapshots | > [!CAUTION] > Do not rely on storage-class snapshots of etcd data volumes for backup. Independent block-volume snapshots of each etcd member can capture divergent Raft state and produce an inconsistent restore. Always use the etcd snapshot procedure described above. > [!NOTE] > If using the `deephaven-nfs` Helm chart, the NFS server's backing PVC (`dh-nfs-pvc`) contains all shared data. Backing up this PVC captures `/db/Systems`, `/db/Users`, and etcd backup snapshots. ## Custom JARs and plugins In Kubernetes, you typically build custom JARs into container images rather than placing them on the file system: - **Build into images**: Use `--customer-coreplus-jar` or `--customer-plugin` flags with `buildAllForK8s.sh`. This is the recommended approach. - **Mount via volumes**: Use `workerExtraVolumes` in Helm values to mount PVCs containing JARs. See [Mounting volumes and secrets to workers](../kubernetes/kubernetes-configuration-settings.md#mounting-volumes-and-secrets-to-workers). If you build custom JARs into images, ensure your source JARs and Dockerfiles are backed up in version control. ## Automating backups ### etcd backups (automatic) When `backup.enabled=true` in the `deephaven-etcd` Helm chart, a CronJob automatically writes snapshots. The default schedule is every 30 minutes. Configure the schedule via Helm values: ```yaml backup: enabled: true schedule: "*/30 * * * *" # Every 30 minutes ``` ### Helm values and Secrets backup script Create a script to back up Helm values and Secrets. Pass your Deephaven namespace as the first argument: ```bash #!/bin/bash # deephaven-k8s-backup.sh # Usage: ./deephaven-k8s-backup.sh [backup-dir] set -euo pipefail umask 077 # Restrict file permissions — backups contain secrets NAMESPACE="${1:?Usage: $0 [backup-dir]}" DH_RELEASE="${2:?Usage: $0 [backup-dir]}" BACKUP_DIR="${3:-./k8s-backups}" BACKUP_DATE=$(date +%Y-%m-%d_%H-%M-%S) BACKUP_PATH="${BACKUP_DIR}/${BACKUP_DATE}" mkdir -p "${BACKUP_PATH}" # Helper function to export a secret without cluster-assigned metadata backup_secret() { local ns="$1" local secret_name="$2" local output_file="$3" if kubectl get secret -n "${ns}" "${secret_name}" &>/dev/null; then kubectl get secret -n "${ns}" "${secret_name}" -o json | \ jq 'del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp, .metadata.managedFields, .metadata.ownerReferences)' > "${output_file}" echo "Backed up: ${secret_name}" else echo "Warning: Secret '${secret_name}' not found" fi } # Back up Helm releases for release in $(helm list -n ${NAMESPACE} -q); do helm get values -n ${NAMESPACE} "${release}" > "${BACKUP_PATH}/${release}-values.yaml" done # Resolve secret names from Helm values (handle both absent and empty as "use default") VALUES_JSON=$(helm get values -n ${NAMESPACE} "${DH_RELEASE}" -o json 2>/dev/null || echo '{}') TLS_SECRET=$(echo "${VALUES_JSON}" | jq -r '.envoyTlsSecret | if . == "" or . == null then "deephaven-tls" else . end') AUTH_TRUST_SECRET=$(echo "${VALUES_JSON}" | jq -r '.authTrustSecret | if . == "" or . == null then "deephaven-auth-trust" else . end') AUTH_USER_SECRET=$(echo "${VALUES_JSON}" | jq -r '.authUserSecret | if . == "" or . == null then "deephaven-auth-user" else . end') # Back up TLS and auth secrets backup_secret "${NAMESPACE}" "${TLS_SECRET}" "${BACKUP_PATH}/${TLS_SECRET}.json" backup_secret "${NAMESPACE}" "${AUTH_TRUST_SECRET}" "${BACKUP_PATH}/${AUTH_TRUST_SECRET}.json" backup_secret "${NAMESPACE}" "${AUTH_USER_SECRET}" "${BACKUP_PATH}/${AUTH_USER_SECRET}.json" # Back up image pull secrets from Helm values for secret in $(echo "${VALUES_JSON}" | jq -r '.imagePullSecrets[]?.name // empty'); do backup_secret "${NAMESPACE}" "${secret}" "${BACKUP_PATH}/${secret}.json" done # Back up etcd secret ETCD_SECRET=$(kubectl get secrets -n ${NAMESPACE} -l app.kubernetes.io/component=etcd \ -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) if [ -n "${ETCD_SECRET}" ]; then backup_secret "${NAMESPACE}" "${ETCD_SECRET}" "${BACKUP_PATH}/etcd-secret.json" fi echo "Backup completed: ${BACKUP_PATH}" ``` ### Copy etcd snapshots off-cluster Add a step to copy etcd snapshots to external storage: ```bash # Get the latest snapshot filename ETCD_POD=$(kubectl get pod -l app.kubernetes.io/component=etcd \ -o jsonpath='{.items[0].metadata.name}') LATEST_SNAPSHOT=$(kubectl exec ${ETCD_POD} -- ls -t /snapshots | head -1) # Copy to local storage kubectl cp ${ETCD_POD}:/snapshots/${LATEST_SNAPSHOT} ./backups/${LATEST_SNAPSHOT} --retries=10 # Upload to external storage (example: S3) # aws s3 cp ./backups/${LATEST_SNAPSHOT} s3://my-bucket/deephaven-backups/ ``` ## Restore procedures ### Restore etcd from snapshot For detailed etcd restore procedures, see [Kubernetes etcd backup and recovery](../kubernetes/kubernetes-etcd-recovery.md). The basic process is: 1. Identify the snapshot to restore. 2. Delete etcd StatefulSet PVCs. 3. Upgrade with `restore.enabled=true`, `restore.snapshotFilename`, and `backup.pvc.existingClaim` (all three are required). 4. Disable restore after pods are healthy. ### Reapply Helm values If you need to redeploy with backed-up values: ```bash # Scale down first (if the release still exists) ./setupTools/scaleAll.sh 0 # Upgrade or install with backed-up values (--install handles missing releases after cluster loss) helm upgrade --install deephaven \ -f deephaven-values-backup.yaml \ --set image.tag="" ``` ### Recreate Secrets If Secrets were lost, recreate them from backups. The backup files have cluster metadata stripped, so they can be applied directly: ```bash # Get secret names from your backed-up Helm values VALUES_JSON=$(cat deephaven-values-backup.yaml | yq -o=json) TLS_SECRET=$(echo "${VALUES_JSON}" | jq -r '.envoyTlsSecret | if . == "" or . == null then "deephaven-tls" else . end') AUTH_TRUST_SECRET=$(echo "${VALUES_JSON}" | jq -r '.authTrustSecret | if . == "" or . == null then "deephaven-auth-trust" else . end') AUTH_USER_SECRET=$(echo "${VALUES_JSON}" | jq -r '.authUserSecret | if . == "" or . == null then "deephaven-auth-user" else . end') # Delete existing secrets (if corrupted) and apply backups kubectl delete secret "${TLS_SECRET}" --ignore-not-found kubectl apply -f "${TLS_SECRET}-backup.json" kubectl delete secret "${AUTH_TRUST_SECRET}" --ignore-not-found kubectl apply -f "${AUTH_TRUST_SECRET}-backup.json" kubectl delete secret "${AUTH_USER_SECRET}" --ignore-not-found kubectl apply -f "${AUTH_USER_SECRET}-backup.json" ``` For the TLS secret, you can also recreate it from the original certificate files: ```bash kubectl create secret tls deephaven-tls \ --cert=tls.crt \ --key=tls.key ``` ### Verify PVC data After restore, verify that PVC data is intact: ```bash # Check PVC status kubectl get pvc # Verify data in the management shell kubectl exec -it deploy/management-shell -- ls -la /db/Systems kubectl exec -it deploy/management-shell -- ls -la /db/Users ``` ## Using dhconfig in Kubernetes To use the [`dhconfig`](../configuration/dhconfig/overview.md) tool for configuration backup and restore in Kubernetes, connect to the management shell: ```bash kubectl exec -it deploy/management-shell -- /bin/bash # Export configuration /usr/illumon/latest/bin/dhconfig properties export -f iris-environment.prop -d /tmp/ /usr/illumon/latest/bin/dhconfig acls export --file /tmp/acls.xml /usr/illumon/latest/bin/dhconfig pq export --file /tmp/pqs.xml exit # Copy all exports out of the pod (get pod name with: kubectl get pods | grep management-shell) kubectl cp :/tmp/iris-environment.prop ./iris-environment.prop kubectl cp :/tmp/acls.xml ./acls.xml kubectl cp :/tmp/pqs.xml ./pqs.xml ``` For more details, see [Configuration properties backup and restoration](./configuration-properties-backup.md). ## Related documentation - [Backup, Restore, and Migration overview](./backup-and-restore.md) - [File system backup and restoration (traditional)](./configuration-files-backup.md) - [File system backup and restoration (Podman)](./configuration-files-backup-podman.md) - [Kubernetes etcd backup and recovery](../kubernetes/kubernetes-etcd-recovery.md) - [Configuration backup and restore](./configuration-properties-backup.md) - [Kubernetes installation](../kubernetes/kubernetes-install-guide.md) - [Kubernetes configuration settings](../kubernetes/kubernetes-configuration-settings.md)