Kubernetes etcd backup and recovery

Etcd backup and restore procedures in a Kubernetes installation differ slightly from the procedures in a default installation. Note that this page refers to the etcd installation used solely by Deephaven for its own purposes and is completely separate from the etcd instance used by the Kubernetes system.

Deephaven's etcd runs from the bundled deephaven-etcd Helm chart, which provides backup and restore natively.

Overview

The deephaven-etcd chart supports three related capabilities:

Backups — when backup.enabled=true is set in the values when etcd is installed, a Kubernetes CronJob periodically writes timestamped snapshot files (named db-YYYY-MM-DD_HH-MM) to a read-write-many (RWX) PersistentVolumeClaim. By default, backups are enabled and written to the /exports/exports/dhsystem/etcd-backup directory on the Deephaven NFS server. However, any RWX volume can be used.
Automatic disaster recovery — because enabling backups also mounts the snapshot PVC on every etcd pod, an etcd pod that starts with an empty data directory and cannot reach its peers automatically restores from the newest snapshot instead of coming up blank.
Manual restore — you can restore the cluster to a specific snapshot with a Helm upgrade.

There are two recovery scenarios:

Restore in place (most common) — restore the existing etcd release to a snapshot. No Deephaven reconfiguration is required.
Restore into a new cluster — stand up a new etcd release from a snapshot (for full disaster recovery, or to migrate off a legacy Bitnami etcd deployment), then repoint Deephaven at it.

Prerequisites

It is assumed that you have kubectl and helm command-line tools installed and configured for your target namespace in your Kubernetes cluster. You will also need the following:

Helm install name of etcd. Define it as a variable named etcd_install_name, or substitute it wherever ${etcd_install_name:?} appears in the examples below.
Kubernetes namespace. Substitute this name wherever <k8s-namespace> appears in the examples below.
Root password for etcd. Define it as a variable named etcd_root_password, or substitute this password wherever ${etcd_root_password:?} appears in the examples below.
The deephaven-etcd Helm chart, which is included in the deephaven-helm distribution used to install Deephaven. The deephaven_etcd image is pulled from repo.deephaven.io/deephaven_etcd using an image pull secret created from credentials provided by your Deephaven associate (see Install the new etcd cluster from the snapshot); alternatively, mirror deephaven_etcd:VAR:DH_ETCD_IMAGE_VERSION into your own repository.

Note

The deephaven-etcd chart names its resources after the Helm release (the "fullname"): <etcd-install-name> if the name already ends in -etcd, otherwise <etcd-install-name>-etcd. This is the StatefulSet, ClusterIP service, and root-password Secret name; etcd pods are <fullname>-0, <fullname>-1, ... <fullname>-n, and the backup CronJob is <fullname>-backup. The examples below use label selectors so you do not have to compute these names.

Finding your etcd Helm install name, namespace, and root password

To find the Helm install name, run helm list and look for the deephaven-etcd listing.

$ helm list
NAME                 NAMESPACE        REVISION    UPDATED                                 STATUS      CHART                 APP VERSION
<etcd-install-name>  <k8s-namespace>  1           2024-03-14 14:58:50.304509 -0400 EDT    deployed    deephaven-etcd-0.1.0  3.6.0

The etcd root user password is stored in a Secret named after the etcd release, key etcd-root-password.

# Find the name of the secret for the etcd root password
$ kubectl get secrets -l app.kubernetes.io/component=etcd
NAME                       TYPE     DATA   AGE
<etcd-secret-name>         Opaque   1      100d

# Get your root etcd password value with this command
$ kubectl get secret <etcd-secret-name> -o jsonpath='{.data.etcd-root-password}' | base64 -d

Determine whether backups are enabled

Backups are enabled by default, but you should confirm this before attempting a restore. If enabled, a recent snapshot is likely already available. Check for the backup CronJob, or inspect the release values for backup.enabled.

# A backup CronJob exists only when backups are enabled
kubectl get cronjob -l app.kubernetes.io/component=backup

# Or inspect the release values
helm get values ${etcd_install_name:?}

An example values section with backups enabled:

backup:
  enabled: true
  schedule: "*/30 * * * *"
  pvc:
    existingClaim: <backup-snapshot-pvc-name>

Take a snapshot of etcd

Snapshots are written to the snapshot PVC (mounted on each etcd pod at /snapshots by default). You can trigger a one-off backup immediately from the CronJob, then list the available snapshots.

# Trigger a one-off backup from the backup CronJob
BACKUP_CRONJOB=$(kubectl get cronjob -l app.kubernetes.io/component=backup \
    -o jsonpath='{.items[0].metadata.name}')
kubectl create job --from=cronjob/${BACKUP_CRONJOB} manual-backup-$(date +%s)

# List the snapshots available on an etcd pod
ETCD_POD=$(kubectl get pod -l app.kubernetes.io/component=etcd \
    -o jsonpath='{.items[0].metadata.name}')
kubectl exec ${ETCD_POD} -- ls -lh /snapshots
db-2025-09-16_19-20
db-2025-09-16_19-30

# Optional: copy the most recent snapshot to your local filesystem
kubectl cp ${ETCD_POD}:/snapshots/db-2025-09-16_19-30 etcd-snapshot.db --retries=10

If backups are not enabled, take a snapshot manually on one of the nodes and copy it from the node.

# Find an etcd pod
ETCD_POD=$(kubectl get pod -l app.kubernetes.io/component=etcd \
    -o jsonpath='{.items[0].metadata.name}')

# Save an etcd snapshot to a file on the pod (the root password is required)
kubectl exec ${ETCD_POD} -- \
    etcdctl --user root:${etcd_root_password:?} snapshot save /tmp/etcd-snapshot.db

# Copy the snapshot to your local filesystem
kubectl cp ${ETCD_POD}:/tmp/etcd-snapshot.db etcd-snapshot.db --retries=10

Restore the existing cluster in place

This is the recommended path when the cluster still exists but its data is corrupt or lost. It restores every pod in the existing etcd release from a named snapshot that is already present on the backup PVC. Because the release name and service are unchanged, no Deephaven reconfiguration is needed.

Identify the snapshot filename on the backup PVC (see Take a snapshot to list them), e.g. db-2025-09-16_19-30.
Delete the etcd StatefulSet PVCs so every pod starts with an empty data directory:
```
kubectl delete pvc -l app.kubernetes.io/instance=<etcd-install-name>
```
Caution

This permanently deletes the current etcd data volumes. Confirm the snapshot you intend to restore exists on the backup PVC before deleting.
Upgrade with restore parameters. --reuse-values preserves the image, backup, and other settings from the original install:
```
helm upgrade <etcd-install-name> deephaven-etcd \
    --reuse-values \
    --set restore.enabled=true \
    --set restore.snapshotFilename=db-2025-09-16_19-30 \
    --set backup.pvc.existingClaim=<backup-snapshot-pvc-name>
```
Each pod finds its data directory empty and restores from the named snapshot before starting etcd. All pods restore from the same file, which is correct and deterministic.

Verify the cluster is healthy:

ETCD_POD=$(kubectl get pod -l app.kubernetes.io/component=etcd \
    -o jsonpath='{.items[0].metadata.name}')
kubectl exec ${ETCD_POD} -- \
    etcdctl --user root:${etcd_root_password:?} member list -w table

Disable restore so pods do not re-restore on their next restart:

helm upgrade ${etcd_install_name:?} deephaven-etcd --reuse-values --set restore.enabled=false

Note

restore.enabled=true requires both restore.snapshotFilename and backup.pvc.existingClaim. The chart fails with a descriptive error if either is missing.

Restore into a new cluster

Use this when you cannot restore in place — for example, the namespace or cluster was lost, or you are migrating off a legacy Bitnami etcd deployment. You will create a new etcd release from a snapshot and then repoint Deephaven at it.

Prepare the snapshot on RWX storage

The snapshot file must be at the root of a read-write-many (RWX) PVC that you will pass as backup.pvc.existingClaim. If you already have a deephaven-etcd backup PVC, its CronJob snapshots are there already — just use one of their filenames. Otherwise (for example, restoring from a snapshot taken elsewhere), place the snapshot on an RWX volume. This example uses the Deephaven NFS server.

The Deephaven NFS server exports /exports via NFS. The /exports/dhsystem subdirectory under /exports contains Deephaven system directories, including etcd-backup where snapshots are stored. Under that are subdirectories used within the Deephaven system, including the etcd-backup directory where etcd snapshots are stored.

Create a new directory named etcd-restore alongside etcd-backup, and copy your snapshot into a directory on that export:

# Create a directory etcd-restore, copy the snapshot, and set ownership to the etcd run user id 1001.
kubectl exec deploy/deephaven-nfs-server -- mkdir -p /exports/exports/dhsystem/etcd-restore
MY_NFS_POD=$(kubectl get pods -l role=deephaven-nfs-server -o custom-columns='NAME:.metadata.name' --no-headers)
kubectl cp etcd-snapshot.db ${MY_NFS_POD}:/exports/exports/dhsystem/etcd-restore/etcd-snapshot.db --retries=10
kubectl exec deploy/deephaven-nfs-server -- chown -R 1001:1001 /exports/exports/dhsystem/etcd-restore

Note

The NFS server mounts its PVC at /exports, and /exports is what is exported to clients — so the container path /exports/exports/dhsystem/... corresponds to the client-side NFS path /exports/dhsystem/....

Create a Persistent Volume and Persistent Volume Claim for this directory. Change the PV/PVC names and the spec.nfs.server value for your environment (if you use non-NFS RWX storage, the YAML will differ):

apiVersion: v1
kind: PersistentVolume
metadata:
  name: <your-etcd-restore-snapshot-pv-name>
spec:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 10Gi
  mountOptions:
    - hard
    - nfsvers=4.1
  nfs:
    # Client-side NFS path (see note above about /exports vs /exports/exports).
    path: /exports/dhsystem/etcd-restore
    # The server value should be: deephaven-nfs.<k8s-namespace>.svc.cluster.local.
    # The DNS name does not always resolve (e.g. in AKS, EKS), so you can use the IP address directly.
    # Get the NFS service IP with 'kubectl get svc deephaven-nfs'.
    server: deephaven-nfs.<k8s-namespace>.svc.cluster.local
  persistentVolumeReclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: <your-etcd-restore-snapshot-pvc-name>
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  volumeName: <your-etcd-restore-snapshot-pv-name>
  resources:
    requests:
      storage: 10Gi

Save as etcd-restore-vol.yaml and apply it:

kubectl apply -f etcd-restore-vol.yaml

Install the new etcd cluster from the snapshot

The deephaven_etcd image is pulled from repo.deephaven.io/deephaven_etcd. If you do not already have an image pull secret for it in this namespace, create one using the username and password provided by your Deephaven associate:

# Set the Deephaven repo username and password provided by your Deephaven associate
kubectl create secret docker-registry repo-deephaven-io-imgpull \
    --docker-server='repo.deephaven.io' \
    --docker-username='${docker_username:?}' \
    --docker-password='${docker_password:?}'

Install a new deephaven-etcd release with restore enabled, referencing the snapshot PVC and file. Substitute a new etcd install name, the RWX PVC name, the snapshot filename, and the root password from your original cluster. (If you mirrored the image into your own repository, use that location for image.repository and the corresponding pull secret.)

# Define ${your_etcd_restore_snapshot_pvc_name:?} with the name used in etcd-restore-vol.yaml in the previous step,
# or replace it in the command below. 
helm install ${new_etcd_install_name:?} deephaven-etcd \
    --set image.repository=repo.deephaven.io \
    --set image.tag=VAR:DH_ETCD_IMAGE_VERSION \
    --set "image.pullSecrets[0].name"=repo-deephaven-io-imgpull \
    --set auth.rootPassword=${etcd_root_password:?} \
    --set backup.enabled=true \
    --set backup.pvc.existingClaim=${your_etcd_restore_snapshot_pvc_name:?} \
    --set restore.enabled=true \
    --set restore.snapshotFilename=etcd-snapshot.db \
    --set replicaCount=3

Monitor with kubectl get pods -l app.kubernetes.io/instance=${new_etcd_install_name:?} -w until all pods show Running and 1/1 ready. Once healthy, disable restore so pods do not re-restore on restart:

helm upgrade ${new_etcd_install_name:?} deephaven-etcd --reuse-values --set restore.enabled=false

Update Deephaven for the new etcd cluster name

Deephaven builds its etcd endpoint from the etcd.release value, so repointing is a values change plus a Helm upgrade — no manual secret editing is required.

When Deephaven was installed, you most likely used a YAML file to define Helm chart values for your environment (Deephaven URL, image repository, NFS server, etc.). That file contains an etcd.release value set to your original etcd fullname. Change it to the new etcd cluster's fullname (the ClusterIP service name — verify with kubectl get svc -l app.kubernetes.io/component=etcd). If you set a custom etcd.endpoint, update it too.

If you do not have that file, retrieve the values with helm get values ${deephaven_installation_name:?} (run helm list if you do not know the Deephaven Helm installation name).

Scale the Deephaven deployments down first with the included setupTools/scaleAll.sh script, then upgrade:

setupTools/scaleAll.sh 0

Then perform the Helm upgrade as described in Install the Deephaven Helm chart, with the updated etcd.release.

Legacy: Bitnami etcd

Deployments created before the deephaven-etcd chart used a Bitnami etcd chart, installed via setup-etcd.sh and distributed as bitnami-etcd-helm-11.3.6.tgz (with images in bitnami-etcd-containers-11.3.6.tar.gz). Those clusters used startFromSnapshot.* and disasterRecovery.* Helm values rather than the restore.*/backup.* values described above. See here to find out which etcd chart you currently have installed.

Bitnami etcd deployments remain supported. To recover a Bitnami etcd cluster, you can reinstall the Bitnami chart with startFromSnapshot.enabled=true. However, migrating to the deephaven-etcd chart is recommended — take a snapshot of the Bitnami cluster and follow Restore into a new cluster above. If you no longer have the Bitnami distribution files, contact Deephaven support.

Such deployments remain supported. To recover a Bitnami etcd cluster you can reinstall the Bitnami chart with startFromSnapshot.enabled=true, or — recommended — migrate to the deephaven-etcd chart by taking a snapshot of the Bitnami cluster and following Restore into a new cluster above. If you no longer have the Bitnami distribution files, contact Deephaven support.

Kubernetes etcd backup and recovery

Overview

Prerequisites

Finding your etcd Helm install name, namespace, and root password

Determine whether backups are enabled

Take a snapshot of etcd

Restore the existing cluster in place

Restore into a new cluster

Prepare the snapshot on RWX storage

Install the new etcd cluster from the snapshot

Update Deephaven for the new etcd cluster name

Legacy: Bitnami etcd

Related documentation