etcd and MySQL backup and restoration

A majority of Deephaven configuration is stored in etcd. The etcd cluster may consist of a single node or any odd number of nodes. Assuming a healthy etcd cluster, the nodes should be automatically synchronized.

etcd backup and restore

etcd can be backed up and restored using etcd's own tools. However, the Deephaven provided etcdctl.sh script is preferred.

Backup can be done using the etcdctl.sh utility with:

sudo -u irisadmin ETCDCTL_ENDPOINTS=https://{ip-of-single-etcd-node}:2379 /usr/illumon/latest/bin/etcdctl.sh snapshot save [filename]

Restore is covered in the etcd documentation, under the topic: "Restart cluster from majority failure". For Deephaven, the etcd restore process is as follows:

  1. Install the new Deephaven cluster.
  2. Connect to the etcd node where restore will be processed.
  3. Use /usr/illumon/latest/bin/etcdctl.sh endpoint status -w table to get a list of etcd nodes in the cluster.
  4. Use /usr/illumon/latest/bin/etcdctl.sh move-leader to force the local node to be the leader of the cluster.
  5. Use /usr/illumon/latest/bin/etcdctl.sh member remove to remove the other nodes from the cluster until only the local node remains.
  6. Use /usr/illumon/latest/bin/etcdctl.sh snapshot restore to restore the etcd database.
  7. Follow the rest of the steps in the linked etcd.io article for adding the other nodes back to the cluster.

MySQL ACL backup and restore

In systems that use MySQL/MariaDB for ACLs, the dbacl_iris database stores Deephaven logins and groups and associated permissions for Deephaven objects. This database can be backed up and restored using MySQL native backup and restore tools, though the Deephaven backup script is the preferred method.

# Backup with:
mysqldump --user [user_name] --password=[password] --databases dbacl_iris > [filename]

# Restore with:
mysql --user [user_name] --password=[password] -e "drop database if exists dbacl_iris"
mysql --user [user_name] --password=[password] -e "create database dbacl_iris"
mysql --user [user_name] --password=[password] dbacl_iris < [filename]

Tip

If the original deployment uses MySQL/MariaDB for ACLs, a migration may be a good time to convert to the preferred etcd ACL storage. The Deephaven backup script ACL export can be imported to etcd on the new deployment after setting a few properties