Backup and restoration

The Backup and Restore process is used to recover an existing cluster or to copy or transfer content to a different Deephaven deployment.

Four areas of configuration should be backed up. They can be used to restore an existing installation after a failure or error, and to deploy a new cluster that maintains the configuration of an old/existing cluster:

  1. etcd, which contains most of the Deephaven system configuration, including:
  2. Table data
    • The WorkspaceData table, which contains web dashboards, workspaces, and notebooks.
    • Intraday and historical data
  3. File system configurations, generally under /etc/sysconfig/deephaven:
    • Process startup configuration
    • Authentication certificates and keys
    • Plugins and custom JARs
  4. MySQL/MariaDB; if used, this contains ACL information such as Deephaven accounts and permissions in the dbacl_iris database.

These are the areas that cover the internal configuration of the Deephaven cluster. Of course, any NFS, directory creation, firewall, OS, and/or other changes made to configure the cluster's operation in the customer's environment are also included.

General migration process

The basic migration process is as follows:

If IP addresses, machine names, or machine roles are different in the new cluster, relative to the one from which the backups were taken, configuration values (mainly in iris-environment.prop, routing.yaml, and maybe in tailer config XML files) need to be modified. This can be done from an etcd restore (after restoring etcd) by exporting, editing, and reimporting iris-environment.prop and routing.yaml; or, if config files have been backed up individually, it may be desirable to export the new files after the new cluster has been installed, and then diff them versus the backup files, to see what values should be changed.

After configuration has been restored or migrated to the new environment, restart all Deephaven services on all nodes:

/usr/illumon/latest/bin/dh_monit restart all

Deephaven backup script

Deephaven provides a script that can be used to back up configuration to a set of files. To use the script, first, create a directory to hold the backups. For example:

sudo -u irisadmin mkdir /db/TempFiles/irisadmin/backup

The script should be run daily, and the resulting files should be copied to a secure remote location. Execution should be automated, such as with a cron job. Once the script has completed, back up the files to another server.

Note

The user running the script should either be the Deephaven administration account (typically irisadmin) or have privileges to sudo to this account without a password.

Backup script parameters

The following parameters can be passed into the backup script:

  • -d, --directory - Specify the directory in which the backup files will be created. If this is not provided, a default directory of /etc/sysconfig/deephaven/backups/system is used.
  • -u, --user - Specify the user account under which the backups will be run. If this is not provided, the system's administration account (usually irisadmin) is used. If the script isn't invoked with this account, the account used to run the tool must be able to sudo to this account.
  • -a, --all - Back up everything. This is equivalent to specifying all the individual backup options except --retain.
  • -e, --etcd - Back up the entire etcd database into a snapshot file.
  • -A, --ACLs - Back up the ACLs.
  • -p, --persistentQueries - Back up the Persistent Queries.
  • -w, --workspaceData - Back up the WorkspaceData table.
  • -s, --schema - Back up the schemas. If there are a lot of schemas, this may take a long time, and the etcd backup may be a better option.
  • -P, --propertyFiles - Back up the property files.
  • -r, --routing - Back up the routing files (both the primary routing file and any configurations loaded with the dhconfig dis commands).
  • -R, --retain - Delete backups this number of days old if this backup is successful.

This example creates backups for the etcd database, ACLs, Persistent Queries, schemas, property files, routing, and workspace data in the specified directory using the system's administration account and deletes backups that are a week old. If run every day, this will keep a rotating set of backups for a week.

/usr/illumon/latest/bin/backup_deephaven --directory /db/TempFiles/irisadmin/backup --all --retain 7

Restoration from backup

The configuration restore section discusses restoration for each configuration type.