Configuration backup, restore, and migration concepts

There are four areas of configuration that need to be backed up and restored in order to deploy a new cluster that maintains the configuration of an old/existing cluster:

  1. MySQL/MariaDB - which contains ACL information such as Deephaven accounts and permissions in the DbAcl database.
  2. etcd - which contains the most of deephaven system configuration, including:
  • Persistent queries
  • Schemata
  • Data routing yaml
  • Properties files
  1. The WorkspaceData table - which contains Web dashboards, workspaces, and notebooks.
  2. File system files - these are under /etc/sysconfig/deephaven:
  • Host Config (if it has been modified to change heap settings for processes, etc): etc/sysconfig/deephaven/illumon.iris.hostconfig
  • Tailer config XML (if a custom one has been created)
  • Any custom certificate files needed for authentication with external services such as LDAP: /etc/sysconfig/deephaven/illumon.d.latest/resources/
  • Any custom jars: /etc/sysconfig/deephaven/illumon.d.latest/java_lib/
  • From the auth server node, dsakeyts.txt to capture any key-based authentication that has been set up for users: /etc/sysconfig/deephaven/auth/; in addition, if this will be replaced from backup on the new system, *.base64.txt key files from /etc/sysconfig/deephaven/auth should also be backed up.
  • It is a good safety measure to back up all of /etc/sysconfig/deephaven/auth to ensure that needed keys and passphrase files are available in case they are later needed for restore operations.

These are the areas that cover the internal configuration of the Deephaven cluster. In addition, there are of course any NFS, directory creation, firewall, OS, and/or other changes that have been made to configure the operation of the cluster in the customer's environment.

Restore versus content migration

Restore is used to recreate an existing cluster on new hardware. Content migration is instead used to transfer content from one installation to another. When migrating content, system configuration entries like property files and routing are usually not transferred, or specific properties may be transferred on a case-by-case basis.

For migration, the items that are needed, in order, are typically:

  1. The dbacl_iris mysql/mariadb database with users and permissions
  2. Table schemata
  3. Persistent queries
  4. Web Workspaces and Dashboards

General restore process

The basic restore process is to first install a new cluster, and then to restore backed up configuration to this new cluster.

If IP addresses, machine names, or machine roles are different in the new cluster, relative to the one from which the backups were taken, configuration values (mainly in iris-environment, routing.yaml, and maybe in tailer config XML files) will need to be modified. This can be done from an etcd restore (after restoring etcd) by exporting, editing, and reimporting iris-environment.prop and routing.yaml; or, if config files have been backed up individually, it may be desirable to export the new files after the new cluster has been installed, and then diff them versus the backup files, to see what values should be changed.

After configuration has been restored or migrated to the new environment, all Deephaven services on all nodes should be restarted:

/usr/illumon/latest/dh_monit restart all

ACL DB backup and restore

The dbacl_iris database in MySQL/MariaDB stores Deephaven logins and groups and associated permissions for Deephaven objects.

It can be backed up and restored using MySQL native backup and restore tools.

Backup with:

mysqldump --user [user_name] --password=[password] --databases dbacl_iris > [backup_file]

Restore with:

mysql --user [user_name] --password=[password] -e "drop database if exists dbacl_iris"

mysql --user [user_name] --password=[password] -e "create database dbacl_iris"

mysql --user [user_name] --password=[password] dbacl_iris < [backup_file]

etcd / properties files backup and restore

etcd can be backed up and restored using etcd's own tools.

Backup can be done with:

DH_ETCD_DIR=/etc/sysconfig/illumon.d/etcd/client/root /usr/illumon/latest/bin/etcdctl.sh snapshot save [file_to_save_to]

Restore is covered in the etcd documentation, under the topic: "Restart cluster from majority failure". For Deephaven, the etcd restore process would be:

  1. Install the new Deephaven cluster.
  2. Connect to the etcd node where restore will be processed.
  3. Use DH_ETCD_DIR=/etc/sysconfig/illumon.d/etcd/client/root /usr/illumon/latest/bin/etcdctl.sh endpoint status -w table to get a list of etcd nodes in the cluster.
  4. Use DH_ETCD_DIR=/etc/sysconfig/illumon.d/etcd/client/root /usr/illumon/latest/bin/etcdctl.sh move-leader to force the local node to be the leader of the cluster.
  5. Use DH_ETCD_DIR=/etc/sysconfig/illumon.d/etcd/client/root /usr/illumon/latest/bin/etcdctl.sh member remove to remove the other nodes from the cluster until only the local node remains.
  6. Use DH_ETCD_DIR=/etc/sysconfig/illumon.d/etcd/client/root /usr/illumon/latest/bin/etcdctl.sh snapshot restore to restore the etcd database.
  7. Follow the rest of the steps in the linked etcd.io article for adding the other nodes back to the cluster.

Alternatively, the data contained in etcd can be exported as separate files and imported into the new cluster.

Note

See These processes and tools are documented in detail in the core Deephaven documentation:

  • Persistent Queries
    • Backup with sudo -u irisadmin /usr/illumon/latest/bin/iris controller_tool --export
    • Restore with sudo -u irisadmin /usr/illumon/latest/bin/iris controller_tool --import
  • Schema
    • Backup with sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schema export -d <path to existing directory where schema files should be written>
    • Restore with sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schema import -d <path to directory created by export>
  • Routing
    • Backup with sudo -u irisadmin /usr/illumon/latest/bin/dhconfig routing export -f <routing file to write>
    • Restore with sudo -u irisadmin /usr/illumon/latest/bin/dhconfig routing import <routing file to read>
  • Property files
    • Backup with sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties export -d <pre-existing directory where property files should be written>
    • Restore with sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import -f <file to import>

The persistent query export file gets written to: /db/TempFiles/irisadmin/controller_tool/controllerToolExport.xml. See the full controller tool documentation.

iris-defaults.prop should not be imported into the new system, regardless of whether performing a migration or restore. In general, properties files should not be imported during a migration.

Dashboard backup and restore

Dashboards and other Web environment data are stored in the WorkspaceData table.

It can be backed up by copying the entire /db/Intraday/DbInternal/WorkspaceData directory structure, or by exporting data to CSV using the WorkspaceDataTool:

sudo -u irisadmin /usr/illumon/latest/bin/iris workspace_data_tool -c -t <type> -f <file to write>

The two types which need to be exported are Workspace and Dashboard; as two separate actions.

Restore is accomplished by restoring the entire /db/Intraday/DbInternal/WorkspaceData directory structure, or by importing CSV data with the WorkspaceDataTool:

sudo -u irisadmin /usr/illumon/latest/bin/iris workspace_data_tool -rc -f <file to read>

See the full WorkspaceDataTool documentation.

If copying the WorkspaceData directory structure, ensure that ownership and permissions are preserved during the copying process.

Making data available to the new system

Environment-specific data files are stored under /db/Systems, /db/Users, /db/Intraday/<Namespace>, and /db/IntradayUser.

Typically /db/Users and /db/Systems are shared across nodes using NFS or some other similar type of shared storage. Attaching (or reattaching) such shared storage to the new servers will make historical and user data available to the new cluster.

/db/Intraday has subdirectories for each namespace. The DbInternal namespace is reserved for Deephaven internal operations. This directory structure should not be copied between installs, but other namespace subdirectories should be copied to the new ingestion server to make intraday data available to the DIS and LTDS processes in the new cluster. Similarly, /db/IntradayUser should be copied to the ingestion or user intraday server in the new cluster.

Ensure that file and directory ownership and permissions are preserved when copying data directories between servers. Note that there is installation-specific information under /db/TempFiles which must not be copied between installations.

id: backup-and-restore title: Configuration backup, restore, and migration concepts

There are four areas of configuration that need to be backed up and restored in order to deploy a new cluster that maintains the configuration of an old/existing cluster:

  1. MySQL/MariaDB - which contains ACL information such as Deephaven accounts and permissions in the DbAcl database.
  2. etcd - which contains the most of deephaven system configuration, including:
  • Persistent queries
  • Schemata
  • Data routing yaml
  • Properties files
  1. The WorkspaceData table - which contains Web dashboards, workspaces, and notebooks.
  2. File system files - these are under /etc/sysconfig/deephaven:
  • Host Config (if it has been modified to change heap settings for processes, etc): etc/sysconfig/deephaven/illumon.iris.hostconfig
  • Tailer config XML (if a custom one has been created)
  • Any custom certificate files needed for authentication with external services such as LDAP: /etc/sysconfig/deephaven/illumon.d.latest/resources/
  • Any custom jars: /etc/sysconfig/deephaven/illumon.d.latest/java_lib/
  • From the auth server node, dsakeyts.txt to capture any key-based authentication that has been set up for users: /etc/sysconfig/deephaven/auth/; in addition, if this will be replaced from backup on the new system, *.base64.txt key files from /etc/sysconfig/deephaven/auth should also be backed up.
  • It is a good safety measure to back up all of /etc/sysconfig/deephaven/auth to ensure that needed keys and passphrase files are available in case they are later needed for restore operations.

These are the areas that cover the internal configuration of the Deephaven cluster. In addition, there are of course any NFS, directory creation, firewall, OS, and/or other changes that have been made to configure the operation of the cluster in the customer's environment.

Restore versus content migration

Restore is used to recreate an existing cluster on new hardware. Content migration is instead used to transfer content from one installation to another. When migrating content, system configuration entries like property files and routing are usually not transferred, or specific properties may be transferred on a case-by-case basis.

For migration, the items that are needed, in order, are typically:

  1. The dbacl_iris mysql/mariadb database with users and permissions
  2. Table schemata
  3. Persistent queries
  4. Web Workspaces and Dashboards

General restore process

The basic restore process is to first install a new cluster, and then to restore backed up configuration to this new cluster.

If IP addresses, machine names, or machine roles are different in the new cluster, relative to the one from which the backups were taken, configuration values (mainly in iris-environment, routing.yaml, and maybe in tailer config XML files) will need to be modified. This can be done from an etcd restore (after restoring etcd) by exporting, editing, and reimporting iris-environment.prop and routing.yaml; or, if config files have been backed up individually, it may be desirable to export the new files after the new cluster has been installed, and then diff them versus the backup files, to see what values should be changed.

After configuration has been restored or migrated to the new environment, all Deephaven services on all nodes should be restarted:

/usr/illumon/latest/dh_monit restart all

ACL DB backup and restore

The dbacl_iris database in MySQL/MariaDB stores Deephaven logins and groups and associated permissions for Deephaven objects.

It can be backed up and restored using MySQL native backup and restore tools.

Backup with:

mysqldump --user [user_name] --password=[password] --databases dbacl_iris > [backup_file]

Restore with:

mysql --user [user_name] --password=[password] -e "drop database if exists dbacl_iris"

mysql --user [user_name] --password=[password] -e "create database dbacl_iris"

mysql --user [user_name] --password=[password] dbacl_iris < [backup_file]

etcd / properties files backup and restore

etcd can be backed up and restored using etcd's own tools.

Backup can be done with:

DH_ETCD_DIR=/etc/sysconfig/illumon.d/etcd/client/root /usr/illumon/latest/bin/etcdctl.sh snapshot save [file_to_save_to]

Restore is covered in the etcd documentation, under the topic: "Restart cluster from majority failure". For Deephaven, the etcd restore process would be:

  1. Install the new Deephaven cluster.
  2. Connect to the etcd node where restore will be processed.
  3. Use DH_ETCD_DIR=/etc/sysconfig/illumon.d/etcd/client/root /usr/illumon/latest/bin/etcdctl.sh endpoint status -w table to get a list of etcd nodes in the cluster.
  4. Use DH_ETCD_DIR=/etc/sysconfig/illumon.d/etcd/client/root /usr/illumon/latest/bin/etcdctl.sh move-leader to force the local node to be the leader of the cluster.
  5. Use DH_ETCD_DIR=/etc/sysconfig/illumon.d/etcd/client/root /usr/illumon/latest/bin/etcdctl.sh member remove to remove the other nodes from the cluster until only the local node remains.
  6. Use DH_ETCD_DIR=/etc/sysconfig/illumon.d/etcd/client/root /usr/illumon/latest/bin/etcdctl.sh snapshot restore to restore the etcd database.
  7. Follow the rest of the steps in the linked etcd.io article for adding the other nodes back to the cluster.

Alternatively, the data contained in etcd can be exported as separate files and imported into the new cluster.

Note

See These processes and tools are documented in detail in the core Deephaven documentation:

  • Persistent Queries
    • Backup with sudo -u irisadmin /usr/illumon/latest/bin/iris controller_tool --export
    • Restore with sudo -u irisadmin /usr/illumon/latest/bin/iris controller_tool --import
  • Schema
    • Backup with sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schema export -d <path to existing directory where schema files should be written>
    • Restore with sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schema import -d <path to directory created by export>
  • Routing
    • Backup with sudo -u irisadmin /usr/illumon/latest/bin/dhconfig routing export -f <routing file to write>
    • Restore with sudo -u irisadmin /usr/illumon/latest/bin/dhconfig routing import <routing file to read>
  • Property files
    • Backup with sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties export -d <pre-existing directory where property files should be written>
    • Restore with sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import -f <file to import>

The persistent query export file gets written to: /db/TempFiles/irisadmin/controller_tool/controllerToolExport.xml. See the full controller tool documentation.

iris-defaults.prop should not be imported into the new system, regardless of whether performing a migration or restore. In general, properties files should not be imported during a migration.

Dashboard backup and restore

Dashboards and other Web environment data are stored in the WorkspaceData table.

It can be backed up by copying the entire /db/Intraday/DbInternal/WorkspaceData directory structure, or by exporting data to CSV using the WorkspaceDataTool:

sudo -u irisadmin /usr/illumon/latest/bin/iris workspace_data_tool -c -t <type> -f <file to write>

The two types which need to be exported are Workspace and Dashboard; as two separate actions.

Restore is accomplished by restoring the entire /db/Intraday/DbInternal/WorkspaceData directory structure, or by importing CSV data with the WorkspaceDataTool:

sudo -u irisadmin /usr/illumon/latest/bin/iris workspace_data_tool -rc -f <file to read>

See the full WorkspaceDataTool documentation.

If copying the WorkspaceData directory structure, ensure that ownership and permissions are preserved during the copying process.

Making data available to the new system

Environment-specific data files are stored under /db/Systems, /db/Users, /db/Intraday/<Namespace>, and /db/IntradayUser.

Typically /db/Users and /db/Systems are shared across nodes using NFS or some other similar type of shared storage. Attaching (or reattaching) such shared storage to the new servers will make historical and user data available to the new cluster.

/db/Intraday has subdirectories for each namespace. The DbInternal namespace is reserved for Deephaven internal operations. This directory structure should not be copied between installs, but other namespace subdirectories should be copied to the new ingestion server to make intraday data available to the DIS and LTDS processes in the new cluster. Similarly, /db/IntradayUser should be copied to the ingestion or user intraday server in the new cluster.

Ensure that file and directory ownership and permissions are preserved when copying data directories between servers. Note that there is installation-specific information under /db/TempFiles which must not be copied between installations.