File system backup and restoration

This page lists the on-disk configuration under /etc/sysconfig/deephaven to back up. Most configuration is stored in etcd, but several files and directories hold additional configuration. File-on-disk configuration is generally related to process startup, authentication, and additional JARs/classes that extend Deephaven's functionality.

/etc/sysconfig/illumon.d and /etc/sysconfig/deephaven/illumon.d.latest are the same location: symbolic links to the directory for the currently installed version. This page uses whichever path is clearer for the item.

Summary of files and directories

Process startup configuration

Process startup override parameters such as heap are stored in /etc/sysconfig/deephaven/illumon.iris.hostconfig. This file exists on each node within a cluster, and its values for a given process can differ between nodes. For example, the db_tdcp (Table Data Cache Proxy) process on one node within the cluster may be assigned 6G of heap, whereas the db_tdcp process on a different node may be assigned 10G of heap.

Authentication files

Back up all of /etc/sysconfig/deephaven/auth so that needed keys and passphrase files are available if you need them for a restore. This directory stores sensitive certificates and keys, and should not be accessible to most users. Particular files of interest in auth include:

• priv-*.base64.txt: These files contain the private keys of system users, which allow those users to log in with a private key. If required, you can generate new keys for users.
• priv-controllerConsole.base64.txt: Despite matching the priv-*.base64.txt naming pattern, this file is not a login key. It is an RSA key pair the Controller uses to encrypt secrets. See Controller encryption key.
• dsakeys.txt: If you have enabled text-file-based key authentication (not enabled by default), you can define key-based authentication for users in dsakeys.txt. This file may be empty or missing if public keys are stored in the ACL store (etcd or MySQL).
• Certificates for authentication with external services such as LDAP or SAML: Properties set their locations, and Deephaven does not place them for you. If you use these services, back up the certificate files wherever your properties point. Prefer /etc/sysconfig/deephaven/auth over /etc/sysconfig/illumon.d/resources/, because resources is readable by all users.

The installer creates authusers.txt and superusers.txt in /etc/sysconfig/illumon.d/resources/, not auth, so backing up auth does not include them. Text-file password authentication is disabled by default (authentication.server.localusers.enabled=false), so authusers.txt matters only if you enabled it. File-based superusers are enabled by default (authentication.server.localsuperusers.enabled=true), and the default superusers.txt contains iris:*. Back up superusers.txt if you changed it or set authentication.server.localsuperusers.file to another path. On Kubernetes, this file is /etc/sysconfig/deephaven/superusers.txt, created from a Secret.

The /etc/sysconfig/deephaven/auth-user directory holds webServices-keystore.p12, the certificate the web service uses, and .webapi_passphrase, its passphrase. Both are readable by the members of the shared query group (dbquerygrp). For a restore on the same cluster, back up both files, because the keystore cannot be opened without its passphrase. Do not copy them to a different cluster. The installer generates the keystore with a subject alternative name based on the web server host name, so it is specific to that host. Instead, generate new credentials when you install the new cluster.

Custom tailer configuration

Store custom tailer configuration XML in /etc/sysconfig/illumon.d/resources. The location can vary between deployments. On Podman, tailers load configuration only from JARs; see File system backup and restoration (Podman).

Plugins and custom JARs

Deephaven is highly customizable, and installations may have customized JARs/classes and plugins that extend the functionality of the system. Customized JARs are generally in /etc/sysconfig/deephaven/illumon.d.latest/java_lib/. Processes started by the standard launch scripts and monit add every *.jar in this directory to their classpath. Loose .class files in java_lib are not loaded, so package classes in a JAR.

Do not copy plugins between Deephaven instances. Instead, re-install and re-activate them in the new environment. Plugins unpack to /etc/sysconfig/deephaven/plugins/, and activated plugins appear in /etc/sysconfig/deephaven/illumon.d.latest/plugins/.

Resources directory

/etc/sysconfig/illumon.d/resources/ holds customer-edited files on the process classpath, such as custom tailer XML, authusers.txt, superusers.txt, customer_jars.txt, and any custom .prop files. It also holds installer-managed files and links to files in trust and auth, so compare it with a fresh installation before restoring it. If DH_LINK_RESOURCES was set at installation, resources is a link to a location you own, so back up the link target.

Installer-managed directories

The installer creates the dh-config, schema, etcd, and monit directories under /etc/sysconfig/deephaven and links them from illumon.d. On Kubernetes, the installer does not link etcd and monit.

• dh-config holds Configuration Server connection files. The installer regenerates them, so regenerate them rather than restoring them.
• monit holds the service .conf and .disabled files that the installer enables according to the roles in cluster.cnf. On native installations, /etc/sysconfig/illumon.d/monit links to /etc/sysconfig/deephaven/monit, so they are the same directory and it also contains your local changes, such as changes to process autostart or monitoring thresholds. Back up this directory on each node. After an upgrade or rebuild, compare it with freshly generated defaults and reapply your site-specific changes rather than restoring the whole directory.
• schema holds schema files that the installer imports into etcd, including any directories you add under /etc/sysconfig/illumon.d/schema/. An etcd backup captures the current schemas, but not your source files, so also keep your schema source files.
• etcd holds etcd client credentials. Keep the dh_etcd_config.tgz archive that config_generator.sh creates. You unpack it with config_packager.sh.

config_packager.sh packages the trust, authentication, query, and connection files, and unpacks etcd configuration packages. It does not package customer schema, monit, or other customer-managed files, so transfer those separately. See Configuration in the filesystem for more about these directories.

Related documentation

• Backup and restoration overview
• File system backup and restoration (Podman)
• File system backup and restoration (Kubernetes)
• etcd and MySQL backup and restoration
• Configuration properties backup and restoration
• WorkspaceData backup and restoration
• Table migration