Multi-node configuration

In multi-node Deephaven deployments, configuration files such as certificates, truststores, etcd credentials, and connection settings must be synchronized across all servers. The config_packager.sh utility creates and extracts tar archives containing these files, simplifying the distribution process.

Location: /usr/illumon/latest/install/config_packager.sh

When to use config_packager.sh

Use config_packager.sh when you need to:

• Set up additional nodes in a cluster after initial installation.
• Replace or update TLS certificates across all nodes.
• Recover from etcd cluster failures.
• Add new etcd nodes to an existing cluster.

Package types

The utility supports four package types, each containing a different set of configuration files:

Choosing the right package type:

• Use trust packages for nodes that only need to verify certificates and connect to the Configuration Server (minimal footprint).
• Use query packages for query server and merge server nodes that need TDCP login keys, controller encryption keys, and etcd client access.
• Use auth packages for nodes running administrative services that need additional etcd client credentials.
• Use etcd packages for etcd cluster configuration, including server setup and client credential distribution.

Prerequisites

Run the script as the installation service account, which has the correct permissions for all operations:

Basic workflow

The typical workflow for distributing configuration is:

1. Package configuration on the source machine (usually the infrastructure server).
2. Copy the archive to target machines using scp or similar.
3. Unpackage on each target machine.

Usage

Syntax:

Types:

• trust — Package or unpackage trust configuration (truststores, Configuration Server connection files).
• query — Package or unpackage query server configuration (everything in trust, plus TDCP keys and etcd client credentials).
• auth — Package or unpackage authentication configuration (everything in query, plus additional etcd credentials).
• etcd — Unpackage etcd server or client configuration (packaging is handled by config_generator.sh).

Actions for auth, trust, and query packages:

Optional mode for auth, trust, and query packages:

An optional third argument specifies the packaging mode:

• lite: Skips private keystores (.p12) and their passphrases. Use this when you plan to generate private keys on the target machine using iris_keygen.sh.
• full (default): Includes all files, including private keystores and passphrases. Use only for disaster recovery scenarios where keys cannot be regenerated on the target machine.

The lite option skips:

• Private keystores (keystore.authserver.p12, keystore.controller.p12, keystore.configuration_server.p12, .webServices-keystore.p12)
• Passphrase files (.auth_passphrase, .controller_passphrase, .configuration_server_passphrase, .webapi_passphrase)

Actions for etcd packages:

The unpackage-global-client action installs etcd credentials for: controller, dispatcher, dispatcher-ro, and dh-resolver. These are the roles needed by query server nodes.

Use unpackage-client on infrastructure nodes that need the full etcd client configuration. Use unpackage-global-client on query server nodes that only need the shared client directories.

etcd configuration workflow

The etcd configuration files are created and distributed through a specific workflow:

1. Generation — The /usr/illumon/latest/install/etcd/config_generator.sh script generates etcd configuration files on the primary infrastructure node.

   Required argument:

   • --servers — List of etcd server addresses

   Common optional arguments:

   • --cluster-id — Specify the dh-etcd cluster ID
   • --max-procs — Max processing units (defaults to nproc)
   • --output-dir — Output directory for the config tar
   • --self-signed — Create self-signed TLS keys
   • --use-tls [y|n] — Use TLS for etcd endpoints (default: y)
   • --client-only [y|n] — Generate client configs only (for Kubernetes deployments where etcd is managed separately)
   • --update <path> — Update an existing dh_etcd_config.tgz package

   Examples:

2. Packaging — The generated files are packaged into /etc/sysconfig/deephaven/etcd/dh_etcd_config.tgz.

3. Distribution — Copy the .tgz file to each target node.

4. Unpacking — Run config_packager.sh to extract the files:

   • For etcd servers: config_packager.sh etcd unpackage-server <server_number>
   • For infrastructure nodes: config_packager.sh etcd unpackage-client (full client directory)
   • For query server nodes: config_packager.sh etcd unpackage-global-client (shared directories only)

The unpacked client files are placed in /etc/sysconfig/deephaven/etcd/client/.

Common workflows

Adding a query server node

On the source machine (infrastructure server):

Copy to the target machine:

On the target machine:

Distributing trust to multiple auth/config servers

When deploying multiple Authentication Servers or Configuration Servers, each needs its own certificate while sharing a common truststore:

1. On the first node, generate keys and create a trust package:

2. Copy iris_trust.tgz to the next node.

3. On the next node, unpack and generate its own keys:

4. Repeat steps 2-3 for each additional node.

5. Copy the final trust package to all nodes (including previously initialized ones) so each has the complete truststore.

Distributing etcd configuration to cluster nodes

After generating etcd configuration with config_generator.sh, distribute to each etcd server:

The server number corresponds to the order in which IP addresses were specified when running config_generator.sh (1-indexed).

Updating trust configuration after certificate changes

Package contents

Trust package (iris_trust.tgz)

• /etc/sysconfig/deephaven/trust/ — Truststores and passphrase
• /etc/sysconfig/deephaven/dh-config/ — Configuration Server client files
• /etc/sysconfig/deephaven/auth/dsakeys.txt — Public keys for unattended logins
• /etc/sysconfig/deephaven/auth/db_acl_ro* — ACL read-only credentials

Query package (dh_query.tgz)

Everything in the trust package, plus:

• /etc/sysconfig/deephaven/auth/priv-tdcp.base64.txt — TDCP private key
• /etc/sysconfig/deephaven/auth/priv-authreconnect.base64.txt — Auth reconnect key
• /etc/sysconfig/deephaven/etcd/client/ — etcd client directories (acl-ro, props-ro, dispatcher, controller, dh-resolver, worker)

Auth package (iris_auth.tgz)

Everything in the query package, plus:

• Private keystores for Authentication Server, Configuration Server, Controller, and Web API
• Keystore passphrases
• Private keys for system accounts (priv-iris.base64.txt, priv-merge.base64.txt)

etcd package (dh_etcd_config.tgz)

• etcd server configuration files
• etcd client configuration directories
• TLS certificates for etcd communication

Backup and recovery

Automatic backups during unpack

Before extracting files, config_packager.sh automatically backs up existing configuration to:

The timestamp format is YYYY-MM-DDTHHMMSS.NNNNN (ISO 8601 date with nanoseconds). For example:

• auth_package.2025-05-14T131532.12345/
• etcd/server.2025-05-14T131532.12345-1/
• etcd/client.2025-05-14T131532.12345/

Within each backup directory, the original directory structure is preserved (e.g., auth/, trust/, etcd/).

To restore from a backup, manually copy the files from the backup directory back to their original locations (such as /etc/sysconfig/deephaven/auth/ or /etc/sysconfig/deephaven/trust/).

Logging

The script logs all operations to /var/log/deephaven/install_configuration/. The most recent log for each operation type is symlinked as current.config_packager_<type>_<action>.log (for example, current.config_packager_query_package.log).

Recovery if unpacking fails

If an unpack operation fails partway through, the system may be in an inconsistent state. Recovery options depend on how far the operation progressed:

1. Check the log file — Detailed logs are written to /var/log/deephaven/install_configuration/. The most recent log is symlinked as current.config_packager_{type}_{command}.log.

2. Restore from automatic backups — If the unpack began overwriting files before failing, the original files are in /etc/sysconfig/deephaven/backups/. Copy them back to their original locations:

3. Re-run the unpack — After fixing the underlying issue (permissions, disk space, etc.), re-run the unpack command. The script will create new backups of the current state before overwriting.

4. Restore from system backup — If automatic backups are incomplete or corrupted, restore from your regular system backups.

Common failure causes

Related documentation

• Configuration file locations
• Replication of security files
• etcd
• etcd recovery
• Add etcd nodes
• Public and private keys
• Troubleshooting certificates