---
title: Multi-node configuration
---

In multi-node Deephaven deployments, configuration files such as certificates, truststores, etcd credentials, and connection settings must be synchronized across all servers. The `config_packager.sh` utility creates and extracts tar archives containing these files, simplifying the distribution process.

**Location:** `/usr/illumon/latest/install/config_packager.sh`

## When to use config_packager.sh

Use `config_packager.sh` when you need to:

- Set up additional nodes in a cluster after initial installation.
- Replace or update TLS certificates across all nodes.
- Recover from etcd cluster failures.
- Add new etcd nodes to an existing cluster.

> [!NOTE]
> During initial cluster installation and regular upgrades (such as certificate rotation), the Deephaven installer handles configuration distribution automatically. Manual use of `config_packager.sh` is primarily for manual maintenance and disaster recovery scenarios.

## Package types

The utility supports four package types, each containing a different set of configuration files:

| Package   | Archive name         | Contents                                                                                                                                                                                        |
| --------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **trust** | `iris_trust.tgz`     | Truststores, truststore passphrase, Configuration Server connection files                                                                                                                       |
| **query** | `dh_query.tgz`       | Everything in trust, plus TDCP login private key, controller encryption keys, and etcd client credentials                                                                                       |
| **auth**  | `iris_auth.tgz`      | Everything in query, plus additional etcd credentials                                                                                                                                           |
| **etcd**  | `dh_etcd_config.tgz` | etcd server and client configuration files (created by [`config_generator.sh`](../management/add-etcd-nodes.md#3-manually-create-the-new-etcd-configuration), unpacked by `config_packager.sh`) |

**Choosing the right package type:**

- Use **trust** packages for nodes that only need to verify certificates and connect to the Configuration Server (minimal footprint).
- Use **query** packages for query server and merge server nodes that need TDCP login keys, controller encryption keys, and etcd client access.
- Use **auth** packages for nodes running [administrative services](../admin-glossary.md#administrative-services) that need additional etcd client credentials.
- Use **etcd** packages for etcd cluster configuration, including server setup and client credential distribution.

## Prerequisites

Run the script as the installation service account, which has the correct permissions for all operations:

```bash
sudo -u irisadmin /usr/illumon/latest/install/config_packager.sh <type> <action>
```

## Basic workflow

The typical workflow for distributing configuration is:

1. **Package** configuration on the source machine (usually the infrastructure server).
2. **Copy** the archive to target machines using `scp` or similar.
3. **Unpackage** on each target machine.

## Usage

**Syntax:**

```bash
config_packager.sh <type> <action> [options]
```

**Types:**

- `trust` — Package or unpackage trust configuration (truststores, Configuration Server connection files).
- `query` — Package or unpackage query server configuration (everything in trust, plus TDCP keys and etcd client credentials).
- `auth` — Package or unpackage authentication configuration (everything in query, plus additional etcd credentials).
- `etcd` — Unpackage etcd server or client configuration (packaging is handled by `config_generator.sh`).

**Actions for auth, trust, and query packages:**

| Action      | Description                                                          |
| ----------- | -------------------------------------------------------------------- |
| `package`   | Create a tar archive in the current directory.                       |
| `unpackage` | Extract and apply files from a tar archive in the current directory. |

**Optional mode for auth, trust, and query packages:**

An optional third argument specifies the packaging mode:

- `lite` (default): Skips private keystores (`.p12`) and their passphrases. This is the standard mode used by the installer.
- `full`: Includes all files, including private keystores and passphrases. Use only for disaster recovery scenarios where keys cannot be regenerated on the target machine.

**Actions for etcd packages:**

| Action                    | Description                                                        |
| ------------------------- | ------------------------------------------------------------------ |
| `unpackage-server <n>`    | Unpackage etcd server configuration for server number `n`.         |
| `unpackage-client`        | Unpackage etcd client configuration.                               |
| `unpackage-global-client` | Unpackage etcd client configuration for all standard client roles. |

The `unpackage-global-client` action installs etcd credentials for: `controller`, `dispatcher`, `dispatcher-ro`, and `dh-resolver`. These are the roles needed by query server nodes.

> [!NOTE]
> The etcd package is created by `config_generator.sh`, not by `config_packager.sh`. The `config_packager.sh` utility only handles unpacking etcd configuration.

## Examples

### Distribute query configuration to a new node

On the source machine (infrastructure server):

```bash
cd /tmp
sudo -u irisadmin /usr/illumon/latest/install/config_packager.sh query package
# Creates dh_query.tgz in /tmp
```

Copy to the target machine:

```bash
scp /tmp/dh_query.tgz targetserver:/tmp/
```

On the target machine:

```bash
cd /tmp
sudo -u irisadmin /usr/illumon/latest/install/config_packager.sh query unpackage
```

### Distribute etcd configuration to cluster nodes

After generating etcd configuration with [`config_generator.sh`](../management/add-etcd-nodes.md#3-manually-create-the-new-etcd-configuration), distribute to each etcd server:

```bash
# Copy dh_etcd_config.tgz to each etcd server, then on each server:
cd /tmp
sudo -u irisadmin /usr/illumon/latest/install/config_packager.sh etcd unpackage-server <server_number>
sudo -u irisadmin /usr/illumon/latest/install/config_packager.sh etcd unpackage-client
```

The server number corresponds to the order in which IP addresses were specified when running `config_generator.sh` (1-indexed).

### Update trust configuration after certificate changes

```bash
# On the infrastructure server after updating certificates:
cd /tmp
sudo -u irisadmin /usr/illumon/latest/install/config_packager.sh trust package

# Copy and unpackage on each node (replace <target-node> with the hostname):
scp /tmp/iris_trust.tgz <target-node>:/tmp/
ssh <target-node> "cd /tmp && sudo -u irisadmin /usr/illumon/latest/install/config_packager.sh trust unpackage"
```

## Backup behavior

Before extracting files, `config_packager.sh` automatically backs up existing configuration to `/etc/sysconfig/deephaven/backups/`. Each backup is timestamped (for example, `trust_package.2025-06-03T110530.12345/`), allowing you to recover previous configurations if needed.

To restore from a backup, manually copy the files from the backup directory back to their original locations (such as `/etc/sysconfig/deephaven/auth/` or `/etc/sysconfig/deephaven/trust/`).

## Logging

The script logs all operations to `/var/log/deephaven/install_configuration/`. The most recent log for each operation type is symlinked as `current.config_packager_<type>_<action>.log` (for example, `current.config_packager_query_package.log`).

## Related documentation

- [Configuration file locations](./configuration-file-locations-overview.md)
- [Replication of security files](../security/hardening-technical-controls.md#replication-of-security-files)
- [Deephaven etcd cluster recovery guide](../ops-guide/etcd-recovery.md)
- [Add etcd nodes](../management/add-etcd-nodes.md)
- [Troubleshooting certificates](../troubleshooting/troubleshooting-certificates.md)