---
id: runbook-config-server
title: Configuration Server runbook
---

The Configuration Server is a core Deephaven system service that serves configuration information to all other Deephaven client and server processes. Most Deephaven configuration files and settings are stored in etcd, and the Configuration Server is responsible for writing configuration to etcd as well as reading configuration from etcd and serving that information over gRPC to all other Deephaven processes. See the [Configuration Server overview](../core-components/configserver.md) for more details.

## Impact of Configuration Server failure

| Level            | Impact                                                                                                                                  |
| :--------------- | :-------------------------------------------------------------------------------------------------------------------------------------- |
| Sev 1 - Critical | None of the system processes will be able to start. Existing processes will continue running but cannot access or update configuration. |

> [!CAUTION]
> The Configuration Server is one of the most critical services. Without it, no other Deephaven services can start or access cluster configuration.

## Configuration Server responsibilities

The Configuration Server manages the following resources:

- **Property files** — Application configuration properties for all Deephaven services.
- **Table schemas** — Table definitions including columns, types, and partitioning.
- **Cluster routing file** — Data location and server routing configuration.
- **Persistent Query definitions** — PQ configurations and schedules.

All configuration changes flow through the Configuration Server, which ensures consistency with etcd.

## Configuration Server dependencies

The Configuration Server requires:

1. **etcd cluster** — Must be running and accessible for storing/retrieving configuration.
2. **etcd client configuration files** — Located at `/etc/sysconfig/deephaven/etcd/client`.

**Optional dependencies:**

- **Authentication Server** — Required to authenticate user requests, but Configuration Server can start without it (will be unable to authenticate user operations).

The Configuration Server is typically the first Deephaven service to start after etcd.

## Checking Configuration Server status

Check process is running with monit:

```bash
dh_monit status configuration_server
```

Expected output should show status `Running`.

Test Configuration Server connectivity:

```bash
# Using dhconfig (connects to Configuration Server by default)
sudo /usr/illumon/latest/bin/dhconfig properties list
```

If this returns a list of property files, the Configuration Server is accessible and functioning.

## Viewing Configuration Server logs

View application log:

```bash
cat /var/log/deephaven/configuration_server/ConfigurationServer.log.current
```

Tail the log to follow in real-time:

```bash
tail -f /var/log/deephaven/configuration_server/ConfigurationServer.log.current
```

List historical log files:

```bash
ls -ltr /var/log/deephaven/configuration_server/configuration_server.log.????-??-??
```

View process stdout/stderr logs:

```bash
cat /var/log/deephaven/configuration_server/configuration_server.log.$(date +%Y-%m-%d)
```

## Restart procedure

Restart the Configuration Server:

```bash
dh_monit restart configuration_server
```

> [!WARNING]
> Restarting the Configuration Server temporarily interrupts configuration access for all Deephaven services. Running processes continue operating, but configuration changes cannot be made and new processes cannot start until the Configuration Server is back online.

Verify the restart was successful:

```bash
dh_monit status configuration_server
```

Monitor the log during startup:

```bash
tail -f /var/log/deephaven/configuration_server/ConfigurationServer.log.current
```

**Expected startup messages:**

- Successful connection to etcd.
- Server listening on configured port (default: 22023).
- Configuration cache initialized.
- Ready to accept requests.

## Configuring the Configuration Server

> [!CAUTION]
> While the Configuration Server can be configured without TLS or using non-standard ports or keystores, it is strongly recommended that you do not alter these default values, as many automated installation processes only support the default values. TLS should never be disabled unless you have very strong network isolation and fully trust all users with access to your network.

### TLS keystore management

The Configuration Server p12 keystore is generated and managed automatically by Deephaven installation processes.

**Keystore location:** `/etc/sysconfig/deephaven/auth/keystore.configuration_server.p12`

**Passphrase location:** `/etc/sysconfig/deephaven/auth/.configuration_server_passphrase`

**Trust store:** `/etc/sysconfig/deephaven/trust/truststore-iris.p12`

The default keystore is created using a self-signed root CA which is added automatically to the system truststore. You should only consider changing these settings if you are fully prepared to own the creation, consumption, and maintenance of this keystore.

### etcd client configuration

The Configuration Server requires etcd client configuration files located at `/etc/sysconfig/deephaven/etcd/client/`. These directories contain:

- `endpoints` — List of etcd node addresses and ports.
- `username` — etcd RBAC username for this role.
- `password` — etcd RBAC password (encrypted).
- `ca.crt` — Certificate authority for TLS verification.

Multiple client configurations exist for different access patterns (read-only, read-write, specific namespaces).

See [etcd runbook](runbook-etcd.md) for detailed etcd client configuration.

## Interacting with the Configuration Server

The primary interface to the Configuration Server is the `dhconfig` command-line tool.

### Using dhconfig

**Default behavior:** `dhconfig` connects to Configuration Server (recommended)

```bash
# List all property files
sudo /usr/illumon/latest/bin/dhconfig properties list

# View a specific property file
sudo /usr/illumon/latest/bin/dhconfig properties export iris-common.prop

# List all table schemas
sudo /usr/illumon/latest/bin/dhconfig schemas list

# View a specific schema
sudo /usr/illumon/latest/bin/dhconfig schemas export MyNamespace.MyTable
```

**Direct etcd access:** When Configuration Server is unavailable

```bash
# Use --etcd flag to bypass Configuration Server
sudo /usr/illumon/latest/bin/dhconfig properties list --etcd
```

See [dhconfig command reference](../configuration/dhconfig/overview.md) for complete documentation.

## Managing configuration through Configuration Server

### Property files

```bash
# Import (update) a property file
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import /path/to/updated.prop
```

### Table schemas

```bash
# Import (deploy) a new schema
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schemas import --file /path/to/schema.xml

# List all schemas
sudo /usr/illumon/latest/bin/dhconfig schemas list --namespace MyNamespace
```

### Routing configuration

```bash
# Export routing configuration
sudo /usr/illumon/latest/bin/dhconfig routing export /tmp/routing.yml

# Import updated routing
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig routing import /tmp/routing.yml
```

### Persistent Queries

```bash
# List all Persistent Queries
sudo /usr/illumon/latest/bin/dhconfig pq status

# View specific PQ
sudo /usr/illumon/latest/bin/dhconfig pq status --serial 12345
```

## Configuration Server caching

The Configuration Server caches configuration in memory for performance:

**Cache behavior:**

- Configuration is cached when first accessed.
- Cache is updated when configuration changes through Configuration Server.
- Cache may become stale if etcd is modified directly.

**Cache invalidation:**

If configuration is modified directly in etcd (bypassing Configuration Server), restart the Configuration Server to reload its cache:

```bash
dh_monit restart configuration_server
```

## Backup and restore

Configuration Server state is entirely stored in etcd. Backup etcd to backup all configuration:

```bash
# Backup etcd (includes all configuration)
sudo -u irisadmin /usr/illumon/latest/bin/etcdctl.sh snapshot save /backup/etcd-snapshot-$(date +%Y%m%d).db
```

See [etcd runbook](runbook-etcd.md) for complete backup and restore procedures.

## Direct etcd access scenarios

Use `dhconfig --etcd` to bypass Configuration Server when:

**Configuration Server is down:**

```bash
sudo /usr/illumon/latest/bin/dhconfig properties export --etcd iris-common.prop
```

**Emergency configuration changes:**

```bash
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import --etcd /path/to/emergency-config.prop
```

**Troubleshooting configuration issues:**

```bash
sudo /usr/illumon/latest/bin/dhconfig schemas list --etcd
```

Direct access uses etcd client configuration files and requires appropriate etcd RBAC permissions.

## Configuration files and locations

**monit configuration:** `/etc/sysconfig/illumon.d/monit/configuration_server.conf`

**Property files:**

- `/etc/sysconfig/illumon.d/resources/iris-common.prop`
- `/etc/sysconfig/illumon.d/resources/configuration_server.prop`

**TLS keystore:** `/etc/sysconfig/deephaven/auth/keystore.configuration_server.p12`

**TLS passphrase:** `/etc/sysconfig/deephaven/auth/.configuration_server_passphrase`

**Truststore:** `/etc/sysconfig/deephaven/trust/truststore-iris.p12`

**etcd client config:** `/etc/sysconfig/deephaven/etcd/client/`

**Log directory:** `/var/log/deephaven/configuration_server/`

## Related documentation

- [Configuration Server overview](../core-components/configserver.md)
- [System processes overview](../architecture/architecture-overview.md)
- [dhconfig command reference](../configuration/dhconfig/overview.md)
- [etcd runbook](runbook-etcd.md)
- [Authentication Server runbook](runbook-authentication-server.md)