Configuration Server runbook

The Configuration Server is a core Deephaven system service that serves configuration information to all other Deephaven client and server processes. Most Deephaven configuration files and settings are stored in etcd, and the Configuration Server is responsible for writing configuration to etcd as well as reading configuration from etcd and serving that information over gRPC to all other Deephaven processes. See the Configuration Server overview for more details.

Impact of Configuration Server failure

Level	Impact
Sev 1 - Critical	None of the system processes will be able to start. Existing processes will continue running but cannot access or update configuration.

Caution

The Configuration Server is one of the most critical services. Without it, no other Deephaven services can start or access cluster configuration.

Configuration Server responsibilities

The Configuration Server manages the following resources:

Property files — Application configuration properties for all Deephaven services.
Table schemas — Table definitions including columns, types, and partitioning.
Cluster routing file — Data location and server routing configuration.
Persistent Query definitions — PQ configurations and schedules.

All configuration changes flow through the Configuration Server, which ensures consistency with etcd.

Configuration Server dependencies

The Configuration Server requires:

etcd cluster — Must be running and accessible for storing/retrieving configuration.
etcd client configuration files — Located at /etc/sysconfig/deephaven/etcd/client.

Optional dependencies:

Authentication Server — Required to authenticate user requests, but Configuration Server can start without it (will be unable to authenticate user operations).

The Configuration Server is typically the first Deephaven service to start after etcd.

Checking Configuration Server status

Check process is running with monit:

dh_monit status configuration_server

Expected output should show status Running.

Test Configuration Server connectivity:

# Using dhconfig (connects to Configuration Server by default)
sudo /usr/illumon/latest/bin/dhconfig properties list

If this returns a list of property files, the Configuration Server is accessible and functioning.

Viewing Configuration Server logs

View application log:

cat /var/log/deephaven/configuration_server/ConfigurationServer.log.current

Tail the log to follow in real-time:

tail -f /var/log/deephaven/configuration_server/ConfigurationServer.log.current

List historical log files:

ls -ltr /var/log/deephaven/configuration_server/configuration_server.log.????-??-??

View process stdout/stderr logs:

cat /var/log/deephaven/configuration_server/configuration_server.log.$(date +%Y-%m-%d)

Restart procedure

Restart the Configuration Server:

dh_monit restart configuration_server

Warning

Restarting the Configuration Server temporarily interrupts configuration access for all Deephaven services. Running processes continue operating, but configuration changes cannot be made and new processes cannot start until the Configuration Server is back online.

Verify the restart was successful:

dh_monit status configuration_server

Monitor the log during startup:

tail -f /var/log/deephaven/configuration_server/ConfigurationServer.log.current

Expected startup messages:

Successful connection to etcd.
Server listening on configured port (default: 22023).
Configuration cache initialized.
Ready to accept requests.

Configuring the Configuration Server

Caution

While the Configuration Server can be configured without TLS or using non-standard ports or keystores, it is strongly recommended that you do not alter these default values, as many automated installation processes only support the default values. TLS should never be disabled unless you have very strong network isolation and fully trust all users with access to your network.

TLS keystore management

The Configuration Server p12 keystore is generated and managed automatically by Deephaven installation processes.

Keystore location: /etc/sysconfig/deephaven/auth/keystore.configuration_server.p12

Passphrase location: /etc/sysconfig/deephaven/auth/.configuration_server_passphrase

Trust store: /etc/sysconfig/deephaven/trust/truststore-iris.p12

The default keystore is created using a self-signed root CA which is added automatically to the system truststore. You should only consider changing these settings if you are fully prepared to own the creation, consumption, and maintenance of this keystore.

etcd client configuration

The Configuration Server requires etcd client configuration files located at /etc/sysconfig/deephaven/etcd/client/. These directories contain:

endpoints — List of etcd node addresses and ports.
username — etcd RBAC username for this role.
password — etcd RBAC password (encrypted).
ca.crt — Certificate authority for TLS verification.

Multiple client configurations exist for different access patterns (read-only, read-write, specific namespaces).

See etcd runbook for detailed etcd client configuration.

Interacting with the Configuration Server

The primary interface to the Configuration Server is the dhconfig command-line tool.

Using dhconfig

Default behavior: dhconfig connects to Configuration Server (recommended)

# List all property files
sudo /usr/illumon/latest/bin/dhconfig properties list

# View a specific property file
sudo /usr/illumon/latest/bin/dhconfig properties export iris-common.prop

# List all table schemas
sudo /usr/illumon/latest/bin/dhconfig schemas list

# View a specific schema
sudo /usr/illumon/latest/bin/dhconfig schemas export MyNamespace.MyTable

Direct etcd access: When Configuration Server is unavailable

# Use --etcd flag to bypass Configuration Server
sudo /usr/illumon/latest/bin/dhconfig properties list --etcd

See dhconfig command reference for complete documentation.

Managing configuration through Configuration Server

Property files

# Import (update) a property file
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import /path/to/updated.prop

Table schemas

# Import (deploy) a new schema
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schemas import --file /path/to/schema.xml

# List all schemas
sudo /usr/illumon/latest/bin/dhconfig schemas list --namespace MyNamespace

Routing configuration

# Export routing configuration
sudo /usr/illumon/latest/bin/dhconfig routing export /tmp/routing.yml

# Import updated routing
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig routing import /tmp/routing.yml

Persistent Queries

# List all Persistent Queries
sudo /usr/illumon/latest/bin/dhconfig pq status

# View specific PQ
sudo /usr/illumon/latest/bin/dhconfig pq status --serial 12345

Configuration Server caching

The Configuration Server caches configuration in memory for performance:

Cache behavior:

Configuration is cached when first accessed.
Cache is updated when configuration changes through Configuration Server.
Cache may become stale if etcd is modified directly.

Cache invalidation:

If configuration is modified directly in etcd (bypassing Configuration Server), restart the Configuration Server to reload its cache:

dh_monit restart configuration_server

Backup and restore

Configuration Server state is entirely stored in etcd. Backup etcd to backup all configuration:

# Backup etcd (includes all configuration)
sudo -u irisadmin /usr/illumon/latest/bin/etcdctl.sh snapshot save /backup/etcd-snapshot-$(date +%Y%m%d).db

See etcd runbook for complete backup and restore procedures.

Direct etcd access scenarios

Use dhconfig --etcd to bypass Configuration Server when:

Configuration Server is down:

sudo /usr/illumon/latest/bin/dhconfig properties export --etcd iris-common.prop

Emergency configuration changes:

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import --etcd /path/to/emergency-config.prop

Troubleshooting configuration issues:

sudo /usr/illumon/latest/bin/dhconfig schemas list --etcd

Direct access uses etcd client configuration files and requires appropriate etcd RBAC permissions.

Configuration files and locations

monit configuration: /etc/sysconfig/illumon.d/monit/configuration_server.conf

Property files:

/etc/sysconfig/illumon.d/resources/iris-common.prop
/etc/sysconfig/illumon.d/resources/configuration_server.prop

TLS keystore: /etc/sysconfig/deephaven/auth/keystore.configuration_server.p12

TLS passphrase: /etc/sysconfig/deephaven/auth/.configuration_server_passphrase

Truststore: /etc/sysconfig/deephaven/trust/truststore-iris.p12

etcd client config: /etc/sysconfig/deephaven/etcd/client/

Log directory: /var/log/deephaven/configuration_server/