Configuration server overview

The Configuration Server manages most Deephaven server configurations. The Configuration Server uses etcd for its underlying data store. etcd is a distributed property store with built-in high availability and fault tolerance features. You can configure a Deephaven deployment with multiple Configuration Server instances to provide fault tolerance.

Server configuration

The Configuration Server process reads its properties files directly from etcd. The server uses the following properties:

  • configuration.server.port - the service will bind to this port.
  • configuration.server.ssl - the service will be secured by TLS unless this is set to false.

If SSL is enabled:

  • configuration.server.keyfile - file containing the server private key or the PKCS12 keystore.
  • configuration.server.key_alias - alias for the key; if it is stored in a keystore, the default is configuration_server.

The password for the key file is determined by the first of the following that is set:

  • environment variable CONFIGURATION_SERVER_PASSPHRASE
  • property configuration.server.passphrase.file
  • property configuration.server.passphrase

The key file is a PKCS12 keystore file protected by the configured password. The certificate and key named by the configured alias are extracted from this file.

Client configuration

Clients find a configuration service by using system options in the following order:

Note

Note that the properties are System Properties (e.g., set using the -D java option when starting a process). They cannot be set in a properties file.

  1. If the system property dh.config.client.host exists:

    • System property dh.config.client.host specifies the configuration server host.
    • System property dh.config.client.port specifies the service port.
    • If dh.config.client.cacert is set, connections use SSL and this specifies a trusted certificate file in PEM format, such as /etc/sysconfig/illumon.d/resources/truststore-iris.pem.
    • If set, system property connection_timeout_millis contains an integer value specifying the connection timeout.

  2. If the system property dh.config.client.bootstrap is set, the value must specify a directory containing one or more subdirectories, each containing files as follows (the installer generates these files and directories):

    • host contains the host's address. This should be an IP address or fully qualified domain name. The service’s certificate must contain this value in the SAN or CN field.
    • port contains the network port for the service. cacert is a PEM-formatted file containing the certificate trust chain for the server certificate (often just the self-signed certificate). This file may contain one or many trusted certificates.

    If multiple configuration directories exist, they will be treated as equivalent servers and automatically provide redundancy to clients. All servers must be configured the same way (see “Equivalent” configuration below).

    The bootstrap directory may optionally contain:

    • authority, which contains a string used for hostname verification on all servers. This string must be in the common name or subject alternative name of all the server certificates. “configuration-server” is the default if nothing is specified.

      If a custom authority is not configured, the host value for each server is used.

    • connection_timeout_millis contains an integer value specifying the timeout for connections to all machines.

  3. The default value for dh.config.client.bootstrap is /etc/sysconfig/illumon.d/dh-config/clients. If this location exists, it will be parsed as above.

  4. If no other configuration applies, the final default configuration is localhost:22023 with cacert path /etc/sysconfig/illumon.d/resources/truststore-iris.pem.

We anticipate that option 3 will be the active choice on the Deephaven servers in most cases. Option 2 is expected to be preferred for most remote clients, such as the Swing console.

“Equivalent” configurations

When multiple configuration servers are specified, they must match:

  • All must be configured for TLS, or none.
  • If an authority is defined (via the authority file), all the host certificates must contain that value in the SAN or CN field; e.g., DNS:configuration-server.
  • The cacert files specified may be independent and will effectively be concatenated by the client.

Accessing files directly through etcd

When the Configuration Server is not running, you can still use the dhconfig tool by adding the --etcd option to access etcd directly. This requires a user with administrative privileges and only works on servers with the necessary etcd client files.

You can examine and change data in etcd using etcdctl.sh. You will never need to do this in normal operation.

If a file or other value has been deleted or overwritten with an invalid file and no backup is available, it may be possible to recover an older version of the value.