Deephaven public and private key configuration

This guide details the server-side configuration of public and private keys and certificates within a Deephaven deployment. It covers TLS keystores and truststores essential for securing communication between Deephaven services, as well as other specialized keys used by internal server components for their operations.

Proper configuration of these keys and certificates is crucial for a secure and functional Deephaven environment. If you encounter connection issues that might be related to SSL/TLS, refer to the Debug SSL/TLS connections section of the troubleshooting guide.

TLS Keystores for Services

Deephaven uses a one certificate per machine model. All processes on a machine use PKCS12 keystores derived from a per-machine certificate. There are two approaches to providing these certificates:

User-supplied certificates — For installations without Envoy, or Envoy installations with DH_GENERATE_CERTIFICATES=false, you supply a TLS certificate for each machine, signed by a trusted CA (your organization's PKI or a public CA). Each certificate should have the minimum possible scope — ideally a single Subject Alternative Name (SAN) matching the machine's configured hostname or FQDN (DH_NODE_N_DOMAIN).

Installer-generated internal certificates — For Envoy installations with DH_CONFIGURE_ENVOY=true and DH_GENERATE_CERTIFICATES=true, the installer generates per-node internal certificates signed by an internal CA. All clients route through Envoy rather than connecting to backend services directly. You must still provide the Envoy-facing certificate via DH_ENVOY_LOCAL_CERT. An external truststore is only generated when DH_CA_CERTIFICATE is set.

The generate_certificates.sh script builds service keystores from the machine's certificate. It has four options that all copy the same input certificate to service-specific output locations:

--auth — Authentication Server keystore
--config — Configuration Server keystore
--controller — Controller keystore
--web — Web API Server keystore (also used by dispatchers, DIS, and most other secured processes)

On single-role nodes, run the script with just the required flag (e.g., generate_certificates.sh --web). On multi-role nodes, combine all required flags in a single invocation.

Caution

The script backs up and removes keystores for any role flags not specified. Running --web alone on a node that also runs auth, config, or controller services will remove those keystores. Always include all applicable flags together: generate_certificates.sh --auth --config --controller --web.

Warning

When certificates, keys, or truststores are not configured correctly, connections will fail. If you encounter SSL/TLS connection problems, see debugging SSL/TLS connections for diagnostic parameters.

Authentication server

The Authentication Server's keystore (generate_certificates.sh --auth) contains the certificate and private key of a TLS-enabled service. You must keep this file private and not distribute it to clients. The keystore file is owned by user irisadmin with permission set to 400:

[-r-------- irisadmin irisadmin]  keystore.authserver.p12

The keystore file is protected by a unique randomly generated password stored in base64 encoded format in a read-only hidden file owned by user irisadmin with permission set to 400:

[-r-------- irisadmin irisadmin]  .auth_passphrase

Properties file sections

These file locations are specified by the following properties, which need to be accessible in the authentication_server and db_acl_write_server contexts.

[service.name=db_acl_write_server|authentication_server] {
    tls.keystore=/etc/sysconfig/illumon.d/auth/keystore.authserver.p12
    tls.passphrase.file=/db/TempFiles/irisadmin/.auth_passphrase
}

Authorized keys file configuration

For private key authentication, the Authentication Server reads public keys from a file specified by the authentication.server.authorizedkeys.file property. This is disabled by default (authentication.server.authorizedkeys.enabled=false).

Each line in the file contains a username and base64-encoded public key, separated by a space:

username1 BASE64_ENCODED_PUBLIC_KEY_1
username2 BASE64_ENCODED_PUBLIC_KEY_2

Example property setting:

[service.name=authentication_server] {
    authentication.server.authorizedkeys.enabled=true
    authentication.server.authorizedkeys.file=dsakeys.txt
}

The default authorized keys file is dsakeys.txt, located in /etc/sysconfig/deephaven/auth/.

Configuration server

The Configuration Server's keystore (generate_certificates.sh --config) contains the certificate and private key of a TLS-enabled service. You must keep this file private and not distribute it to clients. The keystore file is owned by user irisadmin with permission set to 400:

[-r-------- irisadmin irisadmin] keystore.configuration_server.p12

The Configuration Server keystore file is also protected by a unique randomly generated password stored in base64 encoded format in a read-only hidden file owned by user irisadmin with permission set to 400:

[-r-------- irisadmin irisadmin] .configuration_server_passphrase

Properties file sections

These file locations are specified by the following properties, which need to be accessible in the configuration_server context:

[service.name=configuration_server] {
    configuration.server.ssl=true
    configuration.server.keyfile=/etc/sysconfig/illumon.d/auth/keystore.configuration_server.p12
    configuration.server.passphrase.file=/db/TempFiles/irisadmin/.configuration_server_passphrase
}

Web API Service keystore

The Web API Server's keystore (generate_certificates.sh --web) contains the certificate and private key of a TLS-enabled service. You must keep this file private and not store it in a non-secure location such as a git repository. The file is saved as:

[-r-------- irisadmin irisadmin] webServices-keystore.p12

The keystore file is protected by a unique randomly generated password stored in base64 encoded format in a read-only hidden file owned by user irisadmin with permission set to 400:

[-r-------- irisadmin irisadmin] .webapi_passphrase

Properties file sections

These file locations are specified by the following properties, which need to be accessible in the web_api_service context.

[service.name=web_api_service] {
    WebServer.tls.keystore=/etc/sysconfig/illumon.d/auth/webServices-keystore.p12
    WebServer.tls.passphrase.file=/db/TempFiles/irisadmin/.webapi_passphrase
}

TLS Truststore

A TLS truststore file contains the root certificates that issued any certificates that should be trusted. This file is stored in two formats: a PKCS12 file verified by a password, and a PEM file which does not require a password.

An explicit Deephaven truststore is only needed when your certificates are signed by a CA that isn't already in the system/JRE default trust store — for example, your organization's own PKI. If your certificates are signed by a publicly trusted CA, servers and clients can rely on the system default trust store instead.

The generate_truststore.sh script builds the truststore files. It runs automatically during installation only when DH_CA_CERTIFICATE is set (for a user-supplied CA), or to build an internal-only truststore when DH_GENERATE_CERTIFICATES=true (Envoy-generated internal certificates). When a truststore is used, all nodes and clients need access to it, and a single truststore should be used across all Deephaven nodes. If your organization has an existing PKI, you can import your CA's certificate into the truststore for use with Deephaven.

The truststore does not contain secrets, and all clients need access to the password for the password-protected file.

Truststore filenames

/etc/sysconfig/deephaven/trust/truststore-iris.p12
/etc/sysconfig/deephaven/trust/truststore-iris.pem

Passphrase file

/etc/sysconfig/deephaven/trust/truststore_passphrase

Properties

configuration.server.cacertfile=/etc/sysconfig/illumon.d/resources/truststore-iris.pem
tls.truststore=/etc/sysconfig/illumon.d/resources/truststore-iris.p12
tls.truststore.passphrase.file=/etc/sysconfig/illumon.d/resources/truststore_passphrase

These default paths are symlinks to the truststore files under /etc/sysconfig/deephaven/trust/ listed above.

Note

These properties should be available in all contexts.

Updating certificates

Do not manually edit keystore files — they are derived from input certificates and private keys. To update certificates, provide the new certificate materials to the installer and re-run it.

Alternatively, you can manually replace the machine's certificate files and regenerate the keystores:

machine_name="$(< /etc/deephaven/DH_NODE_NAME)"
cp new-certificate.pem "/etc/deephaven/certs/${machine_name}.pem"
cp new-certificate.key "/etc/deephaven/certs/${machine_name}.key"
# This example includes all four role flags; include only the flags for the roles this node runs (see above).
/usr/illumon/latest/install/certs/generate_certificates.sh --auth --config --controller --web

Caution

Always include all role flags applicable to the node in a single invocation. The script backs up and removes keystores for any role flags not specified.

If the new certificate is signed by a different CA than the previous one, you must also update the truststore so that clients and servers trust the new chain:

Update the CA bundle (e.g., /etc/deephaven/certs/ca/ca.crt) with the new CA certificate.
Run generate_truststore.sh to regenerate the truststore files.
Distribute the updated truststore to all nodes and clients.
Restart affected processes.

For more details, see TLS certificates and the basic installation guide.

Authorized keys

A Deephaven-authorized keys file associates usernames with keys and allows non-interactive processes to authenticate with a key instead of a password. The process of authenticating must have the private key corresponding to the public key in this file. This file is disabled by default, with public keys maintained in the ACL store (etcd or MySQL).

As of Grizzly (1.20240517), Deephaven uses the secp256r1 Elliptic Curve to generate key pairs, which is currently recommended by FIPS.186-5. Deephaven's authentication server generates a random nonce, which the client then hashes using SHA256 and signs with the private key. In versions of Deephaven from 1.20221001 to 1.20231218, Deephaven generated 2048-bit DSA keys as defined in FIPS.186-4. Prior to 1.20221001, 512-bit DSA keys were generated. For backward compatibility, Deephaven supports using ECDSA and DSA keys for authentication so that users can continue using their existing keys.

The authorized keys file contains one public key per line. Lines beginning with # are comments. Each line consists of a user name and a base64-encoded key separated by white space. The public key is encoded in DER format. If the key is an Elliptic Curve key, then it is prefaced with EC:. Other keys are assumed to be DSA keys. The same key encoding is used for public keys stored in the ACL database.

System key files

The generate_dsa_keys.sh script generates private key login files for system and service accounts (e.g., priv-iris.base64.txt, priv-merge.base64.txt, priv-tdcp.base64.txt). It also generates an RSA key pair for the controller (priv-controllerConsole.base64.txt), which is used for encrypting controller secrets — not for authentication.

Important

The script generates private keys only. You must separately register the corresponding public keys in the ACL store for authentication to work.

Warning

The dsakeys.txt file-based public key store is deprecated and disabled by default. Manage public keys in the ACL store instead. See Deprecated filesystem-based public keystore for details.

The default authorized keys file location:

/etc/sysconfig/deephaven/auth/dsakeys.txt

This file can contain ECDSA or DSA public keys depending on when they were generated.

Properties file sections (deprecated)

[service.name=authentication_server] {
    authentication.server.authorizedkeys.enabled=true
    authentication.server.authorizedkeys.file=dsakeys.txt
}

User private keyfile

Users may authenticate using private key files. These should be only on the user's client machine, and provide access to Deephaven just as the user's password. The private key file is specified as follows.

The file is structured as a dictionary. Lines beginning with # are comments. Each line is a key and value separated by white space. There are four fields defined:

Field	Value
`user`	Username for authentication.
`operateas`	The user to operate as.
`public`	The public key in base64-encoded DER format. Elliptic Curve keys are prefaced with `EC:` before base64 encoding.
`private`	The private key in base64-encoded DER format. Elliptic Curve keys are prefaced with `EC:` before base64 encoding.

Private key files can be generated by using:

/usr/illumon/latest/bin/generate-iris-keys from the command line.
io.deephaven.enterprise.auth.KeyPairFile#generateNewKeys from Java/Groovy.
deephaven_enterprise.client.auth.AuthClient.generate_keypair from Python.

Property

AuthenticationClientManager.defaultPrivateKeyFile

The public key in the named file must be entered in the server’s Authorized Keys file.

Note

See Database authentication.

Deephaven private keyfile

The controller uses a key to authenticate to the Deephaven remote query dispatchers (query servers and merge servers). The controller must operate as a privileged user because it performs privileged operations such as starting Persistent Queries owned by any user. This key file should be deployed only to the controller, and not readable by other users.

Property

iris.authentication.keyfile=/etc/sysconfig/illumon.d/auth/priv-iris.base64.txt

Controller key pair

The Deephaven controller may store secrets such as JDBC passwords in its configuration. Consoles must be able to update these secrets, and to store them persistently in the Persistent Query Controller's cache files. This file should be only on the controller node. When a client needs to send a secret to the controller, it requests the public key. The controller decrypts secret data (e.g., passwords) using its private key before sending the secret to workers that require it.

[service.name=iris_controller] {
    PersistentQueryController.keyPairFile=/etc/sysconfig/illumon.d/auth/priv-controllerConsole.base64.txt
}

Other key files

A default installation creates system private key login files used by server-side processes for administrative operations. These files are created in /etc/sysconfig/deephaven/auth. Each process configures its key file path via a different property:

priv-tdcp.base64.txt — Used by the Table Data Cache Proxy to read cacheable data. Configured via AuthenticationClientManager.defaultPrivateKeyFile.
priv-merge.base64.txt — Used for authentication by batch merge queries run by server-side processes.
priv-iris.base64.txt — Allows the superuser iris account to perform administrative server-side operations. The controller configures this via iris.authentication.keyfile.

Database authentication

To connect to a database server to run queries, you must authenticate your connection. For interactive utilities, like the Deephaven console, you do so through a username/password dialog box or SAML. For automated batch queries, you must use a private key file. Best practices dictate that system batch jobs are run as a role user.

If you are writing your own batch queries (or interactive utilities), you need to provide a method for authenticating your client before any database connection is made. The database components all use the AuthenticationClientManager.getDefault() instance, so authenticating should be performed using one of the following methods:

Setting the AuthenticationClientManager.defaultPrivateKeyFile property, which must specify a private key file that is known to the authentication server. Authentication will be done automatically using the specified private key. For a batch query, this is the preferred choice.
Calling AuthenticationClientManager.getDefault().challengeResponse(String privateKeyFile), with a private key file that is known to the authentication server.
Implementing a method to get the password from the user. You should not have users store passwords in a file. If the password is typed in, you should make it obscured (for example, by using java.io.Console.readPassword).

How to create Deephaven Authentication Keys

Deephaven can use challenge-response authentication with ECDSA or DSA keys to authenticate users. This is useful for batch or other non-interactive programs that cannot prompt for passwords. These key pairs consist of two parts: (1) a private key that must be protected as if it were a password, and (2) a public key stored in the Deephaven ACL database.

You can generate a public/private key pair for yourself using the Core+ Python client and upload it to the ACL write server. This has the advantage of the private key never leaving the user's machine. Alternatively, a member of iris-acleditors can upload a key for any user.

from deephaven_enterprise.client.session_manager import SessionManager
import deephaven_enterprise.client.auth
import os

# generate a public/private key pair for "user_name"
(priv, pub) = deephaven_enterprise.client.auth.AuthClient.generate_keypair("user_name")

# Write the private key to a file, take care to ensure the permissions are safe.  This key is equivalent to a password.
with open("/tmp/priv.txt", "w") as f:
    f.write(priv)

print("Generated Key")

# Connect to the Deephaven cluster
sm = SessionManager("https://deephaven-cluster.example.com:8123/iris/connection.json")
# You can authenticate using your password, saml, or an existing private key.
sm.password("user_name", "password")
# Upload the public key to the ACL Write Server
sm.upload_key(pub)
sm.close()


# Now reconnect with your private key
sm2 = SessionManager("https://deephaven-cluster.example.com:8123/iris/connection.json")
sm2.private_key("/tmp/priv.txt")
sm2.close()

Alternatively, you can generate a public/private key pair on a Deephaven server as follows:

On a Deephaven server, run: /usr/illumon/latest/bin/generate-iris-keys <deephaven user name>.
This will generate pub-<deephaven user name>.base64.txt and priv-<deephaven user name>.base64.txt in the current directory.
Store the generated public key in the ACL database using the command:
```
/usr/illumon/latest/bin/dhconfig acls publickey import --file <path to key>/pub-<deephaven user name>.base64.txt
```
If you are not running as an administrator, then you should also include the --user parameter to specify which Deephaven account to authenticate as. You can either specify an administrator account or your own account, and dhconfig prompts for your password:
```
/usr/illumon/latest/bin/dhconfig acls publickey import --file <public key file> --user <user>
```
Move priv-<deephaven user name>.base64.txt to the client system connecting to the server. This file is effectively a password. It should be stored only where it will be used and must be protected by appropriate filesystem permissions.

Deephaven public and private key configuration

TLS Keystores for Services

Authentication server

Properties file sections

Authorized keys file configuration

Configuration server

Properties file sections

Web API Service keystore

Properties file sections

TLS Truststore

Truststore filenames

Passphrase file

Properties

Updating certificates

Authorized keys

System key files

Properties file sections (deprecated)

User private keyfile

Property

Deephaven private keyfile

Property

Controller key pair

Other key files

Database authentication

How to create Deephaven Authentication Keys

Related documentation