Deephaven public and private key configuration

This guide details the server-side configuration of public and private keys and certificates within a Deephaven deployment. It covers TLS keystores and truststores essential for securing communication between Deephaven services, as well as other specialized keys used by internal server components for their operations.

Proper configuration of these keys and certificates is crucial for a secure and functional Deephaven environment. If you encounter connection issues that might be related to SSL/TLS, refer to the Debug SSL/TLS connections section of the troubleshooting guide.

TLS Keystores for Services

Authentication server

The Authentication Server’s keystore contains the certificate and private key of a TLS enabled service. You must keep this file private, and not distribute it to clients. Each node running an Authentication Server should have its own keys, with a certificate that is signed (issued) by the CA in the TLS truststore file (see below).

The default self-signed key pair for the Authentication Server is generated and saved to a PKCS12 (.p12) keystore file when iris_keygen.sh is run. The keystore file is owned by user irisadmin with permission set to 400:

[-r-------- irisadmin irisadmin]  keystore.authserver.p12

The keystore file is protected by a unique randomly generated password stored in base64 encoded format in a read-only hidden file owned by user irisadmin with permission set to 400:

[-r-------- irisadmin irisadmin]  .auth_passphrase

Properties file sections

These file locations are specified by the following properties, which need to be accessible in the authentication_server and db_acl_write_server contexts.

[service.name=db_acl_write_server|authentication_server] {
    tls.keystore=/etc/sysconfig/illumon.d/auth/keystore.authserver.p12
    tls.passphrase.file=/db/TempFiles/irisadmin/.auth_passphrase
}

Authorized keys file configuration

For public key authentication (PSK), the Authentication Server needs to know the location of the file(s) containing users' public keys. This is configured using the authentication.psk.authorizedKeysFiles property. This property specifies a comma-separated list of paths to authorized keys files. Each file follows the standard SSH authorized_keys format.

Example property setting:

authentication.psk.authorizedKeysFiles=/etc/deephaven/authorized_keys,/data/additional_authorized_keys

This configuration is typically set in the authentication_server context.

Configuration server

The Configuration Server’s keystore contains the certificate and private key of a TLS enabled service. You must keep this file private, and not distribute it to clients. Each node running a Configuration Server should have its own keys, with a certificate that is signed (issued) by the CA in the TLS truststore file (see below).

The default self-signed key pair for the Configuration Server is generated and saved to the .p12 keystore file when iris_keygen.sh is run. The keystore file is owned by user irisadmin with permission set to 400:

[-r-------- irisadmin irisadmin] keystore.configuration_server.p12

The Configuration Server keystore file is also protected by a unique randomly generated password stored in base64 encoded format in a read-only hidden file owned by user irisadmin with permission set to 400:

[-r-------- irisadmin irisadmin] .configuration_server_passphrase

Properties file sections

These file locations are specified by the following properties, which need to be accessible in the configuration_server context:

[service.name=configuration_server] {
    configuration.server.ssl=true
    configuration.server.keyfile=/etc/sysconfig/illumon.d/auth/keystore.configuration_server.p12
    configuration.server.passphrase.file=/db/TempFiles/irisadmin/.configuration_server_passphrase
}

Web API Service keystore

The Web API Server's keystore contains the certificate and private key of a TLS enabled service. You must keep this file private, and not distribute it or store the file in a non-secure location such as a git repository. The Web API Server's keystore file should be unique per node, with a certificate that is signed (issued) by a trusted CA.

The default self-signed key pair for the Web API Server is generated when installing the iris-config.rpm and saved to .p12 keystore file:

[-r-------- irisadmin irisadmin] webServices-keystore.p12

The keystore file is protected by a unique randomly generated password stored in base64 encoded format in a read-only hidden file owned by user irisadmin with permission set to 400:

[-r-------- irisadmin irisadmin] .webapi_passphrase

Properties file sections

These file locations are specified by the following properties, which need to be accessible in the web_api_service context.

[service.name=web_api_service] {
    WebServer.tls.keystore=/etc/sysconfig/illumon.d/auth/webServices-keystore.p12
    WebServer.tls.passphrase.file=/db/TempFiles/irisadmin/.webapi_passphrase
}

TLS Truststore

A TLS truststore file contains the root certificates that issued any certificates that should be trusted. The Deephaven truststore should contain the root certificate or certificates for all the Authentication Server and Configuration Server certificates. This file is stored in two formats; a PKCS12 file verified by a password, and a PEM file which does not require a password.

All nodes and all clients need to have access to the truststore files. A single truststore should be used across all Deephaven nodes. If your organization has an existing PKI, then you can import your CA's certificate into the truststores for use with Deephaven.

The truststore does not contain secrets, and all clients need access to the password for the password-protected file.

Truststore filenames

• /etc/sysconfig/illumon.d/resources/truststore-iris.p12

• /etc/sysconfig/illumon.d/resources/truststore-iris.pem

Passphrase file

/etc/sysconfig/illumon.d/resources/truststore_passphrase

Properties

• configuration.server.cacertfile=/etc/sysconfig/illumon.d/resources/truststore-iris.pem
• tls.truststore=/etc/sysconfig/illumon.d/resources/truststore-iris.p12
• tls.truststore.passphrase.file=/etc/sysconfig/illumon.d/resources/truststore_passphrase

Using non-default certificates

Instead of using the uniquely generated files from the installation, customers can create their own keystore files with private keys and certificates signed by their own CA and update these properties accordingly.

Import your own root or intermediate CA certificate into the trust store:

    /usr/lib/jvm/java-openjdk/bin/keytool \
    -import -trustcacerts -alias iris \
    -file /path/to/tls.crt -keystore \
    /etc/sysconfig/illumon.d/resources/truststore-iris.p12 \
-storepass $PASS_PHRASE

Internal service and component keys

Authorized keys

A Deephaven-authorized keys file associates usernames with keys and allows non-interactive processes to authenticate with a key instead of a password. The process of authenticating must have the private key corresponding to the public key in this file. This file is disabled by default, with public keys maintained in the ACL store (etcd or MySQL).

As of the Grizzly (1.20240517), Deephaven uses the secp256r1 Elliptic Curve to generate key pairs, which is currently recommended by FIPS.186-5. Deephaven's authentication server generates a random nonce, which the client then hashes using SHA256 and signs with the private key. In versions of Deephaven from 1.20221001 to 1.20231218, Deephaven generated 2048-bit DSA keys as defined in FIPS.186-4. Prior to 1.20221001, 512-bit DSA keys were generated. For backward compatibility, Deephaven supports using ECDSA and DSA keys for authentication so that users can continue using their existing keys.

The authorized keys file contains one public key per line. Lines beginning with # are comments. Each line consists of a user name and a base64-encoded key separated by white space. The public key is encoded in DER format. If the key is an Elliptic Curve key, then it is prefaced with EC:. Other keys are assumed to be DSA keys. The same key encoding is used for public keys stored in the ACL database.

DSA Keys File

/etc/sysconfig/illumon.d/resources/dsakeys.txt

Properties file sections

[service.name=authentication_server] {
    authentication.server.authorizedkeys.enabled=true
    authentication.server.authorizedkeys.file=dsakeys.txt
}

User private keyfile

Users may authenticate using private key files. These should be only on the user's client machine, and provide access to Deephaven just as the user's password. The private key file is specified as follows.

The file is structured as a dictionary. Lines beginning with # are comments. Each line is a key and value separated by white space. There are four fields defined:

Private key files can be generated by using:

• /usr/illumon/latest/bin/generate-iris-keys from the command line.
• io.deephaven.enterprise.auth.KeyPairFile#generateNewKeys from Java/Groovy.
• deephaven_enterprise.client.auth.AuthClient.generate_keypair from Python.

Property

AuthenticationClientManager.defaultPrivateKeyFile

The public key in the named file must be entered in the server’s Authorized Keys file.

Deephaven private keyfile

The controller uses a key to authenticate to the Deephaven remote query dispatchers (query servers and merge servers). The controller must operate as a privileged user because it performs privileged operations such as starting Persistent Queries owned by any user. This key file should be deployed only to the controller, and not readable by other users.

Property

iris.authentication.keyfile=/etc/sysconfig/illumon.d/auth/priv-iris.base64.txt

Controller key pair

The Deephaven controller may store secrets such as JDBC passwords in its configuration. Consoles must be able to update these secrets, and to store them persistently in the Persistent Query Controller's cache files. This file should be only on the controller node. When a client needs to send a secret to the controller, it requests the public key. The controller decrypts secret data (e.g., passwords) using its private key before sending the secret to workers that require it.

Property

[service.name=iris_controller] {
    PersistentQueryController.keyPairFile=/etc/sysconfig/illumon.d/auth/priv-controllerConsole.base64.txt
}

Other key files

The following key files are also installed.

• priv-authreconnect.base64.txt - This is the keyfile used by the authentication server for its public/private encryption. The file is defined by the following property, which can be updated if needed:
  • authentication.server.reconnection.keyfile=/etc/sysconfig/illumon.d/auth/priv-authreconnect.base64.txt
• priv-merge.base64.txt - This keyfile may be used for authentication by batch merge queries.
• priv-superuser.base64.txt - This keyfile may be used by the superuser user for authentication.

Database authentication

To connect to a database server to run queries, you must authenticate your connection. For interactive utilities, like the Deephaven console, you do so through a username/password dialog box or SAML. For automated batch queries, you must use a private key file. Best practices dictate that system batch jobs are run as a role user. If you are writing your own batch queries (or interactive utilities), you need to provide a method for authenticating your client before any database connection is made. The database components all use the AuthenticationClientManager.getDefault() instance, so authenticating should be performed using one of the following methods:

1. Setting the AuthenticationClientManager.defaultPrivateKeyFile property, which must specify a private key file that is known to the authentication server. Authentication will be done automatically using the specified private key. For a batch query, this is the preferred choice.
2. Calling AuthenticationManager.getDefault().challengeResponse(String privateKeyFile), with a private key file that is known to the authentication server.
3. Implementing a method to get the password from the user. You should not have users store passwords in a file. If the password is typed in, you should make it obscured (for example, by using java.io.Console.readPassword).

How to create Deephaven Authentication Keys

Deephaven can use challenge-response authentication with DSA keys to authenticate users. This is useful for batch or other non-interactive programs that cannot prompt for passwords. DSA keys consist of two parts: (1) a private key that must be protected as if it were a password, and (2) a public key stored in the Deephaven ACL database.

You can generate a public/private key pair for yourself using the Core+ Python client and upload it to the ACL write server. This has the advantage of the private key never leaving the user's machine. Alternatively, a member of iris-acleditors can upload a key for any user.

from deephaven_enterprise.client.session_manager import SessionManager
import deephaven_enterprise.client.auth
import os

# generate a public/private key pair for "user_name"
(priv, pub) = deephaven_enterprise.client.auth.AuthClient.generate_keypair("user_name")

# Write the private key to a file, take care to ensure the permissions are safe.  This key is equivalent to a password.
with open("/tmp/priv.txt", "w") as f:
    f.write(priv)

print("Generated Key")

# Connect to the Deephaven cluster
sm = SessionManager("https://deephaven-cluster.example.com:8123/iris/connection.json")
# You can authenticate using your password, saml, or an existing private key.
sm.password("user_name", "password")
# Upload the public key to the ACL Write Server
sm.upload_key(pub)
sm.close()


# Now reconnect with your private key
sm2 = SessionManager("https://deephaven-cluster.example.com:8123/iris/connection.json")
sm2.private_key("/tmp/priv.txt")
sm2.close()

Alternatively, you can generate a public/private key pair on a Deephaven server as follows:

1. On a Deephaven server, run: /usr/illumon/latest/bin/generate-iris-keys <deephaven user name>. The default key length is 2048 bits. You can override that with a second parameter:

   /usr/illumon/latest/bin/generate-iris-keys <deephaven user name> <bits>
   

2. This will generate pub-<deephaven user name>.base64.txt and priv-<deephaven user name>.base64.txt in the current directory.

3. Store the generated public key in the ACL database using the command:

   /usr/illumon/latest/bin/dhconfig acl publickey import -f <path to key>/pub-<deephaven user name>.base64.txt
   

   If you are not running as an administrator, then you should also include the --user parameter to specify which Deephaven account to authenticate as. You can either specify an administrator account or your own account, and dhconfig prompts for your password:

   /usr/illumon/latest/bin/dhconfig acl publickey import -f <public key file> --user <user>
   

4. Move priv-<deephaven user name>.base64.txt to the client system connecting to the server. This file is effectively a password. It should be stored only where it will be used and must be protected by appropriate filesystem permissions.

How to connect to a DB Server

Beyond TLS configurations, several other keys are used by Deephaven server components for their internal operations or for inter-service authentication.

Deephaven private keyfile

The Deephaven controller utilizes a specific private key to authenticate itself to remote query dispatchers (like query servers and merge servers). This key enables the controller, operating as a privileged user, to perform critical operations such as managing Persistent Queries across different user contexts. It's vital that this key file is securely deployed only on the controller node and is not accessible to other users.

This key is often named priv-controller.base64.txt or similar and is specified via a server property like deephaven.controller.privateKeyFile. It must be deployed securely on the controller node and have restricted file permissions.

Controller key pair

The Deephaven controller uses a key pair to manage sensitive information, such as JDBC passwords, stored in its configuration. This mechanism allows consoles to securely update these secrets and for the controller to store them persistently. When a client needs to transmit a secret to the controller, it encrypts the data using the controller's public key. The controller then uses its private key to decrypt this information before distributing it to worker processes. This key pair file should be restricted to the controller node.

This key pair is typically a PEM-formatted file (e.g., controller-keypair.pem) configured via a property such as deephaven.controller.keyPairFile. Secure storage and restricted access on the controller node are paramount.

Other key files

Several other specialized key files are used within Deephaven for specific authentication or operational purposes by server components:

• priv-authreconnect.base64.txt: Used by the authentication server for its public/private encryption needs, particularly for session reconnection.
• priv-merge.base64.txt: Can be used for authentication by batch merge queries when these are run as part of server-side processes.
• priv-superuser.base64.txt: Allows the superuser account to authenticate using a key, typically for administrative server-side operations.

These files are typically located in a secured directory on the server, and their paths are configured through specific Java system properties or application configuration settings relevant to their respective services.

Related documentation

• Deephaven properties files
• ACLs
• Deephaven public and private key authentication