Hardening Technical Controls

This hardening guide provides recommendations from Deephaven, including product and server settings for a more secure and auditable system. For a detailed explanation of the security concepts and system accounts mentioned here, please refer to the Security and Auditing Overview. Many of these settings are installation defaults but should still be audited/verified to ensure system configuration remains secure. These recommendations and associated technical controls can also be used as a configuration compliance standard for demonstrating appropriate configuration in sensitive environments. To learn how to actively monitor the system, refer to the Audit Technical Controls guide.

Access

Type: Open ports

Details

Enable Envoy reverse proxy so that access to Deephaven services can be limited to a single port.

global.websocket.server.enabled=true

For steps on how to verify open ports, see Audit Technical Controls - Checking open ports and Envoy proxy.

Tools

Property inspector

Type: Open ports

Details

Disable admin page of Envoy reverse proxy.

admin: address: socket_address: not set in envoy config yaml

For steps on how to verify the Envoy admin port status, see Audit Technical Controls - Checking open ports and Envoy proxy.

Type: Certificates

Details

Ensure only specific names (no wildcards) are used in Deephaven Web certificates.

Check SAN block of /etc/deephaven/cus-tls/tls.crt.

For steps on how to inspect TLS certificate details, see Audit Technical Controls - Inspecting TLS certificate details.

Tools

openssl

Type: Disable built-in login

Details

After initial sysadmin logins have been added, remove the password for the built-in iris account.

Ensure the iris user's password entry is removed or invalidated in the system's local user password file (e.g., dsusers.txt, typically found in a configuration directory like /etc/deephaven/config/ or as defined by your authentication setup), particularly if local password authentication (authentication.server.localusers.enabled) remains active for other purposes. The primary method for disabling built-in logins is to rely on enterprise authentication (SAML/LDAP).

Audit

Type: Data retention

Details

Verify sufficient space exists for intraday logs under /var/log/deephaven and /Db/Intraday/DbInternal.

Depends on maximum expected activity and resultant logging. Monitor disk usage and establish a baseline. Consider log rotation and archiving strategies based on organizational retention policies.

Tools

File system management tools.

Type: Auditing enabled

Details

Verify various processes' configuration enables audit logging (default).

PersistentQueryController.writeDatabaseAuditLogs=true
AuthenticationServer.writeDatabaseAuditLogs=true
ConfigurationServer.writeDatabaseAuditLogs=true
DbAclWriteServer.writeDatabaseAuditLogs=true
DataImportServer.writeDatabaseAuditLogs=true
TableDataCacheProxy.writeDatabaseAuditLogs=true
RemoteQueryProcessor.writeDatabaseAuditLogs=true
RemoteQueryDispatcher.writeDatabaseAuditLogs=true

For steps on how to verify these audit log properties, see Audit Technical Controls - Verifying audit log properties.

Tool

Property inspector

Authentication

Type: Disable local password authentication

Details

Verify Deephaven password management is disabled.

authentication.server.localusers.enabled

Type: Enable SSO/LDAP authentication

Details

Verify SAML/OAUTH2 authentication is enabled and/or LDAP authentication is enabled.

authentication.server.customauth.class.SAMLAuth = io.deephaven.samlauth.SAMLAuthModule and/or authentication.server.ldap.enabled=true

Tool

Property inspector

Secure configuration management

Deephaven configuration is primarily in key/value property sets. These property sets are stored as BLOBs in etcd. The configuration server process provides a front end to the etcd data and provides configuration data to other Deephaven processes. (See the Deephaven configuration overview for more detailed information.)

Internally, the default organization of these files is:

  • iris-defaults.prop
  • iris-environment.prop
  • iris-endpoints.prop
  • iris-common.prop

The iris-common.prop file is the default properties file that Deephaven processes will load. iris-common.prop includes the other files so that defaults from iris-defaults.prop may be overridden or augmented by customer settings in iris-environment.prop, which may, in turn, be overridden or augmented by installation-generated settings in iris-endpoints.prop.

In addition to the "last in wins" organization of these files, they also use sub-configuration stanzas that look like this:

[service.name=dhconfig|dhctl] {
    # fail fast if the authentication server isn't available
    AuthenticationClientManager.maxWaitClient=1000
}

These can be included only for specific processes and/or for specific servers (by server name, IP address, etc). The example above only applies to the dhctl and dhconfig processes.

Because of the mix of applicability by stanza, checking that configuration settings are correct can be a bit complicated. Deephaven includes the property_inspector tool to allow command-line querying for property values that exist and apply across all the configuration files in etcd.

/usr/illumon/latest/bin/iris_exec property_inspector -- -pn AuthenticationClientManager.maxWaitClient -pf iris-common.prop

The above returns no match, because there is no global default for AuthenticationClientManager.maxWaitClient in any of the files included by iris-common.prop.

Adding the service.name of dhctl, though, finds the setting that was set in the above example stanza:

/usr/illumon/latest/bin/iris_exec property_inspector -j -Dservice.name=dhctl -- -pn AuthenticationClientManager.maxWaitClient -pf iris-common.prop

Name: AuthenticationClientManager.maxWaitClient Value: 1000 --- Line: 1036 --- File: iris-defaults.prop --- Scope: [service.name=dhconfig|dhctl]

In many cases, the concern for verifying configuration settings will be what settings are in effect within Deephaven worker processes. You can check this by invoking the configuration class within a Deephaven query. For example, this sample of Groovy code checks whether auditing is enabled for the current query worker process:

import com.fishlib.configuration.Configuration

final Configuration configuration = Configuration.getInstance()
println configuration.getBooleanWithDefault("RemoteQueryProcessor.writeDatabaseAuditLogs", true)

TLS/SSL certificates and keys

Deephaven uses openssl, provided by the underlying operating system of the server(s), to handle SSL/TLS libraries and ciphers.

During installation, Deephaven generates a set of self-signed certificates to use for service-to-service and worker-to-service secure connections. In Kubernetes environments that include a Cert Manager, the Cert Manager will be used to create these certificates. The Deephaven generated ones currently use:

  • Signature algorithm name: SHA256withRSA
  • Subject Public Key Algorithm: 2048-bit RSA key
  • Version: 3

External communication, such as connections from browsers to the Web UI, or connections from the Swing client launcher to the Client Update Service, are secured using a certificate and key provided by the user running the installation. The certificate and key are provided in files named tls.crt and tls.key. This certificate can also be refreshed/replaced using the iris-keygen.sh tool. This "Web API" certificate can specify any algorithm and key length that is supported in common by the version of openssl installed on the server, client components used by the client browsers, and the versions of Java installed on the server and Java-based clients.

When the Envoy reverse proxy is used, it will act as the SSL endpoint for external connections and will, by default, use the same Web API certificate and key.

Truststores and keystores

Deephaven uses PKCS12 format truststores and keystores to store internally generated and customer-provided certificates.

A keystore (/etc/sysconfig/deephaven/auth-user/webServices-keystore.p12) is used to store the Web API certificate and key (including any root or intermediate certificates that were part of a chain provided in `tls.crt).

A truststore (/etc/sysconfig/deephaven/trust/truststore-iris.p12) is used to store the internal certificates of all the Deephaven services that use TLS, as well as the Web API certificate chain. Deephaven client applications can use this truststore when negotiating connections, which simplifies TLS configuration.

The truststore passphrase is stored in the truststore_passphrase file in the same path as the truststore. It is base64-encoded. The files (truststore and its passphrase) are world-readable so that any Deephaven user or application can validate the truststore when using it to verify the authenticity of connections to Deephaven services. The truststore itself is protected from tampering by file permissions on the file (only writable by the irisadmin user). These files are among those that are copied to client systems using the Deephaven launcher or configuration updater tools, so the client applications can also validate secure connections to Deephaven services.

The Web API keystore passphrase file, .webapi_passphrase, is in the same path as the Web API keystore. The leading . makes it a hidden file. It is also base64-encoded. Deephaven workers each have their own Jetty server to provide a WebSockets connection for the user. They use the Web API certificate to provide TLS for this WebSocket endpoint.

Credential files

etcd clients

etcd is a distributed key/value database. It provides high availability and data redundancy when installed with multiple nodes (three or more, in an odd-numbered total). Deephaven uses etcd to store much of its configuration data, including:

  • Table schemata
  • User accounts, groups, and permissions (when enabled)
  • Persistent query configurations
  • System property settings
  • Data routing configuration

When used with Deephaven, etcd is configured to use multiple user accounts with different levels of access to Deephaven key/value data in the etcd database. These accounts are:

  • acl-ro
  • acl-rw
  • authenticationserver
  • controller
  • datarouting-ro
  • datarouting-rw
  • dispatcher
  • dispatcher-ro
  • props-ro
  • props-rw
  • root
  • schema
  • schema-ro
  • serviceregistry-ro
  • serviceregistry-rw

Note

Some of these accounts were added in more recent Deephaven releases and may not be present on releases v.20221001 and older.

These accounts have their connection information stored in directories under /etc/sysconfig/deephaven/etcd/client. Not all of these accounts' directories will be present on each node of a Deephaven cluster; which directories are present depends on which Deephaven services are running on the node. Only configuration server nodes will have the root directory.

Within each directory will be files that provide the IP addresses of etcd endpoint servers, the certificate with which to verify an SSL connection to the etcd endpoints, and user and password information with which to authenticate. The passwords are base64-encoded. Linux file permissions are used to restrict access to the password files to specific Deephaven accounts, and to restrict write access to all files to the irisadmin user.

When a Deephaven service (usually the configuration server) needs to access etcd for some information, it will use the corresponding client directory to get the endpoint and credential information needed to connect. For example, if the configuration server is writing a new table schema into etcd, it will use the details under the schema directory to connect to etcd, but, when reading schema information, it will use the schema-ro directory details.

Configuration server clients

Most Deephaven processes get their configuration, schema, and routing information from the configuration server. There must be at least one configuration server process in a Deephaven cluster, but there can be multiple instances to provide fault tolerance. Configuration files (host, port, certificate) for each configuration server are stored in directories under /etc/sysconfig/illumon.d/dh-config/clients. Access to the configuration server can be proxied through Envoy.

Client processes connecting to the configuration server will find the configuration directory using the dh.config.client.bootstrap property. Deephaven services will look for these files in the default location.

Connection properties can be overridden with dh.config.client.host, dh.config.client.port, and dh.config.client.cacert.

Deephaven logins

Deephaven allows key-based authentication using key pairs created with the Deephaven generate-iris-keys utility. There are several "out of the box" login key pairs pre-created to allow easy automation with command line scripts. These files are stored in /etc/sysconfig/deephaven/auth:

  • priv-authreconnect.base64.txt
  • priv-controllerConsole.base64.txt
  • priv-iris.base64.txt
  • priv-merge.base64.txt
  • priv-tdcp.base64.txt

Additional key sets for users or automation can be created with the generate-iris-keys utility; the public key is then appended to the dsakeys.txt file (also in /etc/sysconfig/deephaven/auth), along with the Deephaven account to which the key should map, and the authentication server process must then be restarted or reloaded to refresh its internal cache of key mappings.

Replication of security files

For Kubernetes installations, security files, such as private keys, truststores, keystores, and etcd credentials, are distributed across several read-only volumes, which are attached to pods that need the related subset of files for their operation.

For non-Kubernetes installations, the Deephaven config_packager.sh utility is used to create a tar package of needed credentials for specific server roles (for example, query servers), and then to unpackage and apply the file sets contained in the tar at target machines. During the initial installation of a cluster, the installer does this automatically. When manually reconfiguring nodes or replacing certificates, this tool will be used, along with scp or similar, to distribute needed files to different nodes in the cluster.

Permissions

Type: File system permissions

Details

Verify that the dbmerge user and dbmergegrp group do not have write permissions to any data paths other than /db/Users.

Only dbmerge user has write access to data paths.

For guidance on auditing these permissions, see Audit Technical Controls - Verifying file system permissions.

Type: File system permissions

Details

Verify correct ownership and permissions for install-created Deephaven login private keys (stored under /etc/sysconfig/deephaven/auth).

-r--------. irisadmin irisadmin priv-authreconnect.base64.txt
-r--------. irisadmin irisadmin priv-controllerConsole.base64.txt
-r--------. irisadmin irisadmin priv-iris.base64.txt
-r--------. dbmerge dbmergegrp priv-merge.base64.txt
-r--------. irisadmin irisadmin priv-tdcp.base64.txt

For steps on how to verify these key file permissions, see Audit Technical Controls - Verifying file system permissions.

Type: File system permissions

Details

Verify correct ownership and permissions for /etc/sysconfig/deephaven/auth/dsakeys.txt file.

/etc/sysconfig/deephaven/auth/dsakeys.txt file should be owned by irisadmin:irisadmin with 400 permissions.

For steps on how to verify dsakeys.txt permissions, see Audit Technical Controls - Verifying file system permissions.

Type: File system permissions

Details

Verify correct ownership and permissions for /etc/sysconfig/deephaven/trust files.

-rw-r--r--. irisadmin irisadmin truststore-iris.p12
-rw-r--r--. irisadmin irisadmin truststore-iris.pem
-rw-r--r--. irisadmin irisadmin truststore_passphrase

For steps on how to verify truststore file permissions, see Audit Technical Controls - Verifying file system permissions.

Policy

Type: File system permissions

Details

Allow per-user workers.

RemoteQueryDispatcher.startWorkersAsSystemUser=true

Tool

Property inspector

Enabling per-user workers

Services

Type: Minimized Attack Service

Details

Enable etcd storage of ACL informatiion, so that MariaDB/MySQL are not needed and can be removed/omitted.

IrisDB.groupProvider=etcd
IrisDB.permissionFilterProvider=etcd

Tool

Property inspector

Updates

Type: Java

Details

Verify server Java version used by Deephaven is up to date with organization requirements. The specific Java version for Deephaven can be set in illumon.iris.hostconfig with:

export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-11.0.19.0.7-1.el7_9.x86_64"
export ILLUMON_JAVA=$JAVA_HOME/bin/java

Newer Java versions will have newer crypto libraries and newer trusted root certification authorities.

Tools

java -version
javac -version