TLS certificates and trust model

Deephaven uses TLS to secure connections between clients and Deephaven services, between Deephaven services themselves, and between Deephaven services and the etcd configuration store.

This page explains how Deephaven identifies machines, verifies TLS connections, and uses certificates across Kubernetes, Podman, and native installations.

When you need to inspect certificates, verify certificate chains, modify system truststores, or manually update certificates, use the certificate troubleshooting guide.

Certificate changes in Grizzly+

Grizzly+ simplified the certificate model and made the TLS trust easier to reason about.

In previous releases, the native installer mixed a broadly scoped, user-supplied certificate with generated certificates and updated, rather than replaced, a single "trust everything" truststore.

Grizzly+ moves to a cleaner model: the user supplies a different certificate per machine, provides an optional X.509 PEM bundle of trusted certificates, and the native installer will overwrite all remote keystores and truststores with the certificates you provided.

One certificate per machine

Native installations now use the same per-machine certificate model as Podman: one certificate chain and private key for each machine. Each machine presents only its own identity, and each certificate only needs to cover hostname verification for that machine's fully qualified domain name.

Tip

Kubernetes Deephaven installations already handle provisioning and trusting certificates for all internal pods and services. Podman Deephaven installations have always required one certificate per pod created.

When Envoy is enabled in native installations, you must provide the certificate that clients will trust when they connect through the Envoy frontend URL. Deephaven generates the internal per-machine certificates used behind Envoy, unless you disable generated per-machine certificates.

Warning

This model is clearer and safer, but it also makes private-key placement more important. Machines that run user code should not have access to certificates that let them impersonate administrative services. The security hardening guide explains how to isolate those roles.

Optional truststore

Previous versions of Deephaven relied on a Deephaven-managed truststore file even when the client already trusted the certificate chain. In Grizzly+, client truststores are optional, except for Kubernetes installations.

The decision of whether to leave the matter of trusting TLS certificates up to a system CA bundle or to provide your own CA bundle depends largely on who is expected or allowed to modify trust materials on client and server machines.

In some cases, the system administrator with permission to modify system truststores is a different developer than the Deephaven cluster administrator. If the system administrator wants to update trust materials for many different products, using system CA trust materials allows the system administrator to add or remove trusted system certificates without updating any files on the Deephaven server or client machines. The server and client processes will need to be restarted unless you are revoking certificates and have explicitly enabled certificate revocation checking in your JVM processes.

If the Deephaven cluster administrators need to tightly control which certificates are trusted, or configure TLS trust without any root permission, they can supply an explicit PEM bundle of trusted certificates. By opting in to an explicit truststore, clients and servers are forced to verify TLS certificates against the customer-provided CA trust materials.

Using an explicit truststore allows the Deephaven cluster administrator to update the trusted certificates without needing root permission, but this model requires restarting all server and client processes to update the truststore. In this case, an explicit truststore file will be generated in /etc/sysconfig/deephaven/trust/truststore-iris.p12 and the tls.truststore Deephaven property will be configured to instruct TLS clients to only trust certificates signed by your supplied root CA certificates.

Platform-specific advice on configuring the use of a truststore is covered in the TLS trust models section below.

Certificates are always overwritten

Previous certificate-handling practices encouraged administrators to inspect certificates and truststores on disk, modify them in place, and gradually merge two sources of truth.

In Grizzly+, the certificates and trust material you provide are the source of truth. Deephaven regenerates and replaces the derived certificates, keystores, and truststores it manages on disk every time you upgrade or rotate certificates. This is a breaking change for workflows that previously edited keystores or truststores in place on hosts.

Fewer generated certificates

Earlier native releases generated certificates for some services while depending on shared customer-provided certificates for others. That made the trust model harder to expose safely and harder to reason about.

In Grizzly+, native installs without Envoy do not generate any certificates. Native installs with Envoy generate only the internal per-machine certificates and internal trust used behind Envoy.

TLS trust boundaries

Deephaven uses TLS in three trust relationships:

External TLS trust is between clients and the certificate they connect to.
Internal TLS trust is between Deephaven services.
Etcd TLS trust is between Deephaven services and etcd.

External TLS trust

External TLS trust covers browsers, the Deephaven Launcher, Python clients, Java clients, and other outside callers.

When Envoy is enabled, clients trust the certificate Envoy presents. When Envoy is not enabled, clients trust the certificate presented directly by the Deephaven service they reach.

When Deephaven generates client trust, the familiar external truststore is tls.truststore=truststore-iris.p12. Otherwise, clients use system, browser, JVM, or Python trust. In practice, browsers on Windows usually use the Windows certificate store, browsers on macOS usually use Keychain, and Java applications may use either OS trust integration or their own cacerts truststore. Deephaven Launcher 9.00 and later also attempt to use Windows and macOS system truststores.

Internal TLS trust

Internal TLS trust governs how Deephaven services trust one another.

When internal.tls.truststore is set, Deephaven services use that truststore for server-to-server TLS instead of the external tls.truststore. This lets internal TLS trust a narrower set of certificates than client-facing TLS.

When internal.tls.truststore is not set, internal TLS falls back to the same truststore and trust source used externally.

For compatibility, the internal and external truststores use the same passphrase file, but they can trust different certificate sets.

This internal split used to be most visible in Kubernetes. In Grizzly+, native installs now formalize the same idea. Native installs with Envoy use this mechanism automatically by generating internal-truststore.p12 and configuring internal.tls.truststore through generated per-machine certificates.

Etcd TLS trust

Etcd uses TLS for communication between Deephaven services and etcd. Those certificates are generated by Deephaven and are separate from the certificates clients trust.

If etcd should be reached on a different network interface than the machine's main Deephaven identity, use DH_NODE_N_ETCD_IP. When you change the etcd network or TLS settings, make sure your etcd data is backed up. See [link to etcd add node].

Request routing with Envoy

When Envoy fronts Deephaven, clients trust the certificate Envoy presents and send traffic through Envoy to the services behind it. Deephaven services still trust each other using the deployment's internal trust model.

With Envoy, external trust and internal trust are intentionally different. Clients trust the Envoy certificate. Deephaven services trust the internal certificates and internal trust material used behind Envoy.

Request routing without Envoy

When Envoy is not used, clients connect directly to Deephaven services. In this model, external and internal trust use the same certificate set and trust source, since both clients and servers connect to the same fully qualified domain names.

Podman can use either routing style. With Envoy, clients can use a single public path into the deployment. Without Envoy, clients may need direct routes to the pods they connect to.

Deephaven machine identity

Every Deephaven machine has a machine identity: the fully qualified domain name (FQDN) that other parties use to reach that Deephaven endpoint.

Machine identity matters for TLS because Deephaven relies on standard TLS hostname verification: the hostname used to connect must match a DNS: entry in the certificate's SAN. See SAN and hostname verification.

In deployments that use Envoy, there are typically two relevant identities:

External (client-facing) identity: the hostname clients use (usually the Envoy hostname and port).
Internal (service-to-service) identities: the hostnames Deephaven services use to connect to each other (for example, per-machine hostnames in native/Podman, or Kubernetes service DNS names).

Where the identity comes from depends on your Deephaven installation type:

Native: per-machine identity is configured in cluster.cnf via DH_NODE_N_DOMAIN. Deephaven uses these names when writing service addresses, validating provided per-machine certificates, and (when enabled) generating internal per-machine certificates.
Podman: per-machine identity is determined by the --hostname (-h) value passed to start_command.sh. The per-machine certificate from the TLS volume must include that hostname in its SAN entries.
Kubernetes: the external client-facing identity is the envoyFrontProxyUrl (and the deephaven-tls secret certificate must cover that hostname). The internal identities are typically Deephaven-managed Kubernetes service hostnames used for service-to-service TLS.

All machines in a cluster must resolve Deephaven machine names consistently. If different machines resolve the same DNS hostname to different IP addresses, service-to-service TLS and authentication token validation will fail.

Connections to Etcd servers use IP addresses rather than machine hostnames. The native installer derives the default IP address for Etcd servers from the machine's FQDN machine identity. If your Etcd servers listen on a different IP address than your Deephaven Java services, you can manually set DH_NODE_N_ETCD_IP to configure all Etcd traffic to use your configured IP address.

Caution

When using the native Deephaven installer, changing the machine identity of a Deephaven Java service is straightforward. You provide updated certificates and update cluster.cnf to use a new hostname. Changing the machine identity (IP address) of an etcd server is a complex, manual process which requires redistributing TLS signing and trust materials to all Etcd and Deephaven servers, as well as running etcdctl.sh commands to modify the Etcd cluster membership.

SAN and hostname verification

Deephaven performs hostname verification against the certificate's Subject Alternative Name entries. Neither certificate Common Names nor authority overrides are used for Deephaven hostname verification.

Every TLS certificate a Deephaven service presents must include SAN entries for the hostnames that clients or Deephaven services use to reach that server.

In native installations, each machine in the cluster requires its own TLS signing certificate, and that certificate must include SAN DNS: entries matching that machine's configured FQDN.

In Podman, the tls.crt certificate in the --tls-volume passed to start_command.sh must include SAN DNS: entries for the --hostname used to start the dh-ifra pod.

In Kubernetes, the frontend certificate specified by the deephaven-tls secret must include SAN DNS: entries that match the client-accessible hostname defined by envoyFrontProxyUrl in values.yaml.

Wildcard certificates are allowed, but they enable impersonation and should be avoided.

TLS certificate requirements

All Deephaven services serving TLS requests to Deephaven clients or other Deephaven services must present a leaf certificate that is usable as a TLS web-server certificate.

the signing certificate chain used by the server must include the leaf certificate followed by any required intermediate CA certificates;
the leaf certificate's subject alternative name extension must include a DNS:host.name or DNS:*.host.name entry that satisfies hostname verification for the server's configured FQDN;
the leaf certificate's Key Usage permits Digital Signature; and
the leaf certificate's Extended Key Usage permits TLS Web Server Authentication.

Deephaven does not use the certificate Common Name or authority overrides for TLS connection verification. Only the SAN is considered.

When a certificate arrives in DER or .cer form, convert it to PEM before use.

Required TLS certificates

Installation method	Requires per-machine certs	Requires Envoy cert	Generates certificates
Kubernetes	Generated automatically	Always, provided via the `deephaven-tls` Secret	Per-machine via Deephaven-generated or cert-manager-managed certificates
Podman	Always, via `tls.crt` and `tls.key` on the host machine's `--tls-volume`	Optional, via `envoy.pem` in the TLS volume when Envoy should answer on a different hostname	Never, but ships with `create_dh_cert.sh` to generate your own
Native, Envoy enabled	Optional; set `DH_GENERATE_CERTIFICATES=false` to require provided per-machine certs	Always, via `DH_ENVOY_LOCAL_CERT`	Optional; set `DH_GENERATE_CERTIFICATES=true`
Native, no Envoy	Always, one certificate chain and private key per machine	Never	Never

Native installation: certificate inputs are supplied on the Installation Host via cluster.cnf; you must supply a signing certificate chain for the Envoy load balancer, or signing certificates for every machine in the Deephaven cluster. See Basic installation: TLS certificates for details on the specific cluster.cnf properties and required input files.
Podman: you provide the certificate chain and private key as PEM files via the TLS volume (passed with --tls-volume), which is mounted into the container so Deephaven can read the expected cert/key paths at runtime. See the Podman installation guide (TLS volume and certificate file placement).
Kubernetes: you provide the certificate chain and private key for the envoyFrontProxyUrl in a Kubernetes Secret named deephaven-tls, with tls.crt (chain) and tls.key (private key) data fields, which are mounted into the Envoy/cluster pods as files via a volume mount. See the Kubernetes installation guide (TLS Secret configuration and optional cert-manager).

All provided X.509 certificates must include a Subject Alternative Name extension capable of performing hostname verification against the machine identity FQDN used to make the TLS connection.

To put it simply, if you connect to auth.deephaven.example.com, that server must sign TLS requests using a certificate with a SAN entry for DNS:auth.deephaven.example.com (preferred) or DNS:*.deephaven.example.com (discouraged).

In addition to hostname verification, the signing certificate must also form a complete chain of TLS trust. This means that the certificates presented by the server must be issued by certificates present in the client's system truststore or explicitly configured truststore.

Deephaven installations transform provided X.509 PEM certificates into PKCS12 format ".p12 TLS keystore files" which are restricted to Deephaven service accounts in the /etc/sysconfig/deephaven/auth and /etc/sysconfig/deephaven/auth-user directories. Any explicit truststore files (in PKCS12 and PEM format) are world-readable and stored in /etc/sysconfig/deephaven/trust.

All generated keystore and truststore files are overwritten with your supplied certificates whenever you upgrade Deephaven.

Deephaven trust models

Native installation trust model

Native installations have the most configurable trust model.

External trust in native installations

For native installations, Deephaven clients can either rely on system trust or use an explicitly provided X.509 PEM format CA certificate bundle.

When you provide an explicit CA bundle through DH_CA_CERTIFICATE, Deephaven generates the external client truststore at /etc/sysconfig/deephaven/trust/truststore-iris.p12, and configures the tls.truststore Deephaven property to direct processes to use the explicit truststore file instead of the system/Java truststore.

When you do not set DH_CA_CERTIFICATE, no truststore is generated, and clients rely on system, browser, JVM, or Python trust.

When you set DH_CONFIGURE_ENVOY=true and DH_GENERATE_CERTIFICATES=true in cluster.cnf, Deephaven generates an Envoy frontend TLS signing certificate whose subject alternative name TLS extension must match the configured Envoy frontend url defined by DH_ENVOY_FQDN. The DH_CA_CERTIFICATE value is used as the CA bundle to validate this certificate, and should be a CA certificate which cannot have and does not need a SAN entry matching the frontend URL.

When you set DH_CONFIGURE_ENVOY=false or [DH_GENERATE_CERTIFICATES=false] in cluster.cnf, then any DH_CA_CERTIFICATE you specify must contain all CA certificates necessary to trust all TLS signing certificates supplied in DH_LOCAL_CERT_DIR.

Internal trust in native installations

Internal trust verification for TLS connections between Deephaven services depends on how certificates are supplied.

When you set DH_CONFIGURE_ENVOY=true and [DH_GENERATE_CERTIFICATES=true] in cluster.cnf, Deephaven generates a strict internal truststore to secure server-to-server communication at /etc/sysconfig/deephaven/trust/internal-truststore.p12. This internal truststore is specified by the internal.tls.truststore Deephaven property and is generated and updated automatically whenever you rotate certificates.

In all other cases, the native installation internal trust model behaves the same as the external trust model. Deephaven services will use DH_CA_CERTIFICATE if specified, or the system CA certificates if DH_CA_CERTIFICATE is not specified.

Podman installation trust model

TLS trust model for Podman installations depends on passing --ca-cert-file to start_command.sh.

If you pass --ca-cert-file, then the specified X.509 PEM-formatted CA certificate bundle is used as an explicit truststore file. If you do not pass --ca-cert-file, Podman instead relies on the default system/Java truststores inside the containers for TLS verification.

The system truststores for Podman can be updated by passing --ca-cert-file to build_images.sh, which means you can forego passing --ca-cert-file to start_command.sh and opt in to using system truststores.

External trust in Podman installations

Podman can rely on system trust or take explicit trust material through --ca-cert-file when building images or starting the cluster.

When --ca-cert-file is provided to start_command.sh, the Podman installation copies your provided CA certificate bundle into the image at /etc/sysconfig/deephaven/trust/truststore-iris.p12 and configures Deephaven property tls.truststore to instruct both clients and servers to use the CA material you provided.

When --ca-cert-file is not provided to start_command.sh, all client and server TLS connections will rely on their local system/Java truststores.

To alter the system truststores in your Podman images, you can pass --ca-cert-file to build_images.sh.

Internal trust in Podman installations

Separating Podman internal service-to-service trust from external client-to-service trust requires using Envoy in Podman, providing a separate signing certificate for Envoy, and then manually configuring the internal.tls.truststore Deephaven property.

You must place your internal PKCS12 .p12 truststore on the filesystem inside each Podman container that participates in internal service-to-service TLS, typically at /etc/sysconfig/deephaven/trust/truststore-internal.p12, and configure the internal.tls.truststore Deephaven property to point to that absolute path (for example, internal.tls.truststore=/etc/sysconfig/deephaven/trust/truststore-internal.p12) so that internal connections use the internal CA material instead of the external trust configuration.

Podman internal trust follows the routing model you deploy.

When services connect directly, internal trust can match external trust. When Envoy fronts the deployment, Podman can separate the external Envoy certificate from the certificates and trust used internally.

External trust in Kubernetes

Kubernetes uses the frontend certificate provided through the deephaven-tls Secret for client-facing TLS.

Internal trust in Kubernetes

Kubernetes already separates internal and external trust. Internal certificates are generated or managed separately from the frontend certificate.

A Deephaven-managed optional client truststore like native and Podman installations is not yet available for Kubernetes.

Certificate rotation

Certificate rotation always starts with the certificates or trust material you provide. Deephaven then replaces the derived files it manages on disk.

Rotate certificates for native installations

Update the per-machine certificate inputs on the Installation Host.
If you use Envoy, update the certificate supplied through DH_ENVOY_LOCAL_CERT.
If you supply a CA bundle through DH_CA_CERTIFICATE, update it so Deephaven can regenerate the external truststore.
Rerun the supported installer or certificate-rotation workflow.
Restart affected services if needed.

Rotate certificates for Podman installations

Replace the certificate files in the TLS volume, including envoy.pem if Envoy answers on a different hostname.
Restart the affected pod or the full deployment.

If trust is supplied via --ca-cert-file at image build time, rebuild the images when the trusted CA changes.

Rotate certificates for Kubernetes installations

Update the deephaven-tls Secret used for the frontend (Envoy) certificate.
Follow the supported restart or upgrade flow for the deployment.

Internal certificate rotation depends on how the cluster manages those certificates (for example, Deephaven-generated or cert-manager-managed certificates).

Rotate certificates manually

Manual and advanced rotation workflows are covered in the certificate troubleshooting guide.

Admin glossary: Defines the TLS and trust terms used throughout this page.
Cluster configuration guide: Documents native installer certificate and machine-identity settings.
Basic installation: Explains native installation flow and certificate inputs.
Podman installation: Explains Podman TLS volumes, optional envoy.pem, and --ca-cert-file.
Use Envoy as a front proxy: Explains how Envoy fronts Deephaven services.
Troubleshooting certificates: Covers certificate inspection, truststore updates, and repair steps.
Security hardening: Explains how to isolate administrative services and protect machine certificates from user-code hosts.