TLS certificates and trust model

Deephaven uses TLS to secure connections between clients and Deephaven services, between Deephaven services themselves, and between Deephaven services and the etcd configuration store.

This page explains how Deephaven identifies machines, verifies TLS connections, and uses certificates across Kubernetes, Podman, and native installations.

When you need to inspect certificates, verify certificate chains, modify system truststores, or manually update certificates, use the certificate troubleshooting guide.

Certificate changes in Grizzly+

Grizzly+ simplified the certificate model and made the TLS trust easier to reason about.

In previous releases, the native installer mixed a broadly scoped, user-supplied certificate with generated certificates and updated, rather than replaced, a single "trust everything" truststore.

Grizzly+ moves to a cleaner model: the user supplies a different certificate per machine, provides an optional x.509 PEM bundle of trusted certificates, and the native installer will overwrite all remote keystores and truststores with the certificates you provided.

One certificate per machine

Native installations now use the same per-machine certificate model as Podman: one certificate chain and private key for each machine. Each machine presents only its own identity, and each certificate only needs to cover that machine's fully qualified domain name.

When Envoy is enabled in native installations, you must provide the certificate clients will trust when they connect through Envoy. Deephaven generates the internal per-machine certificates used behind Envoy, unless you disable generated per-machine certificates.

Truststore is optional

Previous versions of Deephaven relied on a Deephaven-managed client truststore even when the client already trusted the certificate chain. In Grizzly+, client truststores are optional.

If clients already trust the certificate chain Deephaven presents, they can use system, browser, JVM, or Python trust. If they do not, Deephaven can generate a client truststore from the trust material you provide.

Certificates are always overwritten

Older workflows encouraged administrators to inspect certificates and truststores on disk, modify them in place, and gradually mix two sources of truth.

In Grizzly+, the certificates and trust material you provide are the source of truth. Deephaven regenerates and replaces the derived certificates, keystores, and truststores it manages on disk, every time you upgrade or [rotate certificates].

Fewer generated certificates

Earlier native releases generated certificates for some services while depending on shared customer-provided certificates for others. That made the trust model harder to expose safely and harder to reason about.

In Grizzly+, native installs without Envoy generate no certificates. Native installs with Envoy generate only the internal per-machine certificates and internal trust used behind Envoy.

TLS trust boundaries

Deephaven uses TLS in three trust relationships:

• External TLS trust is between clients and the certificate they connect to.
• Internal TLS trust is between Deephaven services.
• Etcd TLS trust is between Deephaven services and etcd.

External TLS trust

External TLS trust covers browsers, the Deephaven Launcher, Python clients, Java clients, and other outside callers.

When Envoy is enabled, clients trust the certificate Envoy presents. When Envoy is not enabled, clients trust the certificate presented directly by the Deephaven service they reach.

When Deephaven generates client trust, the familiar external truststore is tls.truststore=truststore-iris.p12. Otherwise, clients use system, browser, JVM, or Python trust. In practice, browsers on Windows usually use the Windows certificate store, browsers on macOS usually use Keychain, and Java applications may use either OS trust integration or their own cacerts truststore. Deephaven Launcher 9.00 and later also attempt to use Windows and macOS system truststores.

Internal TLS trust

Internal TLS trust covers how Deephaven services trust each other.

When internal.tls.truststore is set, Deephaven services use that truststore for server-to-server TLS instead of the external tls.truststore. This lets internal TLS trust a narrower certificate set than client-facing TLS.

When internal.tls.truststore is not set, internal TLS falls back to the same truststore and trust source used externally.

For compatibility, the internal and external truststores use the same passphrase file, but they can trust different certificate sets.

This internal split used to be most visible in Kubernetes. In Grizzly+, native installs now formalize the same idea. Native installs with Envoy use this mechanism automatically by generating internal-truststore.p12 and configuring internal.tls.truststore through generated per-machine certificates.

Etcd TLS trust

Etcd uses TLS for communication between Deephaven services and etcd. Those certificates are generated by Deephaven and are separate from the certificates clients trust.

If etcd should be reached on a different network interface than the machine's main Deephaven identity, use DH_NODE_N_ETCD_IP. When you change etcd network or TLS settings, make sure your etcd data is backed up. See [link to etcd add node].

Request routing with Envoy

When Envoy fronts Deephaven, clients trust the certificate Envoy presents and send traffic through Envoy to the services behind it. Deephaven services still trust each other using the deployment's internal trust model.

With Envoy, external trust and internal trust are intentionally different. Clients trust the Envoy certificate. Deephaven services trust the internal certificates and internal trust material used behind Envoy.

Request routing without Envoy

When Envoy is not used, clients connect directly to Deephaven services. In this model, external trust and internal trust, use the same certificate set and the same trust source, as both clients and servers connect to the same fully qualified domain names.

Podman can use either routing style. With Envoy, clients can use a single public path into the deployment. Without Envoy, clients may need direct routes to the pods they connect to.

Deephaven machine identity

Every Deephaven machine has a machine identity: the fully qualified domain name other Deephaven services use when they connect to that machine.

In native installations, the installer uses that machine name when it writes service addresses, validates certificates, and generates internal certificates. A provided per-machine certificate must include that machine name in its SAN entries.

All machines in a cluster should resolve a Deephaven machine name the same way. If different machines resolve the same name to different addresses, service-to-service TLS and token validation can fail.

Etcd can be an exception when it should be reached on a different network interface. In that case, use DH_NODE_N_ETCD_IP to tell Deephaven which address to use for etcd traffic.

SAN and hostname verification

Deephaven performs hostname verification against the certificate's Subject Alternative Name entries. Neither certificate Common Names nor authority overrides are used for Deephaven hostname verification.

Every certificate Deephaven presents must include SAN entries for the name clients or Deephaven services use to reach that server.

In native installations, each provided machine certificate must include the SAN for that machine's configured FQDN. In Podman, the certificate from the TLS volume must include the SAN for the --hostname used to start the pod. In Kubernetes, the frontend certificate specified by the deephaven-tls secret must include the SAN for the externally exposed envoyFrontProxyUrl.

Wildcard certificates are allowed, but they weaken isolation and should be used sparingly.

What a Deephaven server certificate must contain

For Deephaven client-facing TLS, the leaf certificate must be usable as a TLS web-server certificate. In practice, that means:

• the certificate chain is PEM-encoded and includes the leaf certificate followed by any required intermediate certificates;
• the certificate's SAN entries cover the hostname or hostnames clients actually use;
• the certificate's Key Usage permits Digital Signature; and
• the certificate's Extended Key Usage permits TLS Web Server Authentication.

Deephaven does not use the certificate Common Name for hostname verification. When a certificate arrives in DER or .cer form, convert it to PEM before use. The certificate troubleshooting guide shows how to inspect and convert certificate files.

Required TLS certificates

• Native installation: certificate inputs are supplied on the Installation Host via cluster.cnf; you must supply a signing certificate chain for the Envoy load balancer, or signing certificates for every machine in the Deephaven cluster. See Basic installation: TLS certificates for details on the specific cluster.cnf properties and required input files.

• Podman: you provide the certificate chain and private key as PEM files via the TLS volume (passed with --tls-volume), which is mounted into the container so Deephaven can read the expected cert/key paths at runtime. See the Podman installation guide (TLS volume and certificate file placement).

• Kubernetes: you provide the certificate chain and private key for the envoyFrontProxyUrl in a Kubernetes Secret named deephaven-tls, with tls.crt (chain) and tls.key (private key) data fields, which are mounted into the Envoy/cluster pods as files via a volume mount. See the Kubernetes installation guide (TLS Secret configuration and optional cert-manager).

Generated outputs: Deephaven writes derived keystores and truststores to canonical locations under /etc/sysconfig/deephaven/*.

Optional truststores

External trust in native installs

Native installs can either rely on system trust or use Deephaven-managed client trust.

When you provide an explicit CA bundle through DH_CA_CERTIFICATE, Deephaven generates the external client truststore, including truststore-iris.p12. Without that CA bundle, clients rely on system, browser, JVM, or Python trust.

Internal trust in native installs

Native internal trust depends on how certificates are supplied.

With Envoy and generated internal certificates, Deephaven uses a strict internal truststore for server-to-server communication. With provided per-machine certificates, internal trust can come from the same provided CA material as external trust or from system trust.

External trust in Podman

Podman can rely on system trust or take explicit trust material through --ca-cert-file when building images or starting the cluster.

Without --ca-cert-file, Podman uses system trust. With it, Podman uses the CA material you provide.

Internal trust in Podman

Podman internal trust follows the routing model you deploy.

When services connect directly, internal trust can match external trust. When Envoy fronts the deployment, Podman can separate the external Envoy certificate from the certificates and trust used internally.

External trust in Kubernetes

Kubernetes uses the frontend certificate provided through the deephaven-tls Secret for client-facing TLS.

Internal trust in Kubernetes

Kubernetes already separates internal and external trust. Internal certificates are generated or managed separately from the frontend certificate.

A Deephaven-managed optional client truststore like the native model is not yet available here.

Certificate rotation

Certificate rotation always starts with the certificates or trust material you provide. Deephaven then replaces the derived files it manages on disk.

Related items

• Admin glossary: Defines the TLS and trust terms used throughout this page.
• Cluster configuration guide: Documents native installer certificate and machine-identity settings.
• Basic installation: Explains native installation flow and certificate inputs.
• Podman installation: Explains Podman TLS volumes, optional envoy.pem, and --ca-cert-file.
• Use Envoy as a front proxy: Explains how Envoy fronts Deephaven services.
• Troubleshooting certificates: Covers certificate inspection, truststore updates, and repair steps.
• Security hardening: Explains how to isolate administrative services and protect machine certificates from user-code hosts.