---
title: TLS certificates and trust model
sidebar_label: TLS certificates
---
Deephaven uses TLS to secure connections [between clients and Deephaven services](#external-tls-trust), [between Deephaven services themselves](#internal-tls-trust), and [between Deephaven services and the etcd](#etcd-tls-trust) configuration store.
This page explains how Deephaven [identifies machines](../admin-glossary.md#machine-identity), [verifies TLS connections](#san-and-hostname-verification), and uses certificates across [Kubernetes](#rotate-certificates-for-kubernetes-installations), [Podman](#rotate-certificates-for-podman-installations), and [native installations](#rotate-certificates-for-native-installations).
When you need to [inspect certificates](../troubleshooting/troubleshooting-certificates.md#common-certificate-issues), [verify certificate chains](../troubleshooting/troubleshooting-certificates.md#verify-certificate-chains), [modify system truststores](../troubleshooting/troubleshooting-certificates.md#manage-system-truststores), or [manually update certificates](../troubleshooting/troubleshooting-certificates.md#manual-certificate-rotation), use the [certificate troubleshooting guide](../troubleshooting/troubleshooting-certificates.md).
## Certificate changes in Grizzly+
Grizzly+ simplified the certificate model and made the TLS trust easier to reason about.
In previous releases, the native installer mixed a broadly scoped, user-supplied certificate with generated certificates and updated, rather than replaced, a single "trust everything" truststore.
Grizzly+ moves to a cleaner model: the user supplies a [different certificate per machine](#one-certificate-per-machine), provides an [optional X.509 PEM bundle of trusted certificates](#optional-truststore), and the native installer will [overwrite all remote keystores and truststores](#certificates-are-always-overwritten) with the certificates you provided.
### One certificate per machine
Native installations now use the same per-machine certificate model as Podman: one certificate chain and private key for each machine. Each machine presents only its own identity, and each certificate only needs to cover [hostname verification](../admin-glossary.md#hostname-verification) for that machine's [fully qualified domain name](../admin-glossary.md#fqdn).
> [!TIP]
> Kubernetes Deephaven installations already handle provisioning and trusting certificates for all internal pods and services. Podman Deephaven installations have always required one certificate per pod created.
When Envoy is enabled in native installations, you must [provide the certificate](../installation/cluster-config-guide.md#dh_envoy_local_cert) that clients will trust when they connect through [the Envoy frontend URL](../envoy/installing-envoy.md#envoy-frontend-url). Deephaven generates the internal per-machine certificates used behind Envoy, unless you disable [generated per-machine certificates](../installation/cluster-config-guide.md#dh_generate_certificates).
> [!WARNING]
> This model is clearer and safer, but it also makes private-key placement more important. Machines that run user code should not have access to certificates that let them impersonate administrative services. The [security hardening guide](hardening-technical-controls.md#isolate-user-code-execution-from-administrative-services) explains how to isolate those roles.
### Optional truststore
Previous versions of Deephaven relied on a Deephaven-managed truststore file even when the client already trusted the certificate chain. In Grizzly+, client truststores are optional, except for Kubernetes installations.
The decision of whether to leave the matter of trusting TLS certificates up to a [system CA bundle](../admin-glossary.md#system-ca) or to [provide your own CA bundle](../admin-glossary.md#customer-ca) depends largely on who is expected or allowed to modify trust materials on client and server machines.
In some cases, the system administrator with permission to [modify system truststores](../troubleshooting/troubleshooting-certificates.md#manage-system-truststores) is a different developer than the Deephaven cluster administrator. If the system administrator wants to update trust materials for many different products, using [system CA trust materials](../admin-glossary.md#system-ca) allows the system administrator to add or remove trusted system certificates without updating any files on the Deephaven server or client machines. The server and client processes will need to be restarted unless you are revoking certificates and have explicitly [enabled certificate revocation checking](../troubleshooting/troubleshooting-certificates.md#automatic-java-certificate-revocation) in your JVM processes.
If the Deephaven cluster administrators need to tightly control which certificates are trusted, or configure TLS trust without any root permission, they can supply an [explicit PEM bundle of trusted certificates](#deephaven-trust-models). By opting in to an explicit truststore, clients and servers are forced to verify TLS certificates against the [customer-provided CA trust materials](../admin-glossary.md#customer-ca).
Using an explicit truststore allows the Deephaven cluster administrator to update the trusted certificates without needing root permission, but this model requires restarting all server and client processes to update the truststore. In this case, an explicit truststore file will be generated in `/etc/sysconfig/deephaven/trust/truststore-iris.p12` and the `tls.truststore` [Deephaven property](../admin-glossary.md#deephaven-property) will be configured to instruct [TLS clients](../admin-glossary.md#tls-client) to only trust certificates signed by your supplied [root CA certificates](../admin-glossary.md#root-ca-certificate).
Platform-specific advice on configuring the use of a truststore is covered in the [TLS trust models](#deephaven-trust-models) section below.
### Certificates are always overwritten
Previous certificate-handling practices encouraged administrators to inspect certificates and truststores on disk, modify them in place, and gradually merge two sources of truth.
In Grizzly+, the certificates and trust material you provide are the source of truth. Deephaven regenerates and replaces the derived certificates, keystores, and truststores it manages on disk every time you upgrade or [rotate certificates](#certificate-rotation). This is a breaking change for workflows that previously edited keystores or truststores in place on hosts.
### Fewer generated certificates
Earlier native releases generated certificates for some services while depending on shared customer-provided certificates for others. That made the trust model harder to expose safely and harder to reason about.
In Grizzly+, native installs without Envoy do not generate any certificates. Native installs with Envoy generate only the internal per-machine certificates and internal trust used behind Envoy.
## TLS trust boundaries
Deephaven uses TLS in three trust relationships:
- **[External TLS trust](../admin-glossary.md#external-tls-trust)** is between clients and the certificate they connect to.
- **[Internal TLS trust](../admin-glossary.md#internal-tls-trust)** is between Deephaven services.
- **Etcd TLS trust** is between Deephaven services and etcd.
### External TLS trust
External TLS trust covers browsers, the Deephaven Launcher, Python clients, Java clients, and other outside callers.
When Envoy is enabled, clients trust the certificate Envoy presents. When Envoy is not enabled, clients trust the certificate presented directly by the Deephaven service they reach.
When Deephaven generates client trust, the familiar [external truststore](../admin-glossary.md#external-truststore) is `tls.truststore=truststore-iris.p12`. Otherwise, clients use system, browser, JVM, or Python trust. In practice, browsers on Windows usually use the Windows certificate store, browsers on macOS usually use Keychain, and Java applications may use either OS trust integration or their own `cacerts` truststore. Deephaven Launcher 9.00 and later also attempt to use Windows and macOS system truststores.
### Internal TLS trust
Internal TLS trust governs how Deephaven services trust one another.
When `internal.tls.truststore` is set, Deephaven services use that truststore for server-to-server TLS instead of the external `tls.truststore`. This lets internal TLS trust a narrower set of certificates than client-facing TLS.
When `internal.tls.truststore` is not set, internal TLS falls back to the same truststore and trust source used externally.
For compatibility, the internal and external truststores use the same passphrase file, but they can trust different certificate sets.
This internal split used to be most visible in Kubernetes. In Grizzly+, native installs now formalize the same idea. Native installs with Envoy use this mechanism automatically by generating `internal-truststore.p12` and configuring `internal.tls.truststore` through [generated per-machine certificates](../installation/cluster-config-guide.md#dh_generate_certificates).
### Etcd TLS trust
Etcd uses TLS for communication between Deephaven services and etcd. Those certificates are generated by Deephaven and are separate from the certificates clients trust.
If etcd should be reached on a different network interface than the machine's main Deephaven identity, use [`DH_NODE_N_ETCD_IP`](../installation/cluster-config-guide.md#dh_node_n_etcd_ip). When you change the etcd network or TLS settings, make sure your etcd data is backed up. See [link to etcd add node].
## Request routing with Envoy
When Envoy fronts Deephaven, clients trust the certificate Envoy presents and send traffic through Envoy to the services behind it. Deephaven services still trust each other using the deployment's internal trust model.
```mermaid
flowchart TD
subgraph Frontend[" "]
Browser[Browser or client]
end
subgraph Services[" "]
Envoy[Envoy]
WebApi[Web API]
Controller[Controller]
Auth[Authentication Server]
end
subgraph Etcd[" "]
Etcd3[(etcd 3)]
Etcd1[(etcd 1)]
Etcd2[(etcd 2)]
end
Browser -->|external TLS| Envoy
WebApi -->|internal TLS| Controller
Controller -->|internal TLS| Auth
WebApi -->|internal TLS| Auth
Envoy -->|proxied| Auth
Envoy -->|proxied| Controller
Envoy -->|proxied| WebApi
Controller -->|etcd client TLS| Etcd2
Auth -->|etcd client TLS| Etcd1
Etcd2 -->|etcd peer TLS| Etcd3
Etcd3 -->|etcd peer TLS| Etcd1
Etcd1 -->|etcd peer TLS| Etcd2
```
With Envoy, external trust and internal trust are intentionally different. Clients trust the Envoy certificate. Deephaven services trust the internal certificates and internal trust material used behind Envoy.
## Request routing without Envoy
When Envoy is not used, clients connect directly to Deephaven services. In this model, external and internal trust use the same certificate set and trust source, since both clients and servers connect to the same [fully qualified domain names](#deephaven-machine-identity).
```mermaid
flowchart TD
subgraph Client[" "]
Browser[Browser or client]
end
subgraph Services[" "]
WebApi[Web API]
Controller[Controller]
Worker[Worker or query service]
Auth[Authentication Server]
end
subgraph Etcd[" "]
Etcd1[(etcd 1)]
Etcd2[(etcd 2)]
Etcd3[(etcd 3)]
end
Browser -->|external TLS| WebApi
Browser -->|external TLS| Worker
Browser -->|external TLS| Controller
WebApi -->|internal TLS| Controller
WebApi -->|internal TLS| Auth
Worker -->|internal TLS| Controller
Worker -->|internal TLS| Auth
Auth -->|etcd client TLS| Etcd2
Controller -->|etcd client TLS| Etcd1
Etcd1 -->|etcd peer TLS| Etcd2
Etcd2 -->|etcd peer TLS| Etcd3
Etcd3 -->|etcd peer TLS| Etcd1
```
Podman can use either routing style. With Envoy, clients can use a single public path into the deployment. Without Envoy, clients may need direct routes to the pods they connect to.
## Deephaven machine identity
Every Deephaven machine has a [machine identity](../admin-glossary.md#machine-identity): the [fully qualified domain name (FQDN)](../admin-glossary.md#fqdn) that other parties use to reach that Deephaven endpoint.
Machine identity matters for TLS because Deephaven relies on standard TLS [hostname verification](../admin-glossary.md#hostname-verification): the hostname used to connect must match a `DNS:` entry in the certificate's [SAN](../admin-glossary.md#san). See [SAN and hostname verification](#san-and-hostname-verification).
In deployments that use Envoy, there are typically two relevant identities:
- **External (client-facing) identity**: the hostname clients use (usually the Envoy hostname and port).
- **Internal (service-to-service) identities**: the hostnames Deephaven services use to connect to each other (for example, per-machine hostnames in native/Podman, or Kubernetes service DNS names).
Where the identity comes from depends on your [Deephaven installation type](../admin-glossary.md#deephaven-installation-type):
- **Native**: per-machine identity is configured in `cluster.cnf` via [`DH_NODE_N_DOMAIN`](../installation/cluster-config-guide.md#dh_node_n_domain). Deephaven uses these names when writing service addresses, validating provided per-machine certificates, and (when enabled) generating internal per-machine certificates.
- **Podman**: per-machine identity is determined by the `--hostname` (`-h`) value passed to `start_command.sh`. The per-machine certificate from the TLS volume must include that hostname in its SAN entries.
- **Kubernetes**: the **external** client-facing identity is the `envoyFrontProxyUrl` (and the `deephaven-tls` secret certificate must cover that hostname). The **internal** identities are typically Deephaven-managed Kubernetes service hostnames used for service-to-service TLS.
All machines in a cluster must resolve Deephaven machine names consistently. If different machines resolve the same DNS hostname to different IP addresses, service-to-service TLS and authentication token validation will fail.
Connections to Etcd servers use IP addresses rather than machine hostnames. The native installer derives the default IP address for Etcd servers from the machine's [FQDN machine identity](../admin-glossary.md#machine-identity). If your Etcd servers listen on a different IP address than your [Deephaven Java services](../admin-glossary.md#deephaven-services), you can manually set [`DH_NODE_N_ETCD_IP`](../installation/cluster-config-guide.md#dh_node_n_etcd_ip) to configure all Etcd traffic to use your configured IP address.
> [!CAUTION]
> When using the native Deephaven installer, changing the machine identity of a [Deephaven Java service](../admin-glossary.md#deephaven-services) is straightforward. You provide updated certificates and update `cluster.cnf` to use a new hostname.
> Changing the machine identity (IP address) of an [etcd server](../core-components/etcd.md) is a [complex, manual process](../management/replace-etcd-node.md) which requires redistributing TLS signing and trust materials to all Etcd and Deephaven servers, as well as running `etcdctl.sh` commands to modify the Etcd cluster membership.
## SAN and hostname verification
Deephaven performs [hostname verification](../admin-glossary.md#hostname-verification) against the certificate's [Subject Alternative Name](../admin-glossary.md#san) entries. Neither certificate Common Names nor authority overrides are used for Deephaven hostname verification.
Every TLS certificate a [Deephaven service](../admin-glossary.md#deephaven-services) presents must include SAN entries for the [hostnames](../admin-glossary.md#fqdn) that clients or Deephaven services use to reach that server.
In native installations, [each machine in the cluster requires its own TLS signing certificate](#one-certificate-per-machine), and that certificate must include [SAN](../admin-glossary.md#san) `DNS:` entries matching that [machine's configured FQDN](../installation/cluster-config-guide.md#dh_node_n_domain).
In Podman, the `tls.crt` certificate in the [`--tls-volume` passed to `start_command.sh`](../installation/podman-installation.md#supplying-signing-certificates-to-podman) must include SAN `DNS:` entries for the `--hostname` used to [start the `dh-ifra` pod](../installation/podman-installation.md#start-the-cluster).
In Kubernetes, the [frontend certificate specified by the `deephaven-tls` secret](../kubernetes/troubleshooting-kubernetes.md#how-do-i-update-the-deephaven-tls-webserver-certificate) must include SAN `DNS:` entries that match the client-accessible hostname defined by [`envoyFrontProxyUrl`](../kubernetes/kubernetes-configuration-settings.md#configuring-envoy) in `values.yaml`.
Wildcard certificates are allowed, but they enable impersonation and should be avoided.
## TLS certificate requirements
All [Deephaven services](../admin-glossary.md#deephaven-services) serving TLS requests to [Deephaven clients](../admin-glossary.md#deephaven-client) or other Deephaven services must present a [leaf certificate](../admin-glossary.md#leaf-certificate) that is usable as a TLS web-server certificate.
- the [signing certificate chain](../admin-glossary.md#signing-certificate) used by the server must include the leaf certificate followed by any required [intermediate CA certificates](../admin-glossary.md#intermediate-certificate);
- the [leaf certificate's](../admin-glossary.md#leaf-certificate) [subject alternative name extension](../admin-glossary.md#san) must include a `DNS:host.name` or `DNS:*.host.name` entry that satisfies [hostname verification](#san-and-hostname-verification) for the server's [configured FQDN](#deephaven-machine-identity);
- the leaf certificate's Key Usage permits `Digital Signature`; and
- the leaf certificate's Extended Key Usage permits `TLS Web Server Authentication`.
Deephaven does not use the certificate Common Name or authority overrides for TLS connection verification. Only the [SAN](../admin-glossary.md#san) is considered.
When a certificate arrives in DER or `.cer` form, [convert it to PEM](../troubleshooting/troubleshooting-certificates.md#inspect-certificate-contents) before use.
## Required TLS certificates
| Installation method | Requires per-machine certs | Requires Envoy cert | Generates certificates |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| Kubernetes | Generated automatically | Always, provided via the `deephaven-tls` Secret | Per-machine via Deephaven-generated or cert-manager-managed certificates |
| Podman | Always, via [`tls.crt` and `tls.key` on the host machine's `--tls-volume`](../installation/podman-installation.md#supplying-signing-certificates-to-podman) | Optional, via [`envoy.pem` in the TLS volume](../installation/podman-installation.md#supplying-signing-certificates-to-podman) when Envoy should answer on a different hostname | Never, but ships with `create_dh_cert.sh` to generate your own |
| Native, Envoy enabled | Optional; set [`DH_GENERATE_CERTIFICATES=false`](../installation/cluster-config-guide.md#dh_generate_certificates) to require provided per-machine certs | Always, via [`DH_ENVOY_LOCAL_CERT`](../installation/cluster-config-guide.md#dh_envoy_local_cert) | Optional; set [`DH_GENERATE_CERTIFICATES=true`](../installation/cluster-config-guide.md#dh_generate_certificates) |
| Native, no Envoy | Always, one certificate chain and private key per machine | Never | Never |
- **Native installation:** certificate inputs are supplied on the Installation Host via `cluster.cnf`; you must supply a signing certificate chain for the Envoy load balancer, or signing certificates for every machine in the Deephaven cluster.
See [Basic installation: TLS certificates](../installation/basic-install.md#tls-certificates) for details on the specific `cluster.cnf` properties and required input files.
- **Podman:** you provide the certificate chain and private key as PEM files via the TLS volume (passed with `--tls-volume`), which is mounted into the container so Deephaven can read the expected cert/key paths at runtime.
See the [Podman installation guide](../installation/podman-installation.md) (TLS volume and certificate file placement).
- **Kubernetes:** you provide the certificate chain and private key for the `envoyFrontProxyUrl` in a Kubernetes Secret named `deephaven-tls`, with `tls.crt` (chain) and `tls.key` (private key) data fields, which are mounted into the Envoy/cluster pods as files via a volume mount.
See the [Kubernetes installation guide](../kubernetes/kubernetes-install-guide.md) (TLS Secret configuration and optional cert-manager).
All provided [X.509 certificates](../admin-glossary.md#certificate-x509) must include a [Subject Alternative Name](../admin-glossary.md#san) extension capable of performing [hostname verification](../admin-glossary.md#hostname-verification) against the [machine identity FQDN](#deephaven-machine-identity) used to make the TLS connection.
To put it simply, if you connect to `auth.deephaven.example.com`, that server must sign TLS requests using a certificate with a [SAN](../admin-glossary.md#san) entry for `DNS:auth.deephaven.example.com` (preferred) or `DNS:*.deephaven.example.com` (discouraged).
In addition to hostname verification, the signing certificate must also form a [complete chain of TLS trust](../admin-glossary.md#trust-and-truststores). This means that the certificates presented by the server must be issued by certificates present in the client's [system truststore](../admin-glossary.md#system-ca) or [explicitly configured truststore](../admin-glossary.md#customer-ca).
Deephaven installations transform provided [X.509 PEM certificates](../admin-glossary.md#certificate-x509) into `PKCS12` format ".p12 TLS keystore files" which are restricted to [Deephaven service accounts](../admin-glossary.md#deephaven-service-accounts) in the `/etc/sysconfig/deephaven/auth` and `/etc/sysconfig/deephaven/auth-user` directories. Any explicit truststore files (in `PKCS12` and `PEM` format) are world-readable and stored in `/etc/sysconfig/deephaven/trust`.
All generated keystore and truststore files are overwritten with your supplied certificates whenever you upgrade Deephaven.
## Deephaven trust models
### Native installation trust model
Native installations have the [most configurable trust model](../installation/basic-install.md#tls-certificates).
#### External trust in native installations
For native installations, [Deephaven clients](../admin-glossary.md#deephaven-client) can either rely on [system trust](../admin-glossary.md#system-ca) or use an [explicitly provided X.509 PEM format CA certificate bundle](../installation/cluster-config-guide.md#dh_ca_certificate).
When you provide an explicit CA bundle through [`DH_CA_CERTIFICATE`](../installation/cluster-config-guide.md#dh_ca_certificate), Deephaven generates the external client truststore at `/etc/sysconfig/deephaven/trust/truststore-iris.p12`, and configures the `tls.truststore` [Deephaven property](../admin-glossary.md#deephaven-property) to direct processes to use the explicit truststore file instead of the system/Java truststore.
When you do not set `DH_CA_CERTIFICATE`, no truststore is generated, and clients rely on [system, browser, JVM, or Python trust](../admin-glossary.md#system-ca).
When you set [`DH_CONFIGURE_ENVOY=true`](../installation/cluster-config-guide.md#dh_configure_envoy) and [`DH_GENERATE_CERTIFICATES=true`](../installation/cluster-config-guide.md#dh_generate_certificates) in `cluster.cnf`, Deephaven generates an Envoy frontend TLS signing certificate whose [subject alternative name TLS extension](../admin-glossary.md#san) must match the configured [Envoy frontend url](../envoy/installing-envoy.md#envoy-frontend-url) defined by [`DH_ENVOY_FQDN`](../installation/cluster-config-guide.md#dh_envoy_fqdn). The `DH_CA_CERTIFICATE` value is used as the CA bundle to validate this certificate, and should be a [CA certificate](../admin-glossary.md#certificate-authority-ca) which cannot have and does not need a SAN entry matching the frontend URL.
When you set [`DH_CONFIGURE_ENVOY=false`](../installation/cluster-config-guide.md#dh_configure_envoy) or [`DH_GENERATE_CERTIFICATES=false`] in `cluster.cnf`, then any [`DH_CA_CERTIFICATE`](../installation/cluster-config-guide.md#dh_ca_certificate) you specify must contain all [CA certificates](../admin-glossary.md#certificate-authority-ca) necessary to trust [all TLS signing certificates](../installation/basic-install.md#tls-certificates) supplied in [`DH_LOCAL_CERT_DIR`](../installation/cluster-config-guide.md#dh_local_cert_dir).
#### Internal trust in native installations
Internal trust verification for TLS connections between [Deephaven services](../admin-glossary.md#deephaven-services) depends on how certificates are supplied.
When you set [`DH_CONFIGURE_ENVOY=true`](../installation/cluster-config-guide.md#dh_configure_envoy) and [`DH_GENERATE_CERTIFICATES=true`] in `cluster.cnf`, Deephaven generates a strict internal truststore to secure server-to-server communication at `/etc/sysconfig/deephaven/trust/internal-truststore.p12`. This internal truststore is specified by the `internal.tls.truststore` [Deephaven property](../admin-glossary.md#deephaven-property) and is generated and updated automatically whenever you [rotate certificates](#certificate-rotation).
In all other cases, the native installation internal trust model behaves the same as the [external trust model](#external-trust-in-native-installations). Deephaven services will use [`DH_CA_CERTIFICATE`](../installation/cluster-config-guide.md#dh_ca_certificate) if specified, or the [system CA certificates](../admin-glossary.md#system-ca) if `DH_CA_CERTIFICATE` is not specified.
### Podman installation trust model
TLS trust model for Podman installations depends on [passing `--ca-cert-file` to `start_command.sh`](../installation/podman-installation.md#supplying-trusted-certificates-to-podman).
If you pass `--ca-cert-file`, then the specified X.509 PEM-formatted CA certificate bundle is used as an [explicit truststore file](../admin-glossary.md#customer-ca). If you do not pass `--ca-cert-file`, Podman instead relies on the default system/Java truststores inside the containers for TLS verification.
The system truststores for Podman can be updated by [passing `--ca-cert-file` to `build_images.sh`](../installation/podman-installation.md#build_imagessh), which means you can forego passing `--ca-cert-file` to `start_command.sh` and opt in to using system truststores.
#### External trust in Podman installations
Podman can rely on [system trust](../admin-glossary.md#system-ca) or take explicit trust material through `--ca-cert-file` when [building images](../installation/podman-installation.md) or [starting the cluster](../installation/podman-installation.md).
When `--ca-cert-file` is [provided to `start_command.sh`](../installation/podman-installation.md#supplying-trusted-certificates-to-podman), the Podman installation copies your provided CA certificate bundle into the image at `/etc/sysconfig/deephaven/trust/truststore-iris.p12` and configures [Deephaven property](../admin-glossary.md#deephaven-property) `tls.truststore` to instruct both clients and servers to use the CA material you provided.
When `--ca-cert-file` is not [provided to `start_command.sh`](../installation/podman-installation.md#supplying-trusted-certificates-to-podman), all client and server TLS connections will rely on their local [system/Java truststores](../admin-glossary.md#system-ca).
To alter the system truststores in your Podman images, you can [pass `--ca-cert-file` to `build_images.sh`](../installation/podman-installation.md#infra-image-build-arguments).
#### Internal trust in Podman installations
Separating Podman internal service-to-service trust from external client-to-service trust requires [using Envoy in Podman](../envoy/installing-envoy.md#install-envoy-on-podman), [providing a separate signing certificate for Envoy](../installation/podman-installation.md#supplying-trusted-certificates-to-podman), and then manually configuring the `internal.tls.truststore` [Deephaven property](../admin-glossary.md#deephaven-property).
You must place your internal [PKCS12 .p12 truststore](../admin-glossary.md#certificate-x509) on the filesystem inside each Podman container that participates in internal service-to-service TLS, typically at `/etc/sysconfig/deephaven/trust/truststore-internal.p12`, and configure the `internal.tls.truststore` [Deephaven property](../admin-glossary.md#deephaven-property) to point to that absolute path (for example, `internal.tls.truststore=/etc/sysconfig/deephaven/trust/truststore-internal.p12`) so that internal connections use the internal CA material instead of the external trust configuration.
Podman internal trust follows the routing model you deploy.
When services connect directly, internal trust can match external trust. When Envoy fronts the deployment, Podman can separate the external Envoy certificate from the certificates and trust used internally.
#### External trust in Kubernetes
Kubernetes uses the frontend certificate provided through the `deephaven-tls` Secret for client-facing TLS.
#### Internal trust in Kubernetes
Kubernetes already separates internal and external trust. Internal certificates are generated or managed separately from the frontend certificate.
A Deephaven-managed optional client truststore like native and Podman installations is not yet available for Kubernetes.
## Certificate rotation
Certificate rotation always starts with the certificates or trust material you provide. Deephaven then replaces the derived files it manages on disk.
### Rotate certificates for native installations
- Update the per-machine certificate inputs on the Installation Host.
- If you use Envoy, update the certificate supplied through [`DH_ENVOY_LOCAL_CERT`](../installation/cluster-config-guide.md#dh_envoy_local_cert).
- If you supply a CA bundle through [`DH_CA_CERTIFICATE`](../installation/cluster-config-guide.md#dh_ca_certificate), update it so Deephaven can regenerate the external truststore.
- Rerun the supported installer or certificate-rotation workflow.
- Restart affected services if needed.
### Rotate certificates for Podman installations
- Replace the certificate files in the TLS volume, including `envoy.pem` if Envoy answers on a different hostname.
- Restart the affected pod or the full deployment.
If trust is supplied via `--ca-cert-file` at image build time, rebuild the images when the trusted CA changes.
### Rotate certificates for Kubernetes installations
- Update the `deephaven-tls` Secret used for the frontend (Envoy) certificate.
- Follow the supported restart or upgrade flow for the deployment.
Internal certificate rotation depends on how the cluster manages those certificates (for example, Deephaven-generated or cert-manager-managed certificates).
### Rotate certificates manually
Manual and advanced rotation workflows are covered in the [certificate troubleshooting guide](../troubleshooting/troubleshooting-certificates.md#manual-certificate-rotation).
## Related items
- [Admin glossary](../admin-glossary.md): Defines the TLS and trust terms used throughout this page.
- [Cluster configuration guide](../installation/cluster-config-guide.md): Documents native installer certificate and machine-identity settings.
- [Basic installation](../installation/basic-install.md): Explains native installation flow and certificate inputs.
- [Podman installation](../installation/podman-installation.md): Explains Podman TLS volumes, optional `envoy.pem`, and `--ca-cert-file`.
- [Use Envoy as a front proxy](../envoy/envoy.md): Explains how Envoy fronts Deephaven services.
- [Troubleshooting certificates](../troubleshooting/troubleshooting-certificates.md): Covers certificate inspection, truststore updates, and repair steps.
- [Security hardening](hardening-technical-controls.md): Explains how to isolate administrative services and protect machine certificates from user-code hosts.