Troubleshoot certificate errors

Deephaven uses X.509 certificates to secure network communication for the Web UI, internal services, and connections to external systems. An X.509 certificate is a standard digital document that securely binds a public key to an identity, such as a server hostname. This guide helps system administrators and developers diagnose and resolve common certificate-related issues.

Certificate problems can prevent users from accessing the Web UI, block communication between internal Deephaven services, or cause failures when connecting to external systems like Kafka, SAML, or LDAP.

Certificates are used when connecting both internal and external services in Deephaven:

How certificates are used

Certificates are primarily used to authenticate the identity of servers (and sometimes clients), and to facilitate the setup of an encrypted channel (e.g., TLS/SSL) to protect data in transit. This involves using public and private keys for encryption, decryption, and digital signing.

  • Encryption: Data is encrypted using a recipient's public key and can only be decrypted using the corresponding private key. This ensures confidentiality.
  • Digital Signatures: Data is signed using the sender's private key. Anyone with the sender's public key can verify the signature, ensuring data integrity and authenticity.

A crucial concept in certificate validation is the chain of trust. Typically, certificates are issued by Certificate Authorities (CAs). A server's certificate is signed by an Intermediate CA, which in turn is signed by a Root CA. Client systems maintain a list of trusted Root CAs. For a server's certificate to be trusted, the client must be able to trace its lineage back to a trusted Root CA in its store.

  • Encryption uses the public key, and decryption uses the private key. When sending data to another system, the sender must first obtain the public key from the recipient. Only a process with the private key corresponding to the recipient's public key can decrypt that data.
  • Signing is somewhat the reverse of encryption: the sender uses its private key to create a digital signature on a block of data, such as a file or message. Any system with access to the corresponding public key can verify the authenticity of the data and that it has not been tampered with since it was sent.

Most certificate use in Deephaven is to prove identity of endpoints; this is done through signing. The identity validation process also may involve traversing a chain of certificates to find a trusted issuer somewhere up the chain.

Below is a simplified overview of how this identity verification process works:

  1. A client system initiates a connection to a server at https://mydeephaven.mydomain.com.
  2. Because the URL specifies HTTPS, the browser requests the server to send proof of its identity.
  3. The server sends a certificate that has been signed by the certificate authority (CA) that issued it.
  4. The client checks its truststore to see if it has a trusted certificate for the CA.
  5. If is has a trusted certificate that matches the name of the CA, it will use the public key contained in that certificate to verify that the certificate the server sent is authentic. If it does not find a matching trusted CA certificate, it will reject the server's certificate because it cannot be verified.
  6. If the signature on the certificate has been verified, there are still other checks the browser makes to ensure that the certificate is valid for the use being requested, that it is not expired, that the name mydeephaven.mydomain.com is a valid name for this certificate, and that it has not been revoked.

Most certificate problems for Deephaven's own certificates are related to issues with the certificate chain (e.g., missing intermediate certificates, untrusted root CAs), certificate expiration, hostname mismatches, or incorrect installation and configuration.

Deephaven systems can be configured to use certificates issued from a well-known public CA, from a corporate PKI, or even self-signed certificates (typically for testing or internal environments). This guide explains how to diagnose and correct some errors that can occur due to problems with system configuration or the certificates themselves.

Common certificate issues

This guide assumes you have openssl, curl, and keytool installed on your Deephaven system. If you do not, check with your infrastructure team about installing them.

A common starting point for certificate troubleshooting is a browser security warning like "Your connection is not private" or "Warning: Potential Security Risk Ahead". This generic error can be caused by several underlying issues. Recognizing more specific error messages can help you pinpoint the problem faster:

Inspect certificate contents

When debugging certificate failures, inspect the local certificate file before you troubleshoot trust or routing. This quickly confirms whether the certificate is actually a TLS web-server certificate and whether it contains the expected SAN entries.

Look for the following on the leaf certificate:

  • Key Usage: Digital Signature
  • Extended Key Usage (EKU): TLS Web Server Authentication
  • Subject Alternative Name (SAN): the hostname or hostnames clients actually use

Common Deephaven certificate file input locations depend on the installation type:

  • Native Grizzly+ installs typically use a per-machine certificate such as /etc/deephaven/certs/$(< /etc/deephaven/DH_NODE_NAME).pem.
  • Podman commonly uses tls.crt in the TLS volume.
  • Older native installations no longer use the deprecated /etc/deephaven/cus-tls/tls.crt location.

Input certificates are often translated to PKCS12 and stored in .p12 keystores for Java applications to consume. These files are stored in /etc/sysconfig/deephaven/{auth,auth-user,trust} directories, with their passphrase co-located in the same directory. These files are overwritten with your input certificates whenever you rotate certificates automatically and can be updated through manual certificate rotation.

If the certificate is stored in a PKCS#12 / .p12 file, extract the certificate PEM first so you can inspect it with openssl x509:

If the certificate was delivered in DER or .cer form instead of PEM, convert it first:

Verify certificate chains

Inspect the presented chain and verify trust

Certificate-chain problems are broader than missing intermediates. They can include a missing intermediate CA, an incorrect certificate order in a bundle, a chain that does not lead to a trusted root, or a server presenting the wrong certificate entirely.

Common Symptoms:

  • Browser errors such as "Your connection is not private" or "NET::ERR_CERT_AUTHORITY_INVALID"
  • curl: (60) SSL certificate problem: unable to get local issuer certificate
  • Java errors such as javax.net.ssl.SSLHandshakeException: PKIX path building failed
  • OpenSSL errors such as unable to get issuer certificate or self signed certificate in certificate chain

Troubleshooting Steps:

  1. Inspect the chain presented by the server: Use openssl s_client from the failing client machine to see the certificate chain actually returned during the TLS handshake.

    Replace your.server.com with the actual hostname and 443 with the correct port.

  2. Check the leaf certificate file Deephaven is configured to present: The exact file depends on the installation type:

    • Native installs without Envoy commonly use a per-machine PEM file such as ${DH_LOCAL_CERT_DIR}/my-node.pem.
    • Podman commonly uses tls.crt in the TLS volume.
    • Older layouts may use /etc/deephaven/cus-tls/tls.crt.

    Confirm that the file contains the leaf certificate first, followed by any required intermediate CA certificates.

  3. Verify the chain against a trusted CA bundle: If Deephaven has generated a PEM trust bundle, use the .pem bundle rather than the .p12 truststore. A common location is /etc/sysconfig/deephaven/trust/truststore-iris.pem.

    For native installations, the expected certificate name uses the machine's DH_NODE_N_HOST with a .pem extension for the signing certificate, and .key for the private key. This hostname is written to the /etc/deephaven/DH_NODE_NAME file on each machine. Thus, /etc/deephaven/certs/$(< /etc/deephaven/DH_NODE_NAME).pem is the default location where Deephaven installation scripts place and expect certificates.

  4. Check Subject and Issuer relationships: Each certificate's Issuer should match the next certificate's Subject until the chain reaches a trusted root CA.

  5. Confirm the hostname is covered by SANs: A chain can be valid but still fail if the certificate does not match the hostname in use. See Hostname mismatch.

Resolution:

  • If the server is not presenting the full chain, update the configured certificate bundle so it includes the leaf certificate followed by all required intermediate certificates.
  • If the wrong certificate is being presented, correct the deployment input and rerun the supported install or update workflow.
  • If the chain only validates with a private or corporate CA, make sure the clients that connect to Deephaven trust that CA. See Manage system truststores.

Certificate expired

Certificate expired

Hostname mismatch

This error occurs when the hostname you are using to access a service (e.g., my.deephaven.server.com) does not match one of the Subject Alternative Names (SANs) listed in the server's TLS certificate. Certificates are issued for specific hostnames to prevent man-in-the-middle attacks. Modern Deephaven TLS connections use SAN matching; the certificate Common Name (CN) is not used.

Common Symptoms:

  • Browser errors like "NET::ERR_CERT_COMMON_NAME_INVALID" (which can be misleading if the Subject Alternative Name (SAN) is the actual issue, as modern clients prioritize SAN over Common Name (CN)) or "Your connection is not private."
  • Application-level errors indicating a certificate validation failure related to hostname.
  • javax.net.ssl.SSLPeerUnverifiedException: Certificate for <hostname> doesn't match any of the subject alternative names: [list...]
Troubleshooting hostname mismatch

Troubleshooting Steps:

  1. Verify the Hostname: Double-check the exact hostname you are using in your browser URL, application configuration, or client connection string.
  2. Inspect the Certificate:
    • Using a Browser: Click the padlock icon in the browser's address bar when connected to the HTTPS service, view certificate details, and look for the "Common Name" (CN) and "Subject Alternative Name" (SAN) fields. Ensure the hostname you're using is listed in one of these, paying close attention to the SAN entries, as modern browsers and applications prioritize these over the CN.
    • Using openssl: Alternatively, use openssl to check the CN and SAN. The grep will show Subject: CN= and X509v3 Subject Alternative Name:. If the SAN field is missing or doesn't contain the correct hostname, this is the likely cause of the error, even if the CN appears correct. Replace your.server.com with the actual hostname and 443 with the correct port if not standard HTTPS. To see all certificate details, including Key Usage and EKU, remove the | grep ... part.
    • Using curl (especially for non-browser clients or to see connection details): Replace your.server.com with the actual hostname. Examine the output for SSL handshake details (look for lines starting with * SSL) and the certificate presented by the server (lines starting with * Server certificate:). curl can also reveal if the server is presenting an unexpected certificate or if there are other TLS handshake issues.
    • Verify Key Usage and Extended Key Usage (EKU): Ensure the certificate is valid for its intended purpose. For a web server, it should typically have:
      • Key Usage: Digital Signature, Key Encipherment
      • Extended Key Usage (EKU): TLS Web Server Authentication You can see these details in the browser's certificate viewer or in the full output of the openssl x509 -text command.
  3. Resolution:
    • Correct the Hostname: If you are using an incorrect hostname to access the service, update your client or configuration to use one that is listed in the certificate's SAN entries.
    • Reissue the Certificate: If the certificate does not cover all necessary hostnames (e.g., you've added a new alias for the server), the certificate needs to be reissued with the correct CN and/or additional SANs. Consult your certificate authority (CA) or internal PKI team for this process.
    • Wildcard Certificates: If applicable, ensure your wildcard certificate (e.g., *.deephaven.example.com) correctly covers the hostname in question. Remember that wildcards only cover one subdomain level, *.deephaven.example.com does NOT match two.levels.deephaven.example.com. It is recommended to avoid wildcards to prevent impersonation.

Certificate not trusted

This issue occurs when the certificate presented by the server is signed by a Certificate Authority (CA)—or is self-signed—and is not present in the client's trust store. This is the standard behavior for certificates issued by an internal corporate PKI or any other custom CA, and is always the case for self-signed certificates, which must never be used in production environments.

Common Symptoms:

  • Browser errors like "NET::ERR_CERT_AUTHORITY_INVALID".
  • Application-level errors like javax.net.ssl.SSLHandshakeException: PKIX path building failed.
Troubleshooting untrusted certificates

Troubleshooting Steps:

To resolve this, the client system (e.g., a user's web browser, a client application, or another Deephaven service) must be configured to trust the custom CA. This typically involves importing the CA's root and/or intermediate certificates into the appropriate trust store.

See Manage system truststores for platform-specific truststore locations and import methods.

A few additional important considerations for PKI and custom CA setups:

  • Certificate File Format: Deephaven expects PEM-encoded certificate material. If your certificate or key is in a binary format (such as DER), convert it to PEM using a tool like openssl.
  • Platform-specific file names: Native Grizzly+ installs without Envoy use per-machine inputs under DH_LOCAL_CERT_DIR as <node>.pem and <node>.key. Envoy uses a single concatenated PEM bundle. tls.crt and tls.key are Kubernetes Secret data keys, as well as the default certificate name to use when adding certificates to Podman installation. Native installations no longer use tls.crt and tls.key files at all.
  • Complete Certificate Chain: When using certificates issued by a corporate PKI or a custom CA, the certificate chain presented by Deephaven should contain the server certificate plus all required intermediate CA certificates. If the chain is incomplete, clients might fail validation even if they trust the root CA.

Manage system truststores

This section covers the client-side truststores Deephaven users and services commonly rely on when the certificate chain is signed by a private CA, a corporate PKI, or a self-signed root used for testing.

Manage Windows truststore

Windows has a system-wide trust store. Certificates can be managed via certlm.msc (for the local machine) or certmgr.msc (for the current user). Import the CA certificate into the "Trusted Root Certification Authorities" store. In Active Directory environments, enterprise-authorized Microsoft Certificate Services CAs are often added to this store automatically for domain-joined machines.

Manage macOS truststore

On macOS, certificates are managed in the Keychain Access application. You can import CA certificates into the "System" keychain to make them available to all users and most applications, or the "login" keychain for just the current user.

After importing, you must explicitly set the trust level for the certificate. Open the certificate, expand the "Trust" section, and set "When using this certificate" to "Always Trust".

Important

In mid-2022, a change to all versions of Java affects how Java interacts with the macOS keychain trust store. Certificates that appear valid and are trusted by browsers may not be trusted by Java applications.

The following versions of the JDK contain JDK-8278449 ("Only Expose Certificates With Proper Trust Settings as Trusted Certificate Entries in macOS KeychainStore"):

  • JDK 8: 8u332 or later
  • JDK 11: 11.0.15 or later
  • JDK 17: 17.0.3 or later

If you are using one of these versions, you must use the security command-line tool to add the certificate to the keychain and set its trust level. Using the Keychain Access GUI is not sufficient.

Manage Linux truststore

Most Linux distributions use an operating-system trust bundle built from a directory of trusted CA certificates. To trust a private CA system-wide, copy the CA certificate into the distribution's trust-anchor directory and refresh the trust bundle.

Use the command set that matches your distribution family:

  • Debian / Ubuntu:

  • RHEL / Rocky / AlmaLinux:

After updating the trust store, retry the failing browser, curl, Java client, Python client, or Deephaven service connection.

If you need to confirm the CA is now trusted by OpenSSL, inspect the system trust bundle path and verify again:

Some Java distributions on Linux use the OS trust store, while others continue to use their own cacerts file. If a Java-based Deephaven tool still does not trust the certificate after updating Linux system trust, update the Java truststore as well.

Java Truststores (cacerts)

Java applications, including Deephaven services, sometimes use their own truststore, often called cacerts; at other times, your Java installation uses your Operating System trust materials. If a Deephaven service needs to connect to another service secured by a private CA, and you are not using an explicit truststore, you may need to import the CA certificate into the Java truststore.

The default location of the root CA truststore included with a Java distribution is <java_home_directory>/lib/security/cacerts. The default password for this truststore is changeit.

To import a private CA certificate to the root CA truststore, use the Java keytool utility:

Note that the -keystore argument can be used (instead of -cacerts) to indicate a particular installation's truststore. This should be done in cases where there is more than one installation of Java, and the one to be updated for the Launcher is not the default one used in a shell console.

Manual certificate rotation

Warning

This section is intended for advanced troubleshooting and recovery. Prefer the supported rotation workflows for your installation method whenever possible. Incorrect certificate changes can prevent clients from connecting, break internal service-to-service TLS, and may require console access to recover.

Before you rotate certificates manually:

  • Confirm which hostname clients and services use to reach each endpoint. Deephaven validates the certificate against the SAN entries.
  • Confirm you have the full certificate chain (leaf certificate followed by any required intermediates).
  • Confirm you have the matching private key.
  • Plan for a coordinated restart of affected services.

Rotate certificates manually (advanced)

Native installations

Native installations use per-machine certificate inputs. Each machine has its own certificate chain and private key. If Envoy is enabled, Envoy has its own client-facing certificate.

  1. Identify the machine identity used for TLS and hostname verification.

    • The expected machine name is stored in /etc/deephaven/DH_NODE_NAME.
    • The certificate you provide must include that name in its SAN entries.
  2. Update the per-machine certificate inputs on the Installation Host.

    • Ensure each machine has a PEM certificate chain file and matching private key.
    • Ensure the certificate chain includes any required intermediate CA certificates.
  3. If Envoy is enabled, update the Envoy certificate specified by DH_ENVOY_LOCAL_CERT.

  4. If you provide a CA bundle through DH_CA_CERTIFICATE, update it so Deephaven can regenerate the external truststore.

  5. Rerun the supported installer or certificate workflow so Deephaven overwrites the derived keystores and truststores under /etc/sysconfig/deephaven/*.

  6. Restart affected services.

Older native / legacy cus-tls layouts

Some older non-Kubernetes native deployments used /etc/deephaven/cus-tls, iris_keygen.sh, and config_packager.sh instead of the Grizzly+ per-machine certificate workflow above. Keep these steps only for those older layouts.

  1. On the infra server, replace /etc/deephaven/cus-tls/tls.crt and /etc/deephaven/cus-tls/tls.key with the new certificate chain and matching private key.

  2. Regenerate and package the updated certificate materials:

  3. If this is a multi-node deployment, copy the generated archive (typically dh_query.tgz) to each query node, then unpackage it there:

  4. If Envoy is used, rebuild the concatenated Envoy PEM bundle from the machine certificate and key. A common location is /etc/deephaven/certs/envoy.pem:

    The resulting envoy.pem should normally be owned by the Deephaven admin user and have permissions 600. On some systems, the runtime path is controlled by DH_ENVOY_REMOTE_PEM in /etc/sysconfig/deephaven/cluster.cnf.

  5. Restart at least the Web API service, Envoy if used, and all Query or Merge Dispatchers.

Podman

Podman reads certificate inputs from the TLS volume.

  1. Replace the certificate chain and private key in the TLS volume.

    • tls.crt should contain the leaf certificate followed by any required intermediate certificates.
    • tls.key must be the matching private key.
  2. If Envoy answers on a different hostname than the backend services, update envoy.pem in the TLS volume.

  3. Restart the affected pod or the full deployment.

If trust is supplied through --ca-cert-file at image-build time, rebuild images when the trusted CA changes.

Kubernetes

Kubernetes uses the deephaven-tls Secret for the client-facing (frontend) certificate.

  1. Update the deephaven-tls Secret with new tls.crt (certificate chain) and tls.key (private key).

  2. Restart or roll the deployment using the supported upgrade/restart flow.

Internal certificate rotation depends on how the cluster manages internal certificates (for example, Deephaven-generated or cert-manager-managed certificates).

Validate certificate rotation

After rotating certificates, validate from both a client machine and from at least one Deephaven node:

  • Inspect the presented certificate and chain:

  • Confirm hostname verification will succeed by checking SAN entries on the leaf certificate:

  • If trust errors persist, re-check the chain order and whether the client trusts the issuing CA. See Verify certificate chains and Manage system truststores.

Deephaven Launcher certificate problems

Deephaven Launcher certificate problems

The Deephaven Launcher behaves much like a browser when creating or updating a Deephaven instance. It must rely on the system or Java truststores to make the initial HTTPS connection to the client update server. You can follow the instructions in the troubleshooting section above to correct trust issues.

Note

Modern versions of the Deephaven Launcher (9.00 and later) automatically attempt to use the underlying operating system's native truststores on Windows and macOS, in addition to the Java truststore. This often simplifies trust configuration for corporate CAs that are already trusted by the OS.

When creating a new instance, if the server certificate can't be verified, the Launcher will display a security warning similar to the warnings shown by most browsers. In this dialog, you can view the certificate chain. Optionally, you can ignore the security warning temporarily in order to download the starting files for the new instance.

When the Deephaven Launcher starts the Deephaven console program, that application uses a truststore that has been configured for the Deephaven system and downloaded with the instance program files. Once that truststore file has been downloaded, the Launcher is able to use that file to make further connections to the system.

To validate the certificates a client will see, it can be helpful to test the Web UI and Client Update Service endpoints directly from a client machine:

The Deephaven Launcher will look for an optional truststore in the program directory. The expected file names for the truststore and passphrase file are truststore-iris.p12 and truststore_passphrase. The names and locations can be changed with properties:

See Deephaven Launcher properties files for more information about locations and content.

This truststore can be created from source certificate(s) or downloaded from the target Deephaven system. One way to copy the generated files is:

Then copy the two files to the Deephaven Launcher program directory, or to the location you configured in a properties file.

The following steps create a PKCS12 truststore on macOS or Linux:

  • Create a passphrase file:
  • Import certificate myCA.pem with alias your_server. You can omit the -storepass argument and type the password instead.

Invalid keystore format

Invalid keystore format or HmacPBESHA256 error

A new, default hashing algorithm was introduced. This means that an "old" JRE without that algorithm cannot read a PKCS12 file written by a "new" JRE that uses the new default.

The following errors might appear in the Deephaven Launcher log file:

  • keytool error: java.io.IOException: Invalid keystore format
  • keytool error: java.io.IOException: Integrity check failed: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available
  • parseAlgParameters failed: ObjectIdentifier() -- data isn't an object ID (tag = 48)

The only remedy is to use a newer JDK version to start the Deephaven Launcher, preferably one that matches the Deephaven Server.

For reference:

You can verify this problem using keytool: