How to troubleshoot certificate errors

Deephaven uses X509 certificates to secure many services, including several HTTPS services. These can be configured to use certificates issued from a well-known CA, from a corporate PKI, or even self-signed certificates. This guide explains how to diagnose and correct some errors than can occur due to problems with system configuration or the certificates themselves.

Browser cannot make secure connection to Web UI or CUS

When you point a browser at a secure (https) website, you might get an error like one of the following if the server's certificate is not known and trusted:

• Warning: Potential Security Risk Ahead (Firefox)
• Your connection isn't private (Edge)
• Your connection is not private (Chrome)
• This site is not secure (Internet Explorer)

This can also happen because of proxies and firewalls.

You can always view the certificates the browser is complaining about. The method varies between browsers and browser versions. Contact Deephaven support or search the web if you cannot navigate the dialogs and menus to the certificates. You can download the certificates from these dialogs.

The most common errors are "certificate expired" and "untrusted certificate":

• If the certificate has expired, the Deephaven servers must be updated with new certificates. See Replacing certificates.
• If the certificate is not trusted, there are several causes and remedies.

Certificates issued by corporate PKI

It is common for companies to use an internal Private Key Infrastructure (PKI), in which there is a company CA that is not in the list of well-known certificates used by operating systems and browsers. In this case, the IT organization usually arranges for the appropriate CAs to be present in the OS trust stores of computers on the corporate network. The first remedy is to consult your IT organization for help.

Subsequent steps are to examine system trust stores, and to import the relevant certificates into the system trust store.

Microsoft Windows

On Windows, the certlm MMC plugin can be used to manage CA certificates - here, they are typically imported to the Trusted Root Certification Authorities store. Membership in a Windows domain will normally add domain level private CA certificates to this store automatically.

Some certificate in the chain provided by the server must be in the Trusted Root Certification Authorities collection. You can use this UI to import and export certificates.

MacOS

On OSX, the Keychain Access application is used to manage the trust store. Additional CA certificates can be imported to the local or System store. Imported certificates must be explicitly marked as trusted before they are effective. You can use this UI to import and export certificates, and to mark them as trusted. Notice in the image below that "myserver.int.illumon.com" is marked as trusted for all users, with a blue plus icon. Its issuer, "intermediate.int.illumon.com", is not trusted. The trust status of a certificate is shown in the summary at the top when it is selected.

You can also use the security command to add and list certificates in keychains. For example:

security list-keychains
security dump-keychain
security dump-trust
security add-trusted-cert -r trustRoot -k /Library/Keychains/System.keychain myCA.pem
security add-trusted-cert -r trustAsRoot -k /Library/Keychains/System.keychain intermediate.crt

In particular, if a certificate is correctly trusted, it will appear in the output of one of these commands:

security dump-trust
security dump-trust -d

Java

Java applications always have the option of using the truststore shipped with the Java installation. Adding certificates to this store should be necessary only on clients that are not Windows or macOS, or when there is no access to the OS system trust stores (e.g., lack of admin access).

The default location of the root CA truststore included with a Java distribution is <java_home_directory>/lib/security/cacerts. The default password for this truststore is changeit.

To import a private CA certificate to the root CA truststore, use the Java keytool utility:

keytool -cacerts -trustcacerts -import -file <certificate file to import> -alias <a unique (to the truststore) alias to use for this certificate>

Note that the -keystore argument can be used (instead of -cacerts) to indicate a particular installation's truststore. This should be done in cases where there is more than one installation of Java, and the one to be updated for the Launcher is not the default one used in a shell console.

Self-signed certificates

Self-signed certificates are usually used only for testing, or temporarily. These certificates are rarely centrally managed. See the sections above for information about importing the self-signed certificate into a trust store.

Deephaven Launcher

The Deephaven Launcher behaves much like a browser when creating or updating a Deephaven instance. It must rely on the system or Java truststores to make the initial HTTPS connection to the client update server. You can follow the instructions for browser errors to correct trust issues.

When creating a new instance, if the server certificate can't be verified, the Launcher will display a security warning similar to the warnings shown by most browsers. In this dialog, you can view the certificate chain. Optionally, you can ignore the security warning temporarily in order to download the starting files for the new instance.

When the Deephaven Launcher starts the Deephaven console program, that application uses a truststore that has been configured for the Deephaven system, and has been downloaded with the instance program files. Once that truststore file has been downloaded, the Launcher is able to use that file to make further connections to the system.

The Deephaven Launcher will look for an optional truststore in the program directory. The expected file names for the truststore and password file are truststore-iris.p12 and truststore_password (matching the default names used in the Deephaven system). The names and locations can be changed with properties:

IrisConfigurationLauncher.truststore
IrisConfigurationLauncher.truststore.passphrase.file

See Deephaven Launcher properties files for more information about locations and content.

This file can be created from source certificate(s) or downloaded from the target Deephaven system.

This is one method for downloading the files:

scp yourserver.company.com:/etc/sysconfig/illumon.d/resources/truststore-iris.p12 .
scp yourserver.company.com:/etc/sysconfig/illumon.d/resources/truststore_passphrase .

Then copy the two files to the Deephaven Launcher program directory, or to the location you configured in a properties file.

The following steps will create a PKCS12 file on macOS or linux:

• Create a password file:

  echo -n "your password" | base64 > my_truststore_passphrase
  

• Import certificate "myCA.pem" with alias "your_server". You can omit the -storepass argument and type the password instead.

  keytool -importcert -keystore my-truststore-iris.p12 -storepass $(echo my_truststore_passphrase |base64 -d) -alias your_server -file myCA.pem
  

Invalid keystore format due to HmacPBESHA256

A new, default hashing algorithm was introduced. This means that an "old" JRE without that algorithm cannot read a PKCS12 file written by a "new" JRE that uses the new default.

The following errors might appear in the Deephaven Launcher log file:

• keytool error: java.io.IOException: Invalid keystore format
• keytool error: java.io.IOException: Integrity check failed: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available
• parseAlgParameters failed: ObjectIdentifier() -- data isn't an object ID (tag = 48)

The only remedy is to use a newer JDK version to start the Deephaven Launcher, preferably one that matches the Deephaven Server.

For reference:

• JDK-8267880 Upgrade the default PKCS12 MAC algorithm
• JDK-8267701 Revert the change to the default PKCS12 macAlgorithm and macIterationCount props for 11u/8u/7u

You can verify this problem using keytool:

$ java -version
java version "1.8.0_202"
Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)

$ keytool -list -keystore truststore-iris.p12 -storepass $(cat truststore_passphrase |base64 -d)
keytool error: java.io.IOException: Invalid keystore format

$ java -version
openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.10+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.10+9, mixed mode)

$ keytool -list -keystore truststore-iris.p12 -storepass $(cat truststore_passphrase |base64 -d)
keytool error: java.io.IOException: Integrity check failed: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available