PKI-issued certificate usage in Deephaven

Background of web server certificates

Web services and applications use certificates for a variety of operations. A certificate allows an application to do things like prove its identity, encrypt or decrypt data, and sign data to prove that it hasn't been tampered with. In order for this to work, a certificate being presented to an application must be able to trace its lineage back to an issuing authority that is trusted by the application. Certificates can be verified using public keys stored in a trust store.

Certificate issuers are called CAs (certificate authorities). Platforms, such as Java and Chrome, and operating systems, such as Windows and OSX, include a trust store with bundles of well-known root CAs that they will trust by default. When a browser is used to connect to an HTTPS public Web site, the server hosting that site will typically have a Web server certificate that was issued by one of these well-known root CAs, like Verisign. Since the browser and/or OS already has a public key for Verisign (and other trusted root CAs) it can verify that the server's certificate is authentic. The server can then use its certificate to create a session key pair to use for the SSL session and encryption of data between the browser and the server.

Deephaven certificate usage

A certificate is provided with its private key to Deephaven as a tls.crt (pem format) and tls.key file placed in the /etc/deephaven/cus_tls path on the server (or certs directory on the installation machine when initially installing). The certificate must have:

• Digital signature
• Key encipherment TLS web server authentication
• A SAN block containing a name or names matching the URL(s) hosted by the server

Deephaven uses a Web server type certificate for two of its components: the Web API/UI, and the Launcher/Client Update Service (used for the Java/Swing client).

For both usages, the behavior is similar to other Web applications, in that the certificate needs to provide the abilities to encrypt and decrypt data, and prove server identity. The detail about proving server identity is accomplished using the SAN (subject alternative name) section of the certificate. This is an optional section of the certificate, but has become more required in recent times. Historically, the CN (common name) of the certificate's subject was used to match the host name / URL being served, and the SAN could be added to list other names, such as short names, wild card names, or other URLs hosted by the same web server. Starting in about 2017, Web browsers stopped using the subject CN for host name matching, and began requiring the name to be present in the SAN. As such, Deephaven requires that the certificate provided for tls.crt have a SAN block with at least one entry matching the host name part of the URL.

All servers that host Web-accessible workers need to be included in the certificate. A wildcard certificate would accomplish that, but, if a company cannot, or would prefer not to, issue a wildcard cert, they need to have all query and merge server names listed in the SAN block.

Checking a certificate file with openssl

Openssl can be used to dump the contents of a certificate file and verify that it contains the required elements:

sudo openssl x509 -in /etc/deephaven/cus-tls/tls.crt -noout -text
  ...
  Signature Algorithm: sha256WithRSAEncryption
      Issuer: C=US, O=Let's Encrypt, CN=R3
      Validity
          Not Before: Jan 30 14:40:59 2023 GMT
          Not After : Apr 30 14:40:58 2023 GMT
  ...
  X509v3 extensions:
    X509v3 Key Usage: critical
      Digital Signature, Key Encipherment
    X509v3 Extended Key Usage:
      TLS Web Server Authentication
  ...
    X509v3 Subject Alternative Name:
      DNS:<FQDN or wild card matching server URL>
  ...

Certificate chain requirements for verification

The other core requirement for the certificate is that it be verifiable. This means that it is either issued by a well-known root CA, or that the certificate of the private CA that issued it was added to the trust stores of the application or browser that is connecting to the server.

For Windows systems, browsers typically use the Windows certificate store. In an Active Directory environment this certificate store also trusts Microsoft Certificate Servers that are authorized in the AD enterprise. The net result is that certificates issued by an authorized private Microsoft Certificate Server in an Active Directory enterprise will normally be verifiable and trusted by browsers running on Windows machines that are members of that AD enterprise.

Java applications, like the Deephaven Launcher, use a trust store that is installed with the Java JRE or JDK. This store can be managed with the Java keytool command. If a certificate has been issued by a private CA (including Microsoft Certificate Server), it may be necessary to import the CA's certificate, and possibly the certificates of intermediate CAs in the signing chain for the server certificate. The Launcher uses HTTPS to communicate with the Deephaven server when checking for client updates and downloading the packages needed for the Java client. If the certificate used by the client update service (/etc/sysconfig/illumon.d/client_update_service/lighttpd.pem) cannot be verified by the Launcher, the Launcher will be unable to launch the Java client.

The paragraph above states that the trust store may need to be modified because on Windows and OSX there are other options. On Linux, the trust store will always need to have CA certificates added if the server certificate was issued by a private CA.

The default location of the root CA truststore included with a Java distribution is <java_home_directory>/lib/security/cacerts. The default password for this truststore is changeit.

To import a private CA certificate to the root CA truststore, use the Java keytool utility:

keytool -cacerts -trustcacerts -import -file <certificate file to import> -alias <a unique (to the truststore) alias to use for this certificate>

Note that the -keystore argument can be used (instead of -cacerts) to indicate a particular installation's truststore. This should be done in cases where there is more than one installation of Java, and the one to be updated for the Launcher is not the default one used in a shell console.

Windows and OSX provide shared certificate stores which can be used by all applications running on the client machine.

On Windows, the certlm MMC plugin can be used to manage CA certificates - here, they are typically imported to the Trusted Root Certification Authorities store. Membership in a Windows domain will normally add domain level private CA certificates to this store automatically.

On OSX, the Keychain Access application is used to manage the trust store, and additional CA certificates are typically imported to the System store, and then a second step is needed of opening the certificate and explicitly marking it as "Always Trust".

On Windows, versions 9.00 and later of the Deephaven Launcher will automatically try to use the Windows-ROOT store, and Windows-MY stores, in addition to the default Java truststore, to find trusted signing authorities for Deephaven server certificates. To have an older version of the Launcher use the Windows-ROOT certificate store:

1. Right click on the Launcher shortcut icon and select Properties.
2. Add -Djavax.net.ssl.trustStoreType=Windows-ROOT before the -cp argument.

On OSX, versions 9.00 and later of the Deephaven Launcher will automatically try to use Keychain Access, in addition to the default Java truststore, to find trusted signing authorities for Deephaven server certificates. To have an older version of the Launcher use Keychain Access:

1. Find and edit the DeephavenLauncher.sh script.
2. Towards the bottom, make the highlighted changes:

if [ `uname` = "Darwin" ]; then
    MAC_OPTS="-Xdock:icon=iris_desktop.icns"
    MAC_OPT2="-Xdock:name=Deephaven Launcher"
    MAC_OPT3="-Djavax.net.ssl.trustStoreType=KeychainStore"
   (cd $DEVROOT/lib; $JAVA_EXE "$LOGDIR_OPT" $MAC_OPTS "$MAC_OPT2" $MAC_OPT3 -cp "${bootstrapjar}:*"
com.illumon.iris.bootstrap.IrisConfigurationLauncher $*)

curl --verbose https://<server FQDN>:8123/iriside
curl --verbose https://<server FQDN>:8443/iris/getdown.txt

These curl commands will report details of the certificate validation process in cases where the SAN block is correct and can help identify where a verification failure may be occurring. The 8123 endpoint is the default for the Web UI, while the 8443 endpoint is the default for the client update service.

Requesting and issuing certificates

Many different methods exist for requesting and issuing certificates. The most common one is through the use of a certificate signing request (csr). A tool like openssl can be used to create a csr, which is then sent to a CA. When the CA approves the request, the result is a signed certificate, which can then be installed on the server. Other CA systems have their own API or a UI through which certificates can be requested.

When requesting a certificate with a csr and an older version of openssl (pre 1.1.1), it may be necessary to include a configuration file with extra arguments (such as the SAN) in it:

basicConstraints = CA:FALSE
nsCertType = server
nsComment = "Server Certificate for Deephaven Services"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names [alt_names]
DNS.1 = <FQDN or wildcard name of server>

The DNS.1 entry could be the specific fully qualified domain name (e.g., deephaven1.mydomain.com), or could be a wildcard pattern usable on multiple machines (e.g., *.mydomain.com).

Some issuing authorities automatically add the CN name to a SAN block. One such authority is letsencrypt. In these cases it is not necessary to add the configuration file. If signing with an older version of openssl, though, the configuration file is passed as an extra argument to the signing command:

openssl x509 -req -in tls.csr -passin file:<password file> -CA <CA root directory> -CAkey <CA key pem file> -out tls.crt -CAcreateserial -days 365 -sha256 -extfile ./<configuration_file_name>

Letsencrypt is a well-known root certificate authority that provides free certificates with 90 day durations. They can be useful for temporary installations, and they also provide tools to make it easy to automate renewal of certificates.

Verisign, Digicert, and GoDaddy are three common providers of purchasable certificates.

If issuing a certificate from Microsoft Certificate Server, it may be necessary to reconfigure the CA service to support SAN blocks. The SAN details are then added using san: as a prefix in the Attributes section of the certificate request UI for an Advanced request. See Microsoft's documentation.

The certificate returned from an issuer may be in pem format or may be in der format. If the file contains readable text, like this:

-----BEGIN CERTIFICATE----- MIIFKTCCBBGgAwIBAgISA7pCJgJoh07Xh3WcG+yw2xA5MA0GCSqGSIb3DQEBCwUA MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD EwJSMzAeFw0yMTA4MDQxNTQyNDZaFw0yMTExMDIxNTQyNDRaMBwxGjAYBgNVBAMM ESouaW50LmlsbHVtb24uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC AQEA4G89gYSrnW5F7kvYNyE7Lh/6AVznMN07O+VLDMpgHcCMCcuomRvxI8bOTMYP ...

It is already in pem format and can be used by Deephaven as-is. If it does not contain readable text, openssl can be used to convert it to pem format.

openssl x509 -inform der -in certificate.cer -out tls.crt

Replacing certificates

When the main certificate used for services such as CUS, Web, Envoy, etc. expires, follow the steps below to replace it. These commands should be run as the original installing user or as a user with rights to sudo to iris_admin:

1. On the infra server:

• Copy the new tls.crt and tls.key files to /etc/deephaven/cus-tls.
• Then:

   /usr/illumon/latest/install/iris_keygen.sh --force-new-web
   /usr/illumon/latest/install/config_packager.sh query package
  

2. Copy the resultant .tgz (typically dh_query.tgz) file to all query nodes.

The packager tool reports where it's creating this file. When copying it to the target nodes, it can be placed anywhere but must be in the local directory where the config_packager unpackage command is run.

3. On each query node:

   /usr/illumon/latest/install/config_packager.sh query unpackage
   

4. Restart at least the Web API service, the CUS (if used), and all query servers.