PKI-issued certificate usage in Deephaven

Background of web server certificates

Web services and applications use certificates for a variety of operations. A certificate allows an application to do things like prove its identity, encrypt or decrypt data, and sign data to prove that it hasn't been tampered with. In order for this to work, an application must have access to all certificates in the issuing chain, and be able to trace the certificate's lineage back to an issuing authority that is trusted.

Certificate issuers are called Certificate Authorities (CAs). Platforms, such as Java and Chrome, and operating systems, such as Windows and OSX, include a trust store with bundles of well-known root CAs that they will trust by default. When a browser is used to connect to an HTTPS public Web site, the server hosting that site will typically have a Web server certificate that was issued by one of these well-known root CAs, like Verisign. Since the browser and/or OS already has a public key for Verisign (and other trusted root CAs) it can verify that the server's certificate is authentic. The server can then use its certificate to create a session key pair to use for the SSL session and encryption of data between the browser and the server.

Note

X.509 certificates are digital documents that represent a user, computer, service, or device. They contain the public key of the certificate subject and are issued by a certificate authority (CA). Certificate verification is the process of checking whether a certificate is valid and authentic. It involves four steps:

Validate the certificate chain: This step verifies that the certificate is issued by a trusted certificate authority (CA) and that it has not been revoked or expired. The chain of issuers must be valid and complete, ending at a self-signed root CA.
Verify the hostname: This step checks that the certificate subject matches the hostname of the server or client that presents it. It can use either the common name (CN) or the subject alternative name (SAN) fields of the certificate.
Establish trust: This step ensures that the root CA that signs the certificate chain is trusted by the verifier. The CA is checked against either a predefined list of trusted root CAs or a custom trust store.
Verify certificate usage: The certificate must be valid for the way it is being used. This is done by checking the certificate extensions, such as key usage and extended key usage, that specify the purposes and constraints of the certificate.

Deephaven certificate usage

A certificate and private key are provided to Deephaven as tls.crt (pem format) and tls.key files placed in the /etc/deephaven/cus_tls path on the server (or certs directory on the installation machine when initially installing). The certificate must have:

Digital signature
Key encipherment TLS web server authentication
A SAN block containing a name or names matching the URL(s) hosted by the server

Deephaven uses a Web server type certificate for two of its components: the Web API/UI, and the Client Update Service (used by the Deephaven Launcher and Java/Swing client).

For both usages, the behavior is similar to other Web applications, in that the certificate needs to provide the abilities to encrypt and decrypt data, and prove server identity. The server identity is verified using the CN (common name) and SAN (subject alternative name) sections of the certificate. The SAN is an optional section of the certificate, but has become more required in recent times. Historically, the CN (common name) of the certificate's subject was used to match the host name / URL being served, and the SAN could be added to list other names, such as short names, wild card names, or other URLs hosted by the same web server. Starting in about 2017, Web browsers stopped using the subject CN for host name matching, and began requiring the name to be present in the SAN. Deephaven recommends that the certificate provided for tls.crt have a SAN block with at least one entry matching the host name part of the URL.

All servers that host Web-accessible workers need to be included in the certificate. A wildcard certificate would accomplish that, but, if a company cannot, or would prefer not to, issue a wildcard cert, they need to have all query and merge server names listed in the SAN block.

Openssl can be used to dump the contents of a certificate file and verify that it contains the required elements:

sudo openssl x509 -in /etc/deephaven/cus-tls/tls.crt -noout -text
...
X509v3 extensions:
  X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
  X509v3 Extended Key Usage:
    TLS Web Server Authentication
...
  X509v3 Subject Alternative Name:
    DNS:<FQDN or wild card matching server URL>
...

The other core requirement for the certificate is that it be verifiable. This means that it is either issued by a well-known root CA, or that the certificate of the issuing CA was added to a trust store used by the browser or application making the connection.

For Windows systems, browsers typically use the Windows certificate store. In an Active Directory environment this certificate store also trusts Microsoft Certificate Servers that are authorized in the AD enterprise. The net result is that certificates issued by an authorized private Microsoft Certificate Server in an Active Directory enterprise will normally be verifiable and trusted by browsers running on Windows machines that are members of that AD enterprise.

For MacOS systems, browsers use the Keychain Store. A corporate IT department will typically add internal CAs to this trust store.

Java applications, like the Deephaven Launcher, use a trust store that is installed with the Java JRE or JDK by default. These applications can be configured to use trust stores provided by the operating system when they are available (i.e., Windows and MacOS).

Managing the Java truststore:

The default location of the root CA truststore included with a Java distribution is <java_home_directory>/lib/security/cacerts. The default password for this truststore is changeit.

To import a private CA certificate to the root CA truststore, use the Java keytool utility:

keytool -cacerts -trustcacerts -import -file <certificate file to import> -alias <a unique (to the truststore) alias to use for this certificate>

Note that the -keystore argument can be used (instead of -cacerts) to indicate a particular installation's truststore. This should be done in cases where there is more than one installation of Java, and the one to be updated for the Launcher is not the default one used in a shell console.

If Deephaven is configured to use certificates issued by an internal corporate CA, then that CA must be added to client system truststores. Corporate IT procedures usually handle this. If this is not the case, then the certificates will need to be added manually.

Note

The Deephaven Launcher, as of version 9.00, automatically uses Windows and macOS system truststores.

Trusting certificates on Windows platforms

On Windows, the certlm MMC plugin can be used to manage CA certificates - here, they are typically imported to the Trusted Root Certification Authorities store. Membership in a Windows domain will normally add domain level private CA certificates to this store automatically.

Trusting certificates on macOS platforms

On OSX, the Keychain Access application manages the truststore. Additional CA certificates can be imported to the local or System store. Imported certificates must be explicitly marked as trusted before they are effective.

You can also use the security command to add and list certificates in keychains. For example:

security list-keychains
security dump-keychain
security dump-trust
security add-trusted-cert -r trustRoot -k /Library/Keychains/System.keychain myCA.pem
security add-trusted-cert -r trustAsRoot -k /Library/Keychains/System.keychain intermediate.crt

Verification details for a certificate can be checked by viewing certificate details in a browser or by using curl. One item to note is that, if the SAN block is missing or incorrect, a browser will probably report ERR_CERT_COMMON_NAME_INVALID. This error is somewhat misleading, since the CN is not actually used.

curl --verbose https://<server FQDN>:8123/iriside
curl --verbose https://<server FQDN>:8443/iris/getdown.txt

These curl commands will report details of the certificate validation process in cases where the SAN block is correct and can help identify where a verification failure may be occurring. The 8123 endpoint is the default for the Web UI, while the 8443 endpoint is the default for the client update service.

Requesting and issuing certificates

Many different methods exist for requesting and issuing certificates. The most common one is through the use of a certificate signing request (csr). A tool like openssl can be used to create a csr, which is then sent to a CA. When the CA approves the request, the result is a signed certificate, which can then be installed on the server. Other CA systems have their own API or a UI through which certificates can be requested.

When requesting a certificate with a csr and an older version of openssl (pre 1.1.1), it may be necessary to include a configuration file with extra arguments (such as the SAN) in it:

basicConstraints = CA:FALSE
nsCertType = server
nsComment = "Server Certificate for Deephaven Services"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names [alt_names]
DNS.1 = <FQDN or wildcard name of server>

The DNS.1 entry could be the specific fully qualified domain name (e.g., deephaven1.mydomain.com), or could be a wildcard pattern usable on multiple machines (e.g., *.mydomain.com).

Some issuing authorities automatically add the CN name to a SAN block. One such authority is letsencrypt. In these cases it is not necessary to add the configuration file. If signing with an older version of openssl, though, the configuration file is passed as an extra argument to the signing command:

openssl x509 -req -in tls.csr -passin file:<password file> -CA <CA root directory> -CAkey <CA key pem file> -out tls.crt -CAcreateserial -days 365 -sha256 -extfile ./<configuration_file_name>

Letsencrypt is a well-known root certificate authority that provides free certificates with 90 day durations. They can be useful for temporary installations, and they also provide tools to make it easy to automate renewal of certificates.

Verisign, Digicert, and GoDaddy are three common providers of purchasable certificates.

If issuing a certificate from Microsoft Certificate Server, it may be necessary to reconfigure the CA service to support SAN blocks. The SAN details are then added using san: as a prefix in the Attributes section of the certificate request UI for an Advanced request. See Microsoft's documentation.

The certificate returned from an issuer may be in pem format or may be in der format. A file in pem format is plain text and looks something like this:

Deephaven requires a certificate file in pem format. openssl can be used to convert a der file to pem format:

openssl x509 -inform der -in certificate.cer -out tls.crt

Replacing certificates

When the main certificate used for services such as CUS, Web, Envoy, etc. expires, follow the steps below to replace it. These commands should be run as the original installing user or as a user with rights to sudo to irisadmin.

On the infra server:

Copy the new tls.crt and tls.key files to /etc/deephaven/cus-tls.

Then:

/usr/illumon/latest/install/iris_keygen.sh --force-new-web
/usr/illumon/latest/install/config_packager.sh query package

Note

Steps 2 and 3 are not needed on single-node installations.

Copy the resultant .tgz (typically dh_query.tgz) file to all query nodes.

The packager tool reports where it's creating this file. When copying it to the target nodes, it can be placed anywhere but must be in the local directory where the config_packager unpackage command is run.

On each query node:

/usr/illumon/latest/install/config_packager.sh query unpackage

If the CUS (Client Update Service) is used, or the Envoy reverse proxy is used, it will also be necessary to replace the lighttpd.pem.

As irisadmin, or the user that serves this role:

cat /etc/deephaven/cus-tls/tls.key /etc/deephaven/cus-tls/tls.crt > /etc/sysconfig/illumon.d/client_update_service/lighttpd.pem

source /usr/illumon/latest/bin/dh_users
sudo -u "${DH_ADMIN_USER}" cat /etc/deephaven/cus-tls/tls.key /etc/deephaven/cus-tls/tls.crt | \
    sudo -u "${DH_ADMIN_USER}" tee /etc/sysconfig/illumon.d/client_update_service/lighttpd.pem > /dev/null

The permissions on the lighttpd.pem file should be irisadmin:irisadmin (or the equivalent user and group) 640.

On some systems, this file may be located somewhere else, and can be found in the DH_LIGHTTPD_PEM variable in /etc/sysconfig/deephaven/cluster.cnf.

Restart at least the Web API service, the CUS (if used), Envoy (if used), and all query servers.