Web API WebSocket Server and Web Servers

Deephaven includes several Web HTTP(S) Services:

• Client Update Service
• Web API Service
• Web IDE Service
• Query Web Server
• Merge Web Server
• DIS Web Server

Some of the web based services are disabled at install time for the following reasons:

• Bringing up a web service with a self-signed TLS certificate causes web browsers to display an untrusted error message with mitigation instructions.
• Bringing up the web services without TLS (i.e., using http cleartext) is a security risk on an untrusted network and could be an audit violation in many organizations. Furthermore, modern browsers will refuse to serve plain http web services.

It is recommended that only valid TLS certificates signed by a trusted CA be used for all web services.

However, some customers may choose to enable web services without TLS in a POC, Test or Lab environment.

Client Update Service

The Client Update Service (CUS) is hosted on the same web server as the Web API Service. It updates clients with server side components including JAR files, properties, etc. The recommended CUS setup is to use HTTPS as described for the Web API Server; while it is possible to use HTTP, this is not secure and thus not recommended. Scripts are provided with the Deephaven installation to simplify configuring the CUS; this document will help you use these scripts to configure your system, and inform you about the configuration files those scripts maintain (so you can inspect the actual settings used). When enabling the CUS with the cus_init.sh script, it is strongly recommended that a certificate be provided for a secure installation (see the Deephaven Install guide).

Important Client Update Service (CUS) files and scripts

The CUS has two sets of files that control how the server behaves. There are customer-owned files which you should update, and system-owned files which you usually should not update. In some cases, some system files can be safely edited directly, and in other cases, edits will be lost on system upgrade. We will call out which files can and cannot be edited, below.

These configuration files control both the web server and the getdown launcher application (responsible for downloading files into client machine’s workspaces).

Customer-owned CUS configuration files

• /etc/sysconfig/illumon.d/client_update_service/getdown.global
  • Contains all the "getdown settings" which a user may want to override (JVM arguments, branding files, names displayed to swing GUI users).
  • The URLs in this file are populated from configuration properties when reloading the CUS.
• /etc/sysconfig/illumon.d/client_update_service/getdown.java.11 | getdown.java.17
  • Configures JVM-version specific JVM arguments.
  • Only the file which matches the JVM version of the Deephaven installation will be used.

System-owned CUS configuration files

• /etc/sysconfig/illumon.d/client_update_service/iris/iris/getdown.txt.pre
  • A set of system-provided settings that are joined with getdown.global to form the final set of configuration used by the web server to serve files.
  • This file is replaced on every upgrade; you can override values here in getdown.global file.
• getdown.txt
  • Regenerated from getdown.global and getdown.txt.pre every time the CUS process is started or reloaded.
  • Forms the final set of configuration used by the swing GUI launcher.
  • Located in the latest cus directory inside the system's temporary directory, e.g. /tmp/cus1058969184925132201/getdown.txt

CUS Scripts

• /usr/illumon/latest/install/with_root/cus_init.sh
  • This script should be run the first time you set up a machine to run a CUS.
  • The --cus-address flag expects the next argument to be the fully qualified domain name (secure) or IP address (insecure) of the CUS server.
• /usr/illumon/latest/install/as_irisadmin/update_cus_files.sh
  • This script copies files from the latest Deephaven install at /usr/illumon/illumon-db-${DH_VERSION}/cus to /etc/sysconfig/illumon.d/client_update_service/ so that they can be used by the CUS. This set of files includes getdown.txt.pre.
  • You should always invoke update_cus_settings.sh before invoking update_cus_files.sh.
  • This script is automatically invoked whenever upgrading Deephaven.
• /usr/illumon/latest/install/as_irisadmin/update_branding_files.sh
  • Used to place customer’s custom branding resources into the correct system location.
  • The branding resource files are located in /etc/sysconfig/deephaven/branding.
  • This script is automatically invoked whenever Deephaven is upgraded, and when running update_cus_files.sh.
  • You may want to run it directly when you are updating branding files.
• /usr/illumon/latest/install/with_root/monit_init.sh
  • This script is used to tell monit that it should run a web_api_service on the current machine. To run the CUS, the web_api_service must be running.
  • The CUS will be enabled as part of the Web API Service by default. To disable, set the configuration property Webapi.server.cus.enabled to false, and restart the web_api_service.

Access Logs

The Jetty web server hosting the Client Update Service logs GET requests in the standard NCSA format. This can help you track exactly which files clients are downloading from the CUS. You can find these logs on the Web API Service host at /var/log/deephaven/web_api_service/jetty-<date>.access.log.

Client Update Service (CUS) HTTPS details

In order to serve your files over a secure TLS connection, see Web API Server.

CUS Configuration

While the getdown digest and resource files are written to the system's temporary directory, these files must not be directly edited as this directory is regenerated each time the CUS is restarted or reloaded. See Launcher and Client configuration for details on how to update the CUS configuration.

Enable the CUS over cleartext HTTP

Set the configuration property WebServer.tls.keystore to be blank; i.e., WebServer.tls.keystore=. This will tell the web server to set the protocol to http instead of https.

Next, restart the web_api_service:

sudo -u irisadmin monit restart web_api_service

Check that the appbase and ui.install_error URLs in /etc/sysconfig/illumon.d/client_update_service/getdown.global file correctly use http instead of https.

For example:

#appbase = http://WEBHOST/iris/
appbase = http://myhost.domain.com:PORT/iris/
...
#ui.install_error = http://WEBHOST/iris/error.html >ui.install_error = http://myhost.domain.com/iris/error.html

Once the Client Update Service is up and running, you can proceed to install and run the Deephaven Launcher on client desktops. The installers for Windows, Mac and Linux desktops can be downloaded from the Client Update Service on your Deephaven Server at:

http://<IRIS_SERVER_ADDRESS>/launcher

Note: IRIS_SERVER_ADDRESS is the same as WEBHOST above.

Web API Server TLS keystore

.p12 keystore file

The Web API Server's TLS keystore contains the certificate and private key of a TLS enabled service. You must keep this file private, and not distribute it or store the file in a non-secure location such as a git repository. The Web API Server's keystore file should be unique per node, with a certificate that is signed (issued) by a trusted CA.

The default self-signed key pair for the Web API Server is generated when installing the iris-config.rpm and saved to .p12 keystore file:

[-r--r----- irisadmin dbquery ]  webServices-keystore.p12

The Web Server keystore file is also protected by a unique randomly generated password stored in base64 encoded format in a read-only hidden file owned by user irisadmin and readable by dbquery group with permission set to 440:

[-r--r----- irisadmin dbquery]  .webapi_passphrase

Web API Server keystore properties

• Keystore filename: /etc/sysconfig/illumon.d/auth/keystore.webServices-keystore.p12
• Passphrase file: /db/TempFiles/irisadmin/.auth_passphrase
• Properties file: /etc/sysconfig/illumon.d/resources/iris-common.prop
• Keystore property value: WSCommServer.tls.keystore=/etc/sysconfig/illumon.d/auth/webServices-keystore.p12
• Passphrase property: WSCommServer.tls.passphrase.file=/db/TempFiles/irisadmin/.webapi_passphrase

Securing the Web API Server with your CA signed certificate

Obtain a TLS certificate signed by your trusted CA with the domain name matching the Deephaven server, e.g: myserver.mydomain.com.

Back up the existing file keystore file:

sudo cp /etc/sysconfig/illumon.d/auth/webServices-keystore.p12 \
/etc/sysconfig/illumon.d/auth/webServices-keystore.p12.ORG

Import your CA cert and key files to the Web API Service keystore file. For example:

STOREPASS=$(sudo cat /db/TempFiles/irisadmin/.webapi_passphrase | base64 --decode) openssl pkcs12 -export -in /etc/ssl/certs/tls.crt -inkey /etc/ssl/certs/tls.key -name webapi -out /etc/sysconfig/illumon.d/auth/webServices-keystore.p12 -passout pass:$STOREPASS

Set the correct permissions on the web services keystore file:

sudo chown irisadmin:dbquery /etc/sysconfig/illumon.d/auth/webServices-keystore.p12
sudo chmod 440 /etc/sysconfig/illumon.d/auth/webServices-keystore.p12

Update Query Server Prop File:

# Enable Web Sockets for Query Workers
RemoteQueryDispatcher.websocket.enabled=true

Restart Web API Service with Monit:

sudo monit restart web_api_service

Server settings

Web API server

API Server default settings:

# The keystore holding the SSL certificate to use for HTTPS.
# No key file means SSL is disabled and plaintext HTTP will be used instead.
# This is not recommended.
# tls.keystore=

WSCommServer.tls.keystore=/etc/sysconfig/illumon.d/auth/webServices-keystore.p12
WSCommServer.tls.passphrase.file=/db/TempFiles/irisadmin/.webapi_passphrase

# The port listening for incoming http/https connections
Webapi.server.port=8123

# The key file to use when signing, verifying websocket reconnection
authentication.server.reconnection.keyfile=/etc/sysconfig/illumon.d/auth/priv-authreconnect.base64.txt

# The max duration (milliseconds) for a websocket reconnect key to be valid
authentication.server.reconnection.period=600000

# The provider that this installation is using for groups (LDAP, MySQL, etc.).
# Depending on this value, additional properties will be required.
IrisDB.groupProvider=mysql

DIS web server

See Data Routing Configuration via YAML for further details.

DB ACL Write Server web server

DB ACL Write Server web server settings:

# Used in the case of DNS aliasing to infra host. By default, the FQDN of the infra-host is the only enabled origin
DbAclWriteServer.acl.origin=https://deephaven-prod-alias.example.com:8123,https://dh-prod.example.com:8123,https://full-host-name-infra-01.example.com:8123