Web API WebSocket Server and Web Servers

Deephaven includes several Web HTTP(S) Services:

  • Client Update Service
  • Web API Service
  • Web IDE Service
  • Query Web Server
  • Merge Web Server
  • DIS Web Server

Some of the web based services are disabled at install time for the following reasons:

  • Bringing up a web service with a self signed TLS certificate causes web browsers to display an untrusted error message with mitigation instructions.
  • Bringing up the web services without TLS (i.e., using http cleartext) is a security risk on an untrusted network and could be an audit violation in many organizations. Furthermore, modern browsers will refuse to serve plain http web services.

It is recommend that only valid TLS trusted CA signed certificates be used for all web services.

However, some customers may choose to enable web services without TLS in a POC, Test or Lab environment.

Client Update Service

The Client Update Service (CUS) is powered by lighttpd to update clients with server side components including JAR files, properties, etc. The recommended CUS setup is to use HTTPS with an x509 certificate; while it is possible to use HTTP without a certificate, this is not secure and thus not recommended. Scripts are provided with the Deephaven installation to simplify configuring the CUS; this document will help you use these scripts to configure your system, and inform you about the configuration files those scripts maintain (so you can inspect the actual settings used). When enabling the CUS with the cus_init.sh script it is strongly recommended that a certificate be provided for a secure installation (see the Deephaven Install and Upgrade guide).

Important Client Update Service (CUS) files and scripts

The CUS has two sets of files that control how the server behaves. There are customer-owned files which you should update, and system-owned files which you usually should not update. In some cases, some system files can be safely edited directly, and in other cases, edits will be lost on system upgrade. We will call out which files can and cannot be edited, below.

These configuration files control both the lighttpd web server and the getdown launcher application (responsible for downloading files into client machine’s workspaces).

Customer-owned CUS configuration files

  • /etc/sysconfig/illumon.d/client_update_service/getdown.host
    • Contains the fully qualified domain name or IP address of the machine hosting the CUS server.
    • New file since v1.20200928; upon upgrading to this version, getdown.host should be populated automatically for you.
    • On new installs, calling the cus_init.sh, this file will be populated from the flags passed to cus_init.sh.
  • /etc/sysconfig/illumon.d/client_update_service/getdown.protocol
    • Exactly the same as getdown.host, except contains either ‘http’ or ‘https’, and is used to construct the URL to the CUS server and select default port to use (443 for https and 80 for http).
  • /etc/sysconfig/illumon.d/client_update_service/getdown.port
    • Same as getdown.host and getdown.port, except contains the port number.
  • /etc/sysconfig/illumon.d/client_update_service/getdown.global
    • Contains all of the "getdown settings" which a user may want to override (JVM arguments, branding files, names displayed to swing GUI users).
    • The URLs in this file are populated from getdown.host, getdown.port, and getdown.protocol files when invoking the update_cus_settings.sh script.
  • /etc/sysconfig/illumon.d/client_update_service/getdown.java.1.8 | getdown.java.11 | getdown.java.13 | getdown.java.17
    • Configures JVM-version specific JVM arguments.
    • Only the file which matches the JVM version of the Deephaven installation will be used.

System-owned CUS configuration files

  • /var/www/lighttpd/iris/iris/getdown.txt.pre
    • A set of system-provided settings that are joined with getdown.global to form the final set of configuration used by lighttpd to serve files.
    • This file is replaced on every upgrade; you can override values here in getdown.global file.
  • /var/www/lighttpd/iris/iris/getdown.txt
    • Regenerated from getdown.global and getdown.txt.pre every time the CUS process restarts.
    • Forms the final set of configuration used by the swing GUI launcher.
  • /etc/lighttpd/client-update-service.conf
    • Controls the lighttpd web server.
    • This file is NEVER overwritten (so it’s technically safe to make edits here).
    • All URL/port/protocol settings should be automatically edited by user editing getdown.host, getdown.port, and getdown.protocol, and then running update_cus_files.sh (described below).
    • Could be considered "Customer-owned", but it is unlikely you will need to make edits here, and it is normally updated by using update_cus_files.sh.
    • Correct values are inserted for you when initially setting up a machine to run a CUS server using cus_init.sh script.
    • The server.port and ssl.engine values are automatically updated based on getdown.port and getdown.protocol files, respectively.

CUS Scripts

  • /usr/illumon/latest/install/with_root/cus_init.sh
    • This script should be run the first time you setup a machine to run a CUS server.
    • The --cus-address flag expects the next argument to be the fully qualified domain name (secure) or IP address (insecure) of the CUS server.
    • You should always pass centos to the environment flag: --environment centos
    • You must pass one of --secure or --insecure
    • When passing a --secure flag, you must provide a valid TLS certificate in /etc/deephaven/cus-tls/. The certificate must be named tls.crt in base64 x509 pem format, and the private key must be in a file named tls.key in base64 x509 pem format.
  • /usr/illumon/latest/install/as_irisadmin/update_cus_settings.sh
    • This script reads in customer settings from getdown.host, getdown.port, and getdown.protocol, and edits the getdown.global file. If all three of the getdown.* files are not present, this script will create them, choosing default values by inspecting the filesystem.
    • If an older version (pre-1.20200928) of CUS was used with HTTP, this script will setup getdown.protocol=http and getdown.port=80. If it used HTTPS, we will save getdown.protocol=https and getdown.port=443.
    • This script is called automatically by cus_init.sh, so you do NOT need to invoke it yourself during initial setup. It is also called automatically on all version upgrades, so you should only need to use this if you are changing getdown.host, getdown.port, or getdown.protocol.
  • /usr/illumon/latest/install/as_lighttpd/update_cus_files.sh
    • This script reads in customer settings from getdown.host, getdown.port, and getdown.protocol, and edits the client-update-service.conf file.
    • You should always invoke update_cus_settings.sh before invoking update_cus_files.sh.
    • You do not need to invoke this script if you manually edit /etc/lighttpd/client-update-service.conf file.
    • This script is automatically invoked whenever upgrading Deephaven, so you should take care that your getdown.port matches client-update-service.conf’s server.port and getdown.protocol should be https when ssl.engine = "enable" and http when ssl.engine = "disable".
  • /usr/illumon/latest/install/as_lighttpd/update_branding_files.sh
    • Used to place customer’s custom branding resources into the correct system location.
    • The branding resource files are located in /etc/sysconfig/deephaven/branding.
    • This script is automatically invoked whenever Deephaven is upgraded, and when running update_cus_files.sh.
    • You may want to run it directly when you are updating branding files.
  • /usr/illumon/latest/install/with_root/monit_init.sh
    • This script is used to tell monit that it should run a client_update_service on the current machine.
    • You should pass arguments: --service cus
    • This will rename the client_update_service.disabled file in /etc/sysconfig/deephaven/monit to client_update_service.conf.
    • Once finished, tell monit to pick up your changes: sudo monit reload

Client Update Service (CUS) HTTPS details

In order to serve your files over a secure TLS connection, lighttpd must have a valid x509 pem-format base64 encoded certificate containing the web certificate and private key in a single file.

The default location for this file is /etc/ssl/private/lighttpd.pem, and will be validated and constructed for you when running the cus_init.sh script. To have it created for you, the script will look in /etc/deephaven/cus-tls for files named tls.key and tls.crt (both x509 pem-format base64 files, one for the certificate, the other for the private key). Note that we use the tls.key and tls.crt keys for generating the keystore used by the web IDE, so it is preferred but optional to generate this certificate by supplying tls.key and tls.crt and running cus_init.sh.

Once you have your certificate in place, you should also verify the settings in /etc/lighttpd/client-update-service.conf are compatible with a secure CUS server.

The interesting settings are as follows:

  • ssl.pemfile = "/etc/ssl/private/lighttpd.pem"
    • You can edit this to point to anywhere you want.
  • ssl.engine = "enable"
    • Use enable for TLS; disable for plaintext.
    • This should be set for you by passing --secure to cus_init.sh, or by editing getdown.protocol to contain https and then running update_cus_files.sh.
  • server.port = 443
    • This is set for you automatically when you pass --secure to cus_init.sh.
    • To change this value DO NOT EDIT HERE; instead, edit getdown.port and rerun update_cus_files.sh (or else your changes will be lost upon your next upgrade of your Deephaven system).
    • Note that if you change this value, existing client workspaces will be effectively lost, unless you setup some port forwarding / proxy to support the previous port.

CUS Configuration

While the lighttpd digest and resource files are written to /var/www/lighttpd/iris/iris, these files must not be directly edited as this directory is regenerated each time the CUS is restarted. See [Launcher and Client configuration(./launcher-and-client-configuration.md) for details on how to update the CUS configuration.

Securing the Customer Update Service (CUS) with user authentication

By default, the CUS does not require user authentication. The CUS is powered by lighttpd and provides basic and digest authentication methods described by RFC 2617.

To enable authentication with users defined in a file, edit /etc/lighttpd/client-update-service.conf and uncomment the lines for mod_auth and mod_authn_file in the server.modules section. Also uncomment the line (further down in the file) to include "conf.d/iris-auth.conf".

Authorized users are stored in the htpasswd file:

/etc/lighttpd/illumon-cus.user

The htpasswd file contains the username and the crypt()'ed password separated by a colon. Each entry in the file is terminated by a single newline.

iris:$apr1$1xsLWNhw$.qiKafnbTpoNda/d6X77l.

You can use the htpasswd utility from the Apache distribution to manage htpasswd files. Note that not all versions of htpasswd default to use Apache's modified MD5 algorithm for passwords, which is required by lighttpd. You can force most to use MD5 by running:

htpasswd -nbm <user> <password>

Append the output of the above command to:

/etc/lighttpd/illumon-cus.user

More information on configuration options is available in lighttpd's documentation.

Disable authentication

In order to disable the username/password requirement, edit:

/etc/lighttpd/client-update-service.conf

Change (comment out) the following lines:

    include "conf.d/iris-auth.conf"
    include "conf.d/iris-access_log.conf"

to read:

    #include "conf.d/iris-auth.conf"
    #include "conf.d/iris-access_log.conf"

Restart the CUS with monit:

sudo monit restart client_update_service

Enable the CUS on cleartext HTTP port 80

Warning

This is not recommended. Only do this for testing only on a trusted private network.

On the Deephaven Server, edit:

/etc/sysconfig/illumon.d/client_update_service/getdown.protocol

Set the protocol to http instead of https.

Next, rerun:

/usr/illumon/latest/install/as_irisadmin/update_cus_settings.sh

Then rerun:

/usr/illumon/latest/install/as_lighttpd/update_cus_files.sh

Check that the appbase and ui.install_error URLs in /etc/sysconfig/illumon.d/client_update_service/getdown.global file correctly use http instead of https.

For example:

#appbase = http://WEBHOST/iris/
appbase = http://myhost.domain.com/iris/
...
#ui.install_error = http://WEBHOST/iris/error.html >ui.install_error = http://myhost.domain.com/iris/error.html

Next, verify that the server.port and ssl.engine settings in /etc/lighttpd/client-update-service.conf are correct. For insecure http, the server.port should be 80 and ssl.engine should be disable. It’s fine to use a different port than 80, just be sure to include your port when starting the Deephaven client launcher. If in doubt, copy-paste the appbase property from getdown.global to ensure your URL is correct.

Restart the CUS:

sudo monit restart client_update_service

Check the status of the getdown service:

sudo monit status client_update_service

Once the Client Update Service is up and running, you can proceed to install and run the Launcher on client desktops. The installers for Windows, Mac and Linux desktops can be downloaded from the Client Update Service on your Deephaven Server at:

http://<IRIS_SERVER_ADDRESS>/iris

Note: IRIS_SERVER_ADDRESS is the same as WEBHOST above.

Web API Server TLS keystore

.p12 keystore file

The Web API Server's TLS keystore contains the certificate and private key of a TLS enabled service. You must keep this file private, and not distribute it or store the file in a non-secure location such as a git repository. The Web API Server's keystore file should be unique per node, with a certificate that is signed (issued) by a trusted CA.

The default self signed key pair for the Web API Server is generated when installing the iris-config.rpm and saved to .p12 keystore file:

[-r--r----- irisadmin dbquery ]  webServices-keystore.p12

The Web Server keystore file is also protected by a unique randomly generated password stored in base64 encoded format in a read-only hidden file owned by user irisadmin and readable by dbquery group with permission set to 440:

[-r--r----- irisadmin dbquery]  .webapi_passphrase

Web API Server keystore properties

  • Keystore filename: /etc/sysconfig/illumon.d/auth/keystore.webServices-keystore.p12
  • Passphrase file: /db/TempFiles/irisadmin/.auth_passphrase
  • Properties file: /etc/sysconfig/illumon.d/resources/iris-common.prop
  • Keystore property value: WSCommServer.tls.keystore=/etc/sysconfig/illumon.d/auth/webServices-keystore.p12
  • Passphrase property: WSCommServer.tls.passphrase.file=/db/TempFiles/irisadmin/.webapi_passphrase

Securing the Web API Server with your CA signed certificate

Obtain a TLS certificate signed by your trusted CA with the domain name matching the Deephaven server, e.g: myserver.mydomain.com.

Backup the existing file keystore file:

sudo cp /etc/sysconfig/illumon.d/auth/webServices-keystore.p12 \
/etc/sysconfig/illumon.d/auth/webServices-keystore.p12.ORG

Import your CA cert and key files to the Web API Service keystore file. For example:

STOREPASS=$(sudo cat /db/TempFiles/irisadmin/.webapi_passphrase | base64 --decode) openssl pkcs12 -export -in /etc/ssl/certs/tls.crt -inkey /etc/ssl/certs/tls.key -name webapi -out /etc/sysconfig/illumon.d/auth/webServices-keystore.p12 -passout pass:$STOREPASS

Set the correct permissions on the web services keystore file:

sudo chown irisadmin:dbquery /etc/sysconfig/illumon.d/auth/webServices-keystore.p12
sudo chmod 440 /etc/sysconfig/illumon.d/auth/webServices-keystore.p12

Update Query Server Prop File:

/etc/sysconfig/illumon.d/resources/iris-common.prop:
# Enable Web Sockets for Query Workers
RemoteQueryDispatcher.websocket.enabled=true

Restart Web API Service with Monit:

sudo monit restart web_api_service

Server settings

Web API server

API Server properties file: /etc/sysconfig/illumon.d/resources/iris-common.prop

API Server default settings:

# The keystore holding the SSL certificate to use for HTTPS.
# No key file means SSL is disabled and plaintext HTTP will be used instead.
This is not recommended.
# tls.keystore=

WSCommServer.tls.keystore=/etc/sysconfig/illumon.d/auth/webServices-keystore.p12
WSCommServer.tls.passphrase.file=/db/TempFiles/irisadmin/.webapi_passphrase

# The port listening for incoming http/https connections
Webapi.server.port=8123

# The key file to use when signing, verifying websocket reconnection
authentication.server.reconnection.keyfile=/etc/sysconfig/illumon.d/auth/priv-authreconnect.base64.txt

# The max duration (milliseconds) for a websocket reconnect key to be valid authentication.server.reconnection.period=600000 # The provider that this installation is using for groups (LDAP, MySQL, etc.).
# Depending on this value, additional properties will be required. IrisDB.groupProvider=mysql

DIS web server

DIS server property file: /etc/sysconfig/illumon.d/resources/iris-common.prop

See Data Routing Configuration via YAML for further details.

Query web server

Query server properties file: /etc/sysconfig/illumon.d/resources/iris-common.prop

Query web server settings:

# Query web server
RemoteQueryDispatcher.webserver.enabled=true
RemoteQueryDispatcher.webserver.port=8084

# HTTP Basic Auth, SSL must be enabled when authentication is required RemoteQueryDispatcher.webserver.authenticationRequired=true
RemoteQueryDispatcher.webserver.sslRequired=true

WSCommServer.tls.keystore=/etc/sysconfig/illumon.d/auth/webServices-keystore.p12
WSCommServer.tls.passphrase.file=/db/TempFiles/irisadmin/.webapi_passphrase

Merge web server

Merge server property file: /etc/sysconfig/illumon.d/resources/iris-common.prop

Query web server settings:

# Query web server
RemoteQueryDispatcher.webserver.enabled=true
RemoteQueryDispatcher.webserver.port=8084

# HTTP Basic Auth, SSL must be enabled when authentication is required RemoteQueryDispatcher.webserver.authenticationRequired=true RemoteQueryDispatcher.webserver.sslRequired=true

WSCommServer.tls.keystore=/etc/sysconfig/illumon.d/auth/webServices-keystore.p12 WSCommServer.tls.passphrase.file=/db/TempFiles/irisadmin/.webapi_passphrase