Web API Server runbook

The Web API Server is a critical Deephaven infrastructure service that serves the Deephaven web application UI, proxies Web IDE connections to backend services, and hosts the Client Update Service (CUS) for delivering client libraries and resources. It provides the primary user-facing interface for accessing Deephaven.

For an overview of the Web API Server, see Web API Service.

Impact of Web API Server failure

LevelImpact
Sev 1 - CriticalBoth Web API clients and Deephaven Console GUI Users will be impacted. Users will not be able to use the Launcher and Deephaven Clients will not be able to receive any updates from the server.

Caution

The Web API Server proxies Web IDE (browser) connections to the authentication server, controller, and dispatcher for the life of the session — it does not merely broker an initial handshake for those clients. Core+ programmatic clients are different: they use the Web API Server only once, to retrieve connection.json at startup, then connect directly to backend services. A client configured with a local or custom connection.json does not need the Web API Server at all. This means that even if the Web API Server fails, an existing programmatic client session can keep running. However, Web IDE users will lose their session immediately. No new Web IDE sessions can start, nor can new Core+ sessions that rely on the Web API Server for connection.json.

Web API Server dependencies

The Web API Server requires:

  1. Configuration Server — Must be running to access cluster configuration.
  2. Authentication Server — Must be running to validate user logins.
  3. PQ Controller — Must be accessible for Web IDE users to view and create Code Studios and other Persistent Queries.
  4. Remote Query Dispatcher — Must be accessible to allocate worker processes.
  5. etcd cluster — Must be directly accessible; the Web API Server creates its own etcd client for service discovery and connection.json, independent of the Configuration Server.

Optional dependencies:

  • Envoy — If using Envoy, the Web API Server may register with it.

Checking Web API Server status

Check process is running with monit:

Expected output should show status Running.

Test web UI accessibility:

Viewing Web API Server logs

View application log:

Tail the log to follow in real-time:

List historical log files:

View process stdout/stderr logs:

Restart procedure

Restart the Web API Server:

Impact: Restarting the Web API Server will:

  • Disconnect users from the web UI (they'll need to refresh their browsers).
  • Interrupt the initial connection handshake for new sessions.
  • Temporarily prevent access to the Client Update Service.

Existing worker sessions will continue running but users may lose their browser connection.

Verify the restart was successful:

Monitor the log during startup:

Enabling Web API Service

The Web API Service is disabled by default in some configurations.

To enable it:

  1. Remove the .disabled extension from the config file:
  1. Reload monit configuration:
  1. Start the service:

Configuring the Web API Server

Key configuration properties (typically in iris-common.prop or a service-specific property file):

Property NamePurposeDefault Value
WebServer.tls.keystorePath to TLS keystore (p12 format)/etc/sysconfig/deephaven/auth-user/webServices-keystore.p12
WebServer.tls.passphrase.filePath to keystore password file/etc/sysconfig/deephaven/auth-user/.webapi_passphrase
RemoteQueryDispatcherParameters.hostHostname clients use to connect to the dispatcherlocalhost (set to the server's FQDN in production)

TLS certificate management

The Web API Server uses TLS to provide secure HTTPS connections.

Default self-signed certificate

The installer generates a default self-signed certificate during installation:

Keystore location: /etc/sysconfig/deephaven/auth-user/webServices-keystore.p12

Keystore password: Stored in /etc/sysconfig/deephaven/auth-user/.webapi_passphrase

Permissions: Group-owned by the shared query group (dbquerygrp by default; your configured DH_SHARED_QUERY_GROUP if customized), since workers and dispatchers also read this keystore:

Caution

The default self-signed certificate works for testing but presents security warnings in browsers. Always use a CA-signed certificate for production.

Installing a CA-signed certificate

  1. Obtain a TLS certificate signed by your trusted CA with the domain name matching your Deephaven server (e.g., myserver.mydomain.com).

  2. Backup the existing keystore:

  1. Import your CA cert and key files to the keystore:
  1. Set correct permissions. Use your deployment's configured DH_SHARED_QUERY_GROUP (dbquerygrp by default):
  1. Update dispatcher hostname to match certificate:

Edit /etc/sysconfig/illumon.d/resources/iris-common.prop:

  1. Restart Web API Service:
  1. Restart Query Dispatchers to pick up hostname change:

Client Update Service

The Client Update Service (CUS) is hosted by the Web API Server and provides client-side components to remote clients.

Accessing the Client Update Service

The CUS is available at:

Clients can download installers for:

  • Windows desktop launcher.
  • Mac desktop launcher.
  • Linux desktop launcher.

CUS-delivered resources

The CUS delivers:

  • Client JARs (Java libraries for remote clients).
  • Property files (client-side configuration).
  • Business calendars.
  • UI themes.
  • Custom client configurations.

The CUS sources these files from:

  • /usr/illumon/latest/lib/ (JARs).
  • /etc/sysconfig/illumon.d/resources/ (properties).
  • /etc/sysconfig/illumon.d/calendars/ (business calendars).

Reloading the Client Update Service

When server components change (new JARs, updated properties, etc.), reload the CUS:

Navigate to:

Or programmatically:

Note

The CUS reload does NOT refresh properties. If properties affecting CUS configuration have changed, you must restart the Web API Service.

After CUS reload:

  • Clients must exit and restart their launcher to download new components.
  • Clients that don't restart may have outdated code incompatible with the server.

Configuration files and locations

monit configuration: /etc/sysconfig/illumon.d/monit/web_api_service.conf

Property files:

  • /etc/sysconfig/illumon.d/resources/iris-common.prop
  • /etc/sysconfig/illumon.d/resources/web_api_service.prop

TLS keystore: /etc/sysconfig/deephaven/auth-user/webServices-keystore.p12

Keystore passphrase: /etc/sysconfig/deephaven/auth-user/.webapi_passphrase

Log directory: /var/log/deephaven/web_api_service/

CUS source directories:

  • /usr/illumon/latest/lib/
  • /etc/sysconfig/illumon.d/resources/
  • /etc/sysconfig/illumon.d/calendars/