Status Dashboard runbook

The Status Dashboard is an optional Deephaven infrastructure service that exposes a Prometheus-compatible metrics endpoint for external monitoring systems such as Prometheus and Grafana.

Impact of Status Dashboard failure

LevelImpact
Sev 2 - ModerateStatus dashboard data will not be available. Cluster monitoring and metrics visualization will be unavailable, but operational systems continue functioning normally.

Note

The Status Dashboard is purely observational. Its failure does not affect Deephaven's ability to process queries or serve data, only the ability to monitor the system.

Status Dashboard dependencies

The Status Dashboard requires:

  1. Configuration Server — Must be running to access cluster configuration.
  2. All monitored processes — Should be running to collect metrics from them.
  3. etcd cluster — Must be accessible (via Configuration Server).

The Status Dashboard collects metrics from all Deephaven processes via their metrics endpoints.

Checking Status Dashboard status

Check process is running with monit:

Expected output should show status Running.

Access the Prometheus metrics endpoint:

Authentication requires a valid Deephaven user who is a superuser or a member of the group specified by StatusDashboard.allowedGroups (default: dashboard).

Viewing Status Dashboard logs

View application log:

Tail the log to follow in real-time:

List historical log files:

View process stdout/stderr logs:

Restart procedure

Restart the Status Dashboard:

Impact: Restarting the Status Dashboard briefly interrupts access to monitoring dashboards but does not affect operational systems.

Verify the restart was successful:

Monitor the log during startup:

Configuring the Status Dashboard

Key configuration properties (in iris-common.prop or status_dashboard.prop). See Status Dashboard overview for full configuration details.

Property NamePurposeDefault Value
StatusDashboard.prometheus.portPort on which Prometheus data is exposed8112
StatusDashboard.prometheus.namespacePrometheus namespace for the dataDeephaven
StatusDashboard.useSslUse HTTPS for the Prometheus interfacetrue
StatusDashboard.useAuthenticationRequire authentication to access endpointtrue
StatusDashboard.allowedGroupsGroups allowed to access the endpointdashboard

Configuration files and locations

monit configuration: /etc/sysconfig/illumon.d/monit/status_dashboard.conf

Property files:

  • /etc/sysconfig/illumon.d/resources/iris-common.prop
  • /etc/sysconfig/illumon.d/resources/status_dashboard.prop

Log directory: /var/log/deephaven/status_dashboard/