monit runbook

monit is a third-party process supervision tool used to manage Deephaven services on traditional installations and Podman deployments. It monitors processes, automatically restarts failed services, and provides a control interface for starting, stopping, and restarting Deephaven processes. monit is not used in Kubernetes deployments, where pod management is handled by Kubernetes itself.

Impact of monit failure

Level	Impact
Sev 2 - Moderate	Process supervision and automatic restart capabilities are lost. Services continue running but will not automatically recover from failures. Manual intervention required to start or restart processes.

Note

monit failure does not stop running Deephaven processes. They continue functioning until manually stopped or until they fail, at which point they will not automatically restart.

monit dependencies

monit has no dependencies on other services. It is one of the first processes to start on a Deephaven host.

monit manages:

All Deephaven Java processes (Configuration Server, Authentication Server, etc.).
etcd (if installed as part of Deephaven).
Other custom services defined in monit configuration.

Checking monit status

Check monit service is running:

sudo systemctl status monit

Expected output should show active (running).

View summary of all monitored processes:

dh_monit summary

View detailed status:

dh_monit status

Check specific process:

dh_monit status configuration_server

Viewing monit logs

View monit log:

cat /var/log/deephaven/monit/monit.log

Tail the log to follow in real-time:

tail -f /var/log/deephaven/monit/monit.log

View systemd journal for monit:

sudo journalctl -u monit -f

Restart procedure

Restart monit:

sudo systemctl restart monit

Caution

Restarting monit does NOT restart monitored processes. They continue running but are temporarily unmonitored during the restart.

Verify the restart was successful:

sudo systemctl status monit

Check all monitored processes are still tracked:

dh_monit summary

Reload monit configuration

After modifying monit configuration files:

dh_monit reload

This reloads configuration without restarting monit or any monitored processes.

Managing processes with monit

Start a process

dh_monit start configuration_server

Stop a process

dh_monit stop configuration_server

Restart a process

dh_monit restart configuration_server

Start all processes

dh_monit start all

Stop monitoring a process (without stopping it)

dh_monit unmonitor configuration_server

Resume monitoring

dh_monit monitor configuration_server

monit configuration

monit configuration consists of:

Main configuration: /etc/monitrc (overridable via DH_MONIT_RC environment variable)

Deephaven-specific configuration: /etc/sysconfig/illumon.d/monit/

Main monit configuration

Key settings in main configuration file:

# Check interval
set daemon 60  # Check every 60 seconds

# Log file
set log /var/log/deephaven/monit/monit.log

# State file
set statefile /var/lib/deephaven/monit/monit.state

# Include Deephaven process configurations
include /etc/sysconfig/illumon.d/monit/*.conf

Process configuration files

Each Deephaven process has its own configuration file:

Example: /etc/sysconfig/illumon.d/monit/configuration_server.conf

check process configuration_server
    with pidfile /var/run/deephaven/configuration_server.pid
    start program = "/usr/illumon/latest/bin/start-service configuration_server"
        as uid irisadmin gid dbquery
    stop program = "/usr/illumon/latest/bin/stop-service configuration_server"
        as uid irisadmin gid dbquery
    if failed port 22023 then restart
    if 5 restarts within 5 cycles then timeout

Key elements:

pidfile — Location of process PID file.
start program — Command to start process.
stop program — Command to stop process.
health checks — Port checks, resource limits.
restart policy — When and how to restart.

Process startup order

monit can enforce process startup order through dependencies:

Example dependency chain:

etcd (no dependencies).
Configuration Server (depends on etcd).
Authentication Server (depends on Configuration Server).
Other services (depend on Configuration Server, Authentication Server).

Configuration:

check process authentication_server
    depends on configuration_server
    start program = "..."

monit will not start a process until its dependencies are running.

Disabling and enabling processes

Temporarily disable a service

Prevent monit from starting a process:

# Stop and unmonitor
dh_monit stop configuration_server
dh_monit unmonitor configuration_server

Permanently disable a service

Rename configuration file:

cd /etc/sysconfig/illumon.d/monit
mv configuration_server.conf configuration_server.conf.disabled
dh_monit reload

Re-enable a service

Rename configuration file back:

cd /etc/sysconfig/illumon.d/monit
mv configuration_server.conf.disabled configuration_server.conf
dh_monit reload
dh_monit start configuration_server

monit user and permissions

monit runs as root but executes process commands as specified users:

monit daemon: Runs as root (required for process supervision).

Process management: Commands run as irisadmin user.

Deephaven processes: Run as various users (irisadmin, dbquery, dbmerge).

Control commands: Should be run as irisadmin:

dh_monit status

Testing monit configuration

Before reloading or restarting:

# Test configuration syntax
sudo monit -t

# Verbose test
sudo monit -t -v

Expected output: Control file syntax OK

Monitoring monit itself

Ensure monit is always running:

systemd management

Enable monit to start on boot:

sudo systemctl enable monit

Check if enabled:

sudo systemctl is-enabled monit

Configuration files and locations

systemd service: Managed by systemd

Service control: systemctl {start|stop|restart|status} monit