---
title: Deephaven process management
sidebar_label: Process management
---

This document provides a comprehensive guide to managing Deephaven processes across all deployment types. It covers process supervision, startup and shutdown procedures, configuration, and monitoring.

For quick reference commands, see the [Deephaven operations guide](../ops-guide/ops-guide-overview.md).

## Deployment types

Deephaven process management varies by deployment type:

| Deployment                      | Process supervision    | Primary tools      |
| ------------------------------- | ---------------------- | ------------------ |
| **Traditional** (bare metal/VM) | monit + systemd        | `dh_monit`, `iris` |
| **Podman**                      | monit (containerized)  | `dh_monit`, `iris` |
| **Kubernetes**                  | Kubernetes controllers | `kubectl`, Helm    |

## Deephaven services

A Deephaven deployment consists of multiple services, each running as a separate process:

| Process name             | Run user    | Description                                                                 |
| ------------------------ | ----------- | --------------------------------------------------------------------------- |
| `configuration_server`   | `irisadmin` | Provides configuration to other services                                    |
| `authentication_server`  | `irisadmin` | Handles user authentication                                                 |
| `iris_controller`        | `irisadmin` | Manages [Persistent Query](../../query-management/pq-overview.md) lifecycle |
| `db_query_server`        | `dbquery`   | Dispatcher for query workers                                                |
| `db_merge_server`        | `dbmerge`   | Dispatcher for merge workers                                                |
| `web_api_service`        | `dbquery`   | Serves the [Web IDE](../../interfaces/web/code-studio.md)                   |
| `log_aggregator_service` | `irisadmin` | Aggregates binary log entries                                               |
| `db_dis`                 | `dbmerge`   | Data Import Server                                                          |
| `tailer1`                | `irisadmin` | Streams data from binary logs                                               |
| `db_tdcp`                | `dbquery`   | Table Data Cache Proxy                                                      |
| `db_ltds`                | `dbquery`   | Local Table Data Server                                                     |
| `db_acl_write_server`    | `irisadmin` | ACL Write Server                                                            |

### Process dependencies

Services must start in a specific order due to dependencies:

```
etcd (no dependencies)
  └── configuration_server
        └── authentication_server
              └── All other services
```

The `dh_monit` wrapper automatically handles this sequencing.

## System users

Deephaven processes run under specific system users for security and access control:

| User       | Purpose                                 | Default name |
| ---------- | --------------------------------------- | ------------ |
| Admin user | Infrastructure services, administration | `irisadmin`  |
| Query user | Query workers, web services             | `dbquery`    |
| Merge user | Merge workers, data import              | `dbmerge`    |
| Monit user | Process supervision daemon              | `irisadmin`  |
| etcd user  | etcd database                           | `etcd`       |

You can override these defaults in `cluster.cnf`. See [Cluster configuration files](../installation/cluster-config-guide.md) for details.

## Traditional and Podman deployments

### Process supervision architecture

Traditional and Podman deployments both use monit for process supervision, but differ in how monit itself is managed:

**Traditional deployments (bare metal/VM):**

![Process supervision architecture](../../assets/sys-admin/systemd_diagram.png)

In traditional deployments, systemd manages the monit daemon. Use `systemctl` commands to start, stop, or check monit status.

**Podman deployments:**

In Podman deployments, the container entrypoint script starts monit directly — systemd is not involved. Do not use `systemctl` commands for monit management in Podman; instead, manage the container itself via `podman` commands.

### The `dh_monit` wrapper

<!--TODO: will go away in sanluis-->

Deephaven provides `dh_monit` as a wrapper around [monit](https://mmonit.com/monit/) that adds Deephaven-specific functionality:

- **Privilege handling** — Runs commands as the appropriate user without requiring root.
- **Sequenced startup** — Starts `configuration_server` and `authentication_server` first, waiting for each to be ready before continuing.
- **Blocking waits** — Optionally waits for all services to reach their target state.

#### Basic commands

```bash
# View status of all services
dh_monit summary

# View detailed status
dh_monit status

# Check specific service
dh_monit status configuration_server
```

#### Starting and stopping services

```bash
# Start all services with proper sequencing
dh_monit up

# Stop all services
dh_monit down

# Start/stop with blocking (wait for completion)
dh_monit up --block
dh_monit down --block

# Start/stop individual services
dh_monit start configuration_server
dh_monit stop authentication_server

# Restart a service
dh_monit restart iris_controller

# Start/stop all without sequencing
dh_monit start all
dh_monit stop all
```

#### Monitoring and unmonitoring

```bash
# Stop supervising a service (process keeps running)
dh_monit unmonitor configuration_server

# Resume supervision
dh_monit monitor configuration_server

# Report running vs. total processes
dh_monit report
```

### The `iris` script

The `iris` script is the actual process launcher invoked by monit. It handles:

- Java environment setup (`JAVA_HOME`, `JAVA_FLAGS`)
- Classpath construction
- Workspace and log directory creation
- Process user switching via `sudo`
- PID file management

For advanced troubleshooting, you can invoke it directly:

```bash
# Start a service with debug output
/usr/illumon/latest/bin/iris --debug start iris_controller

# Stop a service
/usr/illumon/latest/bin/iris stop iris_controller
```

### Configuration files

#### monit configuration

| Location                                | Purpose                                               |
| --------------------------------------- | ----------------------------------------------------- |
| `/etc/monitrc`                          | Main monit configuration (override via `DH_MONIT_RC`) |
| `/etc/sysconfig/illumon.d/monit/*.conf` | Per-service configuration                             |

Example service configuration (`/etc/sysconfig/illumon.d/monit/configuration_server.conf`):

```bash
check process configuration_server with pidfile /etc/deephaven/run/configuration_server.pid
    start program = "/usr/illumon/latest/bin/iris start configuration_server"
    stop program  = "/usr/illumon/latest/bin/iris stop configuration_server"
```

#### Host configuration

Process launch parameters are configured in hostconfig files:

| Location                                         | Purpose                             |
| ------------------------------------------------ | ----------------------------------- |
| `/etc/sysconfig/illumon.confs/hostconfig.system` | System defaults (installer-managed) |
| `/etc/sysconfig/illumon`                         | Custom overrides                    |

See [Java process launch configuration](../configuration/java-process-launch.md) for details.

### Enabling and disabling services

#### Temporarily disable

```bash
dh_monit stop configuration_server
dh_monit unmonitor configuration_server
```

#### Permanently disable

```bash
cd /etc/sysconfig/illumon.d/monit
mv configuration_server.conf configuration_server.conf.disabled
dh_monit reload
```

#### Re-enable

```bash
cd /etc/sysconfig/illumon.d/monit
mv configuration_server.conf.disabled configuration_server.conf
dh_monit reload
dh_monit start configuration_server
```

## Kubernetes deployments

In Kubernetes, process management is handled by Kubernetes itself rather than monit.

### Pod-based architecture

Deephaven services run as Kubernetes deployments and StatefulSets:

| Resource type    | Services                                                                                   |
| ---------------- | ------------------------------------------------------------------------------------------ |
| **Deployments**  | configuration-server, authserver, controller, query-server, merge-server, webapi, dis, las |
| **StatefulSet**  | etcd cluster                                                                               |
| **Dynamic pods** | Worker pods (created by dispatchers)                                                       |

### Managing services

#### Check status

```bash
# List all pods
kubectl get pods -n <namespace>

# Check specific deployment
kubectl get deployment configuration-server -n <namespace>

# Describe a pod for events and details
kubectl describe pod <pod-name> -n <namespace>
```

#### Restart services

```bash
# Restart a deployment (rolling restart)
kubectl rollout restart deployment/configuration-server -n <namespace>

# Check rollout status
kubectl rollout status deployment/configuration-server -n <namespace>
```

#### View logs

```bash
# View logs for a pod
kubectl logs <pod-name> -n <namespace>

# Follow logs in real-time
kubectl logs -f <pod-name> -n <namespace>

# View logs for a deployment
kubectl logs deployment/configuration-server -n <namespace>
```

#### Scale services

```bash
# Scale query servers
kubectl scale deployment/query-server --replicas=3 -n <namespace>
```

### Configuration

Kubernetes deployments are configured through Helm values rather than file system access:

```yaml
# my-values.yaml
process:
  controller:
    jvmArgsMemory: "-XX:+UseContainerSupport -XX:MaxRAMPercentage=70"

resources:
  query-server:
    requests:
      cpu: 2
      memory: 8Gi
    limits:
      memory: 16Gi
```

Apply configuration:

```bash
helm upgrade <release-name> deephaven/deephaven -f my-values.yaml -n <namespace>
```

See [Kubernetes process launch configuration](../kubernetes/kubernetes-process-launch.md) for details.

## Monitoring process status

### Traditional/Podman

```bash
# Quick status overview
dh_monit summary

# Detailed status with uptime
dh_monit status

# Watch status in real-time
watch dh_monit summary

# Check monit logs
tail -f /var/log/deephaven/monit/monit.log
```

### Kubernetes

```bash
# Pod status
kubectl get pods -n <namespace>

# Resource usage
kubectl top pods -n <namespace>

# Pod events
kubectl get events -n <namespace> --sort-by='.lastTimestamp'
```

## Troubleshooting

### Stale PID files

After an improper shutdown, stale PID files may prevent services from starting:

```bash
# Remove stale PID files
rm -f /etc/deephaven/run/*.pid

# Or add to crontab for automatic cleanup on reboot
sudo crontab -e
# Add: @reboot rm -f /etc/deephaven/run/*.pid
```

### Interactive debugging

For services that fail to start, bypass monit for interactive debugging:

```bash
# Switch to the appropriate user
sudo su - irisadmin

# Start with debug output
/usr/illumon/latest/bin/iris --debug start iris_controller
```

### Monit won't start

If monit fails to start after an improper shutdown:

**Traditional deployments:**

```bash
# Remove stale monit PID file
rm -f ~irisadmin/.monit.pid

# Restart monit
sudo systemctl restart monit
```

**Podman deployments:**

Restart the container to restart monit. The container entrypoint automatically cleans stale PID files on startup.

### Common issues

| Symptom                                                      | Likely cause              | Solution                                                 |
| ------------------------------------------------------------ | ------------------------- | -------------------------------------------------------- |
| Service cycles between "Initializing" and "Execution failed" | Dependency not running    | Check `configuration_server` and `authentication_server` |
| "Address already in use"                                     | Port conflict             | Find conflicting process: `lsof -i :<port>`              |
| "Permission denied"                                          | Wrong user or permissions | Verify file ownership and run user                       |
| Services won't start after reboot                            | Stale PID files           | Remove `/etc/deephaven/run/*.pid`                        |

See [Process startup troubleshooting](../troubleshooting/process-startup-troubleshooting.md) for more details.

## Related documentation

- [Startup / Shutdown of Deephaven processes](../ops-guide/startup-shutdown.md) — Quick reference
- [Java process launch configuration](../configuration/java-process-launch.md) — Traditional deployments
- [Kubernetes process launch configuration](../kubernetes/kubernetes-process-launch.md) — Kubernetes deployments
- [Process startup troubleshooting](../troubleshooting/process-startup-troubleshooting.md)
- [Process restart guide](../architecture/resilience-planning/process-restart-guide.md) — Maintenance recommendations
- [monit runbook](../runbooks/runbook-monit.md) — Detailed monit reference