---
id: runbook-remote-query-dispatcher
title: Remote Query Dispatcher runbook
---

The [Remote Query Dispatcher (RQD)](../pq-controller/dispatcher.md) is a critical Deephaven data management service that creates and manages worker processes. It spawns worker processes on demand for Persistent Queries, console sessions, and remote API clients. There are two types of dispatchers: `db_query` for read-only workers and `db_merge` for read-write workers.

## Impact of Remote Query Dispatcher failure

| Level            | Impact                                                                                                                                                  |
| :--------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Sev 1 - Critical | Any running query workers will terminate, and new ones cannot be started. This includes all running persistent queries as well as interactive consoles. |

> [!CAUTION]
> Worker process termination affects all users connected through that dispatcher, causing active console sessions and running queries to fail immediately.

## Remote Query Dispatcher types

**`db_query` (Query Dispatcher):**

- Spawns read-only worker processes.
- Used for most user console sessions.
- Used for read-only Persistent Queries.
- Can read from all data sources.
- Cannot write to tables or execute data imports.

**`db_merge` (Merge Dispatcher):**

- Spawns read-write worker processes.
- Required for data import and merge operations.
- Used for Persistent Queries that write data.
- Can both read and write to tables.
- Required for DIS (Data Import Server) operations.

**Typical deployment:** Most nodes run both `db_query` and `db_merge` dispatchers to support all worker types.

## Remote Query Dispatcher dependencies

The Remote Query Dispatcher requires:

1. **Configuration Server** — Must be running to access cluster configuration.
2. **Authentication Server** — Must be running to validate tokens and check ACLs.
3. **Log Aggregator Service** — Should be running (workers can fail if LAS is unavailable).
4. **etcd cluster** — Must be accessible (via Configuration Server).

**Optional dependencies:**

- **Envoy** — If using Envoy for ingress, RQD registers workers with Envoy.
- **Table Data Cache Proxy** — If configured, workers use TDCP for data access.

## Checking Remote Query Dispatcher status

Check query dispatcher is running with monit:

```bash
dh_monit status db_query_server
```

Check merge dispatcher is running with monit:

```bash
dh_monit status db_merge_server
```

Expected output should show status `Running`.

## Viewing Remote Query Dispatcher logs

### Query Dispatcher logs

View application log:

```bash
cat /var/log/deephaven/query_server/RemoteQueryDispatcher.log.current
```

Tail the log to follow in real-time:

```bash
tail -f /var/log/deephaven/query_server/RemoteQueryDispatcher.log.current
```

List historical log files:

```bash
ls -ltr /var/log/deephaven/query_server/db_query_server.log.????-??-??
```

### Merge Dispatcher logs

View application log:

```bash
cat /var/log/deephaven/merge_server/RemoteQueryDispatcher.log.current
```

List historical log files:

```bash
ls -ltr /var/log/deephaven/merge_server/db_merge_server.log.????-??-??
```

## Restart procedure

### Restart Query Dispatcher

```bash
dh_monit restart db_query_server
```

### Restart Merge Dispatcher

```bash
dh_monit restart db_merge_server
```

> [!WARNING]
> Restarting a dispatcher terminates all worker processes spawned by that dispatcher, affecting all users and queries using those workers.

Verify the restart was successful:

```bash
dh_monit status db_query_server
dh_monit status db_merge_server
```

Monitor the log during startup:

```bash
tail -f /var/log/deephaven/query_server/RemoteQueryDispatcher.log.current
```

## Managing worker processes

### View active workers

Workers spawned by the dispatcher are visible in process lists:

```bash
ps aux | grep DbWorker
```

Each worker runs as a separate Java process.

### View worker logs

Worker logs are stored in the dispatcher's log directory with the worker's serial ID:

```bash
ls -ltr /var/log/deephaven/query_server/workers/
```

View a specific worker's log:

```bash
cat /var/log/deephaven/query_server/workers/<WORKER_SERIAL>/DbWorker.log.current
```

## TLS configuration

Dispatchers and workers use TLS for secure communication.

**Key TLS properties:**

```properties
RemoteQueryDispatcher.tls.keystore=/etc/sysconfig/illumon.d/auth-user/webServices-keystore.p12
RemoteQueryDispatcher.tls.passphrase.file=/etc/sysconfig/deephaven/auth-user/.webapi_passphrase
```

See [TLS certificates](../configuration/pki-certificate.md) for certificate management.

## Configuration files and locations

**monit configuration:**

- `/etc/sysconfig/illumon.d/monit/db_query_server.conf`
- `/etc/sysconfig/illumon.d/monit/db_merge_server.conf`

**Property files:**

- `/etc/sysconfig/illumon.d/resources/iris-common.prop`
- `/etc/sysconfig/illumon.d/resources/iris-query-server.prop`
- `/etc/sysconfig/illumon.d/resources/iris-merge-server.prop`

**Log directories:**

- `/var/log/deephaven/query_server/`
- `/var/log/deephaven/merge_server/`
- `/var/log/deephaven/query_server/workers/<SERIAL>/` (individual worker logs)

## Related documentation

- [Remote Query Dispatcher configuration](../pq-controller/dispatcher.md)
- [Worker heap size](../pq-controller/worker-heap-size.md)
- [Dispatcher usage restrictions](../pq-controller/dispatcher-usage-restrictions.md)
- [TLS certificates](../configuration/pki-certificate.md)
- [Envoy runbook](runbook-envoy.md)
- [Log Aggregator Service runbook](runbook-log-aggregator-service.md)
- [PQ Controller runbook](runbook-pq-controller.md)
- [System processes overview](../architecture/architecture-overview.md)