Remote Query Dispatcher runbook

The Remote Query Dispatcher (RQD) is a critical Deephaven data management service that creates and manages worker processes. It spawns worker processes on demand for Persistent Queries, console sessions, and remote API clients. There are two types of dispatchers: db_query for read-only workers and db_merge for read-write workers.

Impact of Remote Query Dispatcher failure

Remote Query Dispatcher types

db_query (Query Dispatcher):

• Spawns read-only worker processes.
• Used for most user console sessions.
• Used for read-only Persistent Queries.
• Can read from all data sources.
• Cannot write to tables or execute data imports.

db_merge (Merge Dispatcher):

• Spawns read-write worker processes.
• Required for data import and merge operations.
• Used for Persistent Queries that write data.
• Can both read and write to tables.
• Required for DIS (Data Import Server) operations.

Typical deployment: Most nodes run both db_query and db_merge dispatchers to support all worker types.

Remote Query Dispatcher dependencies

The Remote Query Dispatcher requires:

1. Configuration Server — Must be running to access cluster configuration.
2. Authentication Server — Must be running to validate tokens and check ACLs.
3. Log Aggregator Service — Should be running (workers can fail if LAS is unavailable).
4. etcd cluster — Must be accessible (via Configuration Server).

Optional dependencies:

• Envoy — If using Envoy for ingress, RQD registers workers with Envoy.
• Table Data Cache Proxy — If configured, workers use TDCP for data access.

Checking Remote Query Dispatcher status

Check query dispatcher is running with monit:

Check merge dispatcher is running with monit:

Expected output should show status Running.

Viewing Remote Query Dispatcher logs

Query Dispatcher logs

View application log:

Tail the log to follow in real-time:

List historical log files:

Merge Dispatcher logs

View application log:

List historical log files:

Restart procedure

Restart Query Dispatcher

Restart Merge Dispatcher

Verify the restart was successful:

Monitor the log during startup:

Managing worker processes

View active workers

Workers spawned by the dispatcher are visible in process lists:

Each worker runs as a separate Java process.

View worker logs

Worker logs are stored in the dispatcher's log directory with the worker's serial ID:

View a specific worker's log:

TLS configuration

Dispatchers and workers use TLS for secure communication.

Key TLS properties:

See TLS certificates for certificate management.

Configuration files and locations

monit configuration:

• /etc/sysconfig/illumon.d/monit/db_query_server.conf
• /etc/sysconfig/illumon.d/monit/db_merge_server.conf

Property files:

• /etc/sysconfig/illumon.d/resources/iris-common.prop
• /etc/sysconfig/illumon.d/resources/iris-query-server.prop
• /etc/sysconfig/illumon.d/resources/iris-merge-server.prop

Log directories:

• /var/log/deephaven/query_server/
• /var/log/deephaven/merge_server/
• /var/log/deephaven/query_server/workers/<SERIAL>/ (individual worker logs)

Related documentation

• Remote Query Dispatcher configuration
• Worker heap size
• Dispatcher usage restrictions
• TLS certificates
• Envoy runbook
• Log Aggregator Service runbook
• PQ Controller runbook
• System processes overview