Persistent Query Controller runbook

The Persistent Query Controller (PQ Controller) is a core Deephaven infrastructure service that manages the lifecycle of Persistent Queries. It schedules, monitors, and maintains managed worker processes that run data operations on a defined schedule.

Impact of PQ Controller failure

High availability behavior: In multi-controller deployments, the PQ Controllers use leader election. When the leader fails, another controller automatically becomes the leader and takes over PQ management, migrating running queries when possible.

PQ Controller dependencies

The PQ Controller requires:

1. etcd cluster — Must be running and accessible for storing PQ definitions and state.
2. Configuration Server — Must be running to access PQ configurations from etcd.
3. Authentication Server — Must be running for token validation.
4. Remote Query Dispatcher — Must be available on target nodes to spawn worker processes.
5. etcd client configuration files — Located at /etc/sysconfig/deephaven/etcd/client.

Checking PQ Controller status

Check process is running with monit:

Expected output should show status Running.

Check which controller is the current leader:

Or check the log for election events:

Viewing PQ Controller logs

View application log:

Tail the log to follow in real-time:

List historical log files:

View process stdout/stderr logs:

Restart procedure

Restart the PQ Controller:

Verify the restart was successful:

Monitor the log during startup:

Managing Persistent Queries

The PQ Controller manages queries defined in etcd. Use the dhconfig pq command to manage PQ definitions.

List all Persistent Queries

View a specific Persistent Query

Stop a Persistent Query

Restart a Persistent Query

Export all Persistent Queries

Import Persistent Queries

Keep original serial IDs (skip if serial already exists):

Keep original serial IDs and overwrite existing queries:

Backup and restore

Backup procedure

Create a backup of all Persistent Query definitions:

Best practices:

• Take daily automated backups.
• Store backups separate from etcd backups (provides additional recovery options).
• Retain backups according to your data retention policy.
• Include PQ backups in disaster recovery testing.

Restore procedure

Import queries from backup:

Configuration files and locations

monit configuration: /etc/sysconfig/illumon.d/monit/iris_controller.conf

Property files:

• /etc/sysconfig/illumon.d/resources/iris-common.prop
• /etc/sysconfig/illumon.d/resources/iris_controller.prop

Log directory: /var/log/deephaven/iris_controller/

Related documentation

• Persistent Query Controller overview
• System processes overview
• Managing Persistent Queries
• Persistent Query overview
• dhconfig pq command reference
• Remote Query Dispatcher runbook
• Configuration Server runbook