---
id: runbook-pq-controller
title: Persistent Query Controller runbook
---

The [Persistent Query Controller](../core-components/controller.md) (PQ Controller) is a core Deephaven infrastructure service that manages the lifecycle of Persistent Queries. It schedules, monitors, and maintains managed worker processes that run data operations on a defined schedule.

## Impact of PQ Controller failure

| Level            | Impact                                                                                                                                                                                                                                                                                                                     |
| :--------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Sev 1 - Critical | When configured to run with multiple controllers, running Core+ queries are migrated to a running controller. Core+ queries that are not yet running are terminated. All Legacy queries, including WebClientData, are terminated. Until the WebClientData reinitializes, users are not able to load the Deephaven console. |

**High availability behavior:** In multi-controller deployments, the PQ Controllers use leader election. When the leader fails, another controller automatically becomes the leader and takes over PQ management, migrating running queries when possible.

## PQ Controller dependencies

The PQ Controller requires:

1. **etcd cluster** — Must be running and accessible for storing PQ definitions and state.
2. **Configuration Server** — Must be running to access PQ configurations from etcd.
3. **Authentication Server** — Must be running for token validation.
4. **Remote Query Dispatcher** — Must be available on target nodes to spawn worker processes.
5. **etcd client configuration files** — Located at `/etc/sysconfig/deephaven/etcd/client`.

## Checking PQ Controller status

Check process is running with monit:

```bash
dh_monit status iris_controller
```

Expected output should show status `Running`.

Check which controller is the current leader:

```bash
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig pq leader
```

Or check the log for election events:

```bash
grep "Elected as leader" /var/log/deephaven/iris_controller/PersistentQueryController.log.current
```

## Viewing PQ Controller logs

View application log:

```bash
cat /var/log/deephaven/iris_controller/PersistentQueryController.log.current
```

Tail the log to follow in real-time:

```bash
tail -f /var/log/deephaven/iris_controller/PersistentQueryController.log.current
```

List historical log files:

```bash
ls -ltr /var/log/deephaven/iris_controller/iris_controller.log.????-??-??
```

View process stdout/stderr logs:

```bash
cat /var/log/deephaven/iris_controller/iris_controller.log.$(date +%Y-%m-%d)
```

## Restart procedure

Restart the PQ Controller:

```bash
dh_monit restart iris_controller
```

Verify the restart was successful:

```bash
dh_monit status iris_controller
```

Monitor the log during startup:

```bash
tail -f /var/log/deephaven/iris_controller/PersistentQueryController.log.current
```

## Managing Persistent Queries

The PQ Controller manages queries defined in etcd. Use the `dhconfig pq` command to manage PQ definitions.

### List all Persistent Queries

```bash
sudo /usr/illumon/latest/bin/dhconfig pq status
```

### View a specific Persistent Query

```bash
sudo /usr/illumon/latest/bin/dhconfig pq status --serial <SERIAL_ID>
```

### Stop a Persistent Query

```bash
sudo /usr/illumon/latest/bin/dhconfig pq stop --serial <SERIAL_ID>
```

### Restart a Persistent Query

```bash
sudo /usr/illumon/latest/bin/dhconfig pq restart --serial <SERIAL_ID>
```

### Export all Persistent Queries

```bash
sudo /usr/illumon/latest/bin/dhconfig pq export --file /tmp/PersistentQueryBackup.xml
```

### Import Persistent Queries

```bash
sudo /usr/illumon/latest/bin/dhconfig pq import --file /tmp/PersistentQueryBackup.xml
```

Keep original serial IDs (skip if serial already exists):

```bash
sudo /usr/illumon/latest/bin/dhconfig pq import --file /tmp/PersistentQueryBackup.xml --retain-serial
```

Keep original serial IDs and overwrite existing queries:

```bash
sudo /usr/illumon/latest/bin/dhconfig pq import --file /tmp/PersistentQueryBackup.xml --retain-serial --replace-existing
```

## Backup and restore

### Backup procedure

Create a backup of all Persistent Query definitions:

```bash
sudo /usr/illumon/latest/bin/dhconfig pq export --file /backup/pq-backup-$(date +%Y%m%d-%H%M%S).xml
```

**Best practices:**

- Take daily automated backups.
- Store backups separate from etcd backups (provides additional recovery options).
- Retain backups according to your data retention policy.
- Include PQ backups in disaster recovery testing.

### Restore procedure

Import queries from backup:

```bash
sudo /usr/illumon/latest/bin/dhconfig pq import --file /backup/pq-backup-20250101-120000.xml
```

> [!NOTE]
> The PQ Controller automatically detects new or updated PQ definitions and reconciles the running state.

## Configuration files and locations

**monit configuration:** `/etc/sysconfig/illumon.d/monit/iris_controller.conf`

**Property files:**

- `/etc/sysconfig/illumon.d/resources/iris-common.prop`
- `/etc/sysconfig/illumon.d/resources/iris_controller.prop`

**Log directory:** `/var/log/deephaven/iris_controller/`

## Related documentation

- [Persistent Query Controller overview](../core-components/controller.md)
- [System processes overview](../architecture/architecture-overview.md)
- [Managing Persistent Queries](../architecture/resilience-planning/managing-persistent-queries.md)
- [Persistent Query overview](../../query-management/pq-overview.md)
- [dhconfig pq command reference](../configuration/dhconfig/pq.md)
- [Remote Query Dispatcher runbook](runbook-remote-query-dispatcher.md)
- [Configuration Server runbook](runbook-config-server.md)