---
title: Add merge and validation PQs
sidebar_label: Add merge and validation PQs
---

Deephaven ships ready-made [Persistent Query](../../query-management/pq-overview.md) (PQ) templates that merge and validate the [critical tables requiring regular merges](../architecture/resilience-planning/resilience-planning-data-and-configuration.md#critical-tables-requiring-regular-merges). These templates are installed on every server but are **not** imported automatically, so an administrator can review, customize, and import them on their own schedule.

## Prerequisites

These PQs merge `DbInternal` intraday data into historical storage, so your cluster needs writable historical partitions for the `DbInternal` namespace before enabling them. If it's not already been done, have your administrator [set up historical data storage](../architecture/scaling.md#the-storage-layer) based on long-term storage needs. You can follow a simple [historical data storage example](../../crash-course/data-in/streaming-binlogs.md#set-up-historical-data-storage) using `namespace="DbInternal"` instead of `namespace="ExampleNamespace"`, but this example isn't intended for long-term, expandable storage.

## What's installed

Every server installation includes one XML file per critical table under:

```
/usr/illumon/latest/install/ExamplePersistentQueries/
```

| File                                    | Table                               | Default Merge start time (America/New_York) |
| --------------------------------------- | ----------------------------------- | ------------------------------------------- |
| `AuditEventLog.xml`                     | `AuditEventLog`                     | 01:00                                       |
| `PersistentQueryConfigurationLogV2.xml` | `PersistentQueryConfigurationLogV2` | 01:10                                       |
| `PersistentQueryStateLog.xml`           | `PersistentQueryStateLog`           | 01:20                                       |
| `WorkspaceData.xml`                     | `WorkspaceData`                     | 01:30                                       |
| `ProcessEventLog.xml`                   | `ProcessEventLog`                   | 01:40                                       |
| `LogoutSchedules.xml`                   | `LogoutSchedules`                   | 01:50                                       |

Each file contains a pair of [`DeephavenCommunity`](../../query-management/worker-configuration.md) (Core+) PQs:

- A **Merge** PQ, scheduled daily and staggered 10 minutes apart from the other tables' Merge PQs so they don't all start at once.
- A **Validate** PQ (running the `DynamicValidator` and `RowCountValidator` validators) scheduled to run only after its table's Merge PQ completes successfully, using the standard [Dependent scheduling](../../query-management/ui-queries.md#dependent) type.

Both PQs in every pair:

- Use a 4 GB heap and target the `AutoMerge` [DB server](../../query-management/ui-queries.md#db-server).
- Are imported with `Enabled` set to `false`, so nothing runs until an administrator opts in.

Every Validate PQ also sets `DeleteIntradayData` to `true`, so once enabled it deletes the table's
intraday data for that partition after each successful validation (see [Delete Intraday Data?](../../data-guide/validation.md#validate-settings)). To keep intraday data after merging,
change `DeleteIntradayData` to `false` in the XML before importing, or uncheck **Delete intraday
data** on the Validate PQ's **Validate Settings** tab in Query Monitor before enabling it.

Nothing in the install process references these files, so they are never imported automatically — they exist purely as a starting point.

## Importing

Import a table's pair of PQs with [`dhconfig pq import`](../configuration/dhconfig/pq.md#import):

```bash
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig pq import \
    --file /usr/illumon/latest/install/ExamplePersistentQueries/AuditEventLog.xml \
    --retain-serial
```

Repeat for each table you want to manage this way, or import all six files at once:

```bash
for f in /usr/illumon/latest/install/ExamplePersistentQueries/*.xml; do
    sudo -u irisadmin /usr/illumon/latest/bin/dhconfig pq import --file "$f" --retain-serial
done
```

`dhconfig pq import` only accepts one `--file` at a time, so importing every table is a loop over the directory rather than a single command. The same files can also be imported through the Query Monitor's [Import Persistent Queries](../../query-management/ui-queries.md#import-persistent-queries) UI.

Import each table's Merge and Validate PQs together, in the same `dhconfig pq import` call (or the same Query Monitor import), as shown above. The Validate PQ's dependent schedule references its Merge PQ by serial number, but both the CLI and the Query Monitor UI automatically remap that reference to the new serial when you import the pair together — so either `--retain-serial` (the **Keep** option) or new serials (the **New** option) preserves the dependency. If you instead import the Validate PQ's file separately from an already-imported Merge PQ using new serials, the import fails with a dependency error rather than silently breaking the link.

## Enabling

After import, every PQ is disabled. Review the Merge PQ's schedule, heap, and `DbServerName` for your environment, then enable it (and its dependent Validate PQ) from the Query Monitor or with `dhconfig pq` once you're ready for it to run.

## Related documentation

- [Set up historical data storage](../../crash-course/data-in/streaming-binlogs.md#set-up-historical-data-storage)
- [Table storage overview](../table-storage/table-storage-overview.md)
- [NFS partition methodologies](../table-storage/nfs.md)
- [Data and configuration storage](../architecture/resilience-planning/resilience-planning-data-and-configuration.md)
- [Managing Persistent Queries](../architecture/resilience-planning/managing-persistent-queries.md)
- [Data validation](../../data-guide/validation.md)
- [`dhconfig pq`](../configuration/dhconfig/pq.md)
- [Import Persistent Queries](../../query-management/ui-queries.md#import-persistent-queries)
- [Persistent Query XML Schema](../../query-management/pq-xml-schema.md)
