Add merge and validation PQs
Deephaven ships ready-made Persistent Query (PQ) templates that merge and validate the critical tables requiring regular merges. These templates are installed on every server but are not imported automatically, so an administrator can review, customize, and import them on their own schedule.
Prerequisites
These PQs merge DbInternal intraday data into historical storage, so your cluster needs writable historical partitions for the DbInternal namespace before enabling them. If it's not already been done, have your administrator set up historical data storage based on long-term storage needs. You can follow a simple historical data storage example using namespace="DbInternal" instead of namespace="ExampleNamespace", but this example isn't intended for long-term, expandable storage.
What's installed
Every server installation includes one XML file per critical table under:
| File | Table | Default Merge start time (America/New_York) |
|---|---|---|
AuditEventLog.xml | AuditEventLog | 01:00 |
PersistentQueryConfigurationLogV2.xml | PersistentQueryConfigurationLogV2 | 01:10 |
PersistentQueryStateLog.xml | PersistentQueryStateLog | 01:20 |
WorkspaceData.xml | WorkspaceData | 01:30 |
ProcessEventLog.xml | ProcessEventLog | 01:40 |
LogoutSchedules.xml | LogoutSchedules | 01:50 |
Each file contains a pair of DeephavenCommunity (Core+) PQs:
- A Merge PQ, scheduled daily and staggered 10 minutes apart from the other tables' Merge PQs so they don't all start at once.
- A Validate PQ (running the
DynamicValidatorandRowCountValidatorvalidators) scheduled to run only after its table's Merge PQ completes successfully, using the standard Dependent scheduling type.
Both PQs in every pair:
- Use a 4 GB heap and target the
AutoMergeDB server. - Are imported with
Enabledset tofalse, so nothing runs until an administrator opts in.
Every Validate PQ also sets DeleteIntradayData to true, so once enabled it deletes the table's
intraday data for that partition after each successful validation (see Delete Intraday Data?). To keep intraday data after merging,
change DeleteIntradayData to false in the XML before importing, or uncheck Delete intraday
data on the Validate PQ's Validate Settings tab in Query Monitor before enabling it.
Nothing in the install process references these files, so they are never imported automatically — they exist purely as a starting point.
Importing
Import a table's pair of PQs with dhconfig pq import:
Repeat for each table you want to manage this way, or import all six files at once:
dhconfig pq import only accepts one --file at a time, so importing every table is a loop over the directory rather than a single command. The same files can also be imported through the Query Monitor's Import Persistent Queries UI.
Import each table's Merge and Validate PQs together, in the same dhconfig pq import call (or the same Query Monitor import), as shown above. The Validate PQ's dependent schedule references its Merge PQ by serial number, but both the CLI and the Query Monitor UI automatically remap that reference to the new serial when you import the pair together — so either --retain-serial (the Keep option) or new serials (the New option) preserves the dependency. If you instead import the Validate PQ's file separately from an already-imported Merge PQ using new serials, the import fails with a dependency error rather than silently breaking the link.
Enabling
After import, every PQ is disabled. Review the Merge PQ's schedule, heap, and DbServerName for your environment, then enable it (and its dependent Validate PQ) from the Query Monitor or with dhconfig pq once you're ready for it to run.