Programmatic query management

This guide covers programmatic scheduling, management, and monitoring of Persistent Queries (PQs). PQs are queries that are scheduled to run regularly to perform data ingestion and analysis, create dashboards, and more. Deephaven has both Python and Groovy APIs for managing PQs, both of which are discussed.

Programmatic management allows you to create more dynamic process workflows that would be difficult or impossible using only manual management in the UI. You can monitor, create, stop, restart, and delete PQs programmatically.

This guide does not cover the Query Monitor, a graphical interface for managing PQs. See Query management in the UI for more details.

Monitor

To monitor PQs programmatically, start with the SessionManager class in Python, and the ControllerClientFactory in Groovy.

Note

The code blocks below are run from Code Studios within a Deephaven installation. If connecting remotely, be sure to provide connection details and authentication. See Python client and/or Java client for details.

With this, you can get information on each PQ in the system in a map:

This map uses the serial number of each PQ as the key. The following code prints the name and serial number of each PQ:

You can access individual PQs by their serial number:

Create

This section presents an example of creating a merge PQ. This process applies to most other PQ types, as the steps are similar.

First, each language requires import statements:

Next, define a required method/closure for type encoding that will be used during creation:

With that done, you must first create a query configuration. This will use the previously defined method/closure to encode the type-specific fields:

See the Javadoc for a full list of type-specific fields for merge PQs.

Next, define scheduling, which determines when the PQ runs. The following code defines a daily scheduler that runs from all seven days of the week from 07:55:00 to 23:55 in the US/East timezone, along with some additional scheduling parameters:

Note

The Python code below uses many of the default values in generate_daily_scheduler. See the link for details on the defaults.

For other ways to define scheduling, see Programmatic scheduling.

Lastly, specify timeout values:

With all of the setup done, create the PQ:

The entire code block is as follows:

Full code

Schedule

Earlier, Python used GenerateScheduling to create scheduling details for a PQ. You can alternatively do this closer to how Groovy did it, by passing a list of scheduling parameters to the configuration. The following example creates a daily scheduler that runs from all seven days of the week from 07:55:00 to 23:55 in the US/East timezone, along with some additional scheduling parameters:

The earlier Groovy code added scheduling parameters to the configuration one at a time in a loop. You can instead add them all at once:

Available schedulers

Deephaven offers the following types of programmatic scheduling for PQs:

Use GenerateScheduler.generate_continuous_scheduler or set SchedulerType=com.illumon.iris.controller.IrisQuerySchedulerContinuous in the properties list to create a continuous scheduler.

Use GenerateScheduling.generate_daily_scheduler or set SchedulerType=com.illumon.iris.controller.IrisQueryScheduler in the properties list to create a daily scheduler.

Use GenerateScheduling.generate_dependent_scheduler or set SchedulerType=com.illumon.iris.controller.IrisQuerySchedulerDependent in the properties list to create a dependent scheduler.

Use GenerateScheduling.generate_monthly_scheduler or set SchedulerType=com.illumon.iris.controller.IrisQuerySchedulerMonthly in the properties list to create a monthly scheduler.

Use GenerateScheduling.generate_range_scheduler or set SchedulerType=com.illumon.iris.controller.IrisQuerySchedulerRange in the properties list to create a range scheduler.

Use GenerateScheduling.generate_temporary_scheduler or set SchedulerType=com.illumon.iris.controller.IrisQuerySchedulerTemporary in the properties list to create a temporary scheduler.

Deephaven offers the following types of programmatic scheduling for PQs:

Set SchedulerType=com.illumon.iris.controller.IrisQuerySchedulerContinuous to create a continuous scheduler.

Set SchedulerType=com.illumon.iris.controller.IrisQuerySchedulerDaily to create a daily scheduler.

Set SchedulerType=com.illumon.iris.controller.IrisQuerySchedulerDependent to create a dependent scheduler.

Set SchedulerType=com.illumon.iris.controller.IrisQuerySchedulerMonthly to create a monthly scheduler.

Set SchedulerType=com.illumon.iris.controller.IrisQuerySchedulerRange to create a range scheduler.

Set SchedulerType=com.illumon.iris.controller.IrisQuerySchedulerTemporary to create a temporary scheduler.

Stop

Stopping a PQ programmatically is simple - all you need is its serial number.

Restart

Restarting a PQ programmatically is simple - all you need is its serial number.

Delete

Deleting a PQ programmatically is simple - all you need is its serial number.

Caution

Deleting a PQ is permanent and cannot be undone.