Persistent Query Controller Tool

All Deephaven persistent queries are under the control of the Persistent Query Controller process. This process stores all persistent queries, and is responsible for starting and stopping them at the appropriate times. The Persistent Query Controller Tool is an administrative tool that interacts with the controller to perform various functions.

On a standard installation, the controller tool is available as part of the installation through the standard iris script. It is accessed by running the command:

sudo /usr/illumon/latest/bin/iris controller_tool <options>

All operations generate a detailed log, typically in:

/var/log/deephaven/misc/PersistentQueryControllerTool.log.<date-time>

Reload

Certain configuration parameters within the controller can be reloaded without requiring a controller restart. See Persistent Query Controller Configuration for details on controller configuration parameters, including the reloadable parameters.

To have the controller dynamically reload its configuration, log on to the server where the controller is running and use the following command:

sudo /usr/illumon/latest/bin/iris controller_tool --reload

Export

The export option exports the specified queries to an XML file, which can then be imported to a different controller. Unless further options are specified, all queries except for temporary queries and internal Deephaven queries are exported. Use the following command:

sudo /usr/illumon/latest/bin/iris controller_tool --export

By default, the file is named controllerToolExport.xml and placed in the controller tool's workspace at:

[root@host controller_tool]# pwd
/db/TempFiles/irisadmin/controller_tool

All the options shown in the General options section apply to export operations. In addition, the following option can be used:

  • -x, --xmlFile - Specify the name and directory of the produced XML file.

Import

An XML file produced by the controller tool's export option can be imported to any controller running the same Deephaven version. Importing to an earlier version of Deephaven may result in errors, as some query options may be unavailable. By default, all queries in the XML file, except for temporary queries and internal Deephaven queries, are imported. Importing only adds new queries and does not remove old queries. All Persistent Queries must have unique names and serial numbers. You can change a query's name by importing it with the same serial number, but cannot change a query's serial number.

sudo /usr/illumon/latest/bin/iris controller_tool --import

All the options shown in the General options section apply to import operations. In addition, the following options can be used:

  • -x, --xmlFile - Specify the name and directory of the XML file to import.
  • -v, --overrideXmlFile - Specify the name and directory of an optional XML file containing properties that will override any defined in the persistent query XML file being imported.
  • -C, --continueAfterError - If an error occurs during the import operation, continue anyway. By default, all queries to be imported are validated before any are imported, and any errors (either during validation or during the import) cause the import to stop.
    • --continueAfterError=true
  • -N, --newOwner - Change the owner of the imported queries to the specified value.
  • -O, --serverOverride - If specified, this updates the server on which the imported queries will run. It is in the form <old server name>:<new server name>. Each query in the export file that originally ran on the old server name will be updated during the import to use the new server name. This option may be specified more than once. For example, specifying both of the following options would change the server named Merge_1 to the new name MergeServer2, and the server named Query_1 to the new name QueryServer2, in any imported queries:
    • --serverOverride=Merge_1:MergeServer2
    • --serverOverride=Query_1:QueryServer2
  • -R, --retainSerial - Every Persistent Query has a unique, controller-assigned identifier called a serial. By default, all imported queries are assigned new serials (meaning that an imported query is treated as a new query). The retainSerial parameter changes this behavior as follows:
    • --retainSerial=keep - Keep the original query's serial as specified in the XML file, and do not import the query if another query has the same serial.
    • --retainSerial=replace - Keep the original query's serial as specified in the XML file, and replace an existing query if it has the same serial.
  • -b, —dryRunBasic - Perform a check of the file to be imported that will report any obvious problems with the file. This basic check does not connect to a query controller.
  • -f, —dryRunFull - Performs the basic check, and if that does not discover a problem a connection to the controller is made and a more comprehensive check is done against that controller's configuration settings which may impose further limitations on allowable values. For example, a controller might specify a maximum allowable heap size that the basic check would not see as invalid.

Override files

This example shows how to import a persistent query file that has been exported previously, using an additional override configuration file to modify the HeapSizeInGB property. This might be done to export queries from a test environment where smaller amounts of data and less resources exist, and to import those queries into a production environment.

sudo /usr/illumon/latest/bin/iris controller_tool --import --xmlFile persistentQueries.xml --overrideXmlFile overrides.xml

The persistentQueries.xml file contains query configurations previously exported with sudo /usr/illumon/latest/bin/iris controller_tool --export and contains something similar to below:

<?xml version="1.0" encoding="UTF-8"?>
<PersistentQueryRoot>
  <PersistentQuery>
    <Serial>12345</Serial>
    <Version>1</Version>
    <Name xml:space="preserve">My Persistent Query</Name>
    <HeapSizeInGB>0.25</HeapSizeInGB>
    <ScriptCode xml:space="preserve">tt=db.timeTable(“00:00:01”)</ScriptCode>
	...
  </PersistentQuery>
  <PersistentQuery>
    ...
  </PersistentQuery>
</PersistentQueryRoot>

In order for an override to apply changes to a persistent query, it must contain a Serial element whose value matches that of the targeted persistent query

Below, the overrides.xml file contains an override for the persistent query with a serial value of 12345 and changes the HeapSizeInGB property from 0.25 to 1.0:

<?xml version="1.0" encoding="UTF-8"?>
<PersistentQueryOverrideRoot>  <!-- Note this root element is named PersistentQueryOverrideRoot, not PersistentQueryRoot -->
  <PersistentQueryOverride>    <!-- Note this element is named PersistentQueryOverride, not PersistentQuery -->
    <Serial>12345</Serial>
    <HeapSizeInGB>1.0</HeapSizeInGB>
  </PersistentQueryOverride>
</PersistentQueryOverrideRoot>

In some limited use cases, an override XML can be useful, such as when running the same script under a different environment. The following fields are example candidates for override:

  • HeapSizeInGB
  • JVMProfile
  • DbServerName

Tip

Other fields can also be added to overrides. Whether doing so is trivial or complex will depend on the nature of the change desired and whether or not validation will allow it. In many use cases requiring a substantial change, it is usually preferable to simply change the original PQ in an appropriate manner rather than relying on overrides.

In particular, we do not recommend changing Configuration Type, except when the configuration types are very similar.

Stop and Restart

The controller tool can be used to stop and restart queries. Unless further options are specified, all queries except for temporary queries and internal Deephaven queries are stopped or restarted when these options are chosen. All the general options are available to restrict which queries are stopped or restarted.

sudo /usr/illumon/latest/bin/iris controller_tool --stop
sudo /usr/illumon/latest/bin/iris controller_tool --restart

General options

The following options are available to restrict the queries operated on by the import, export, stop, and restart commands.

  • -S, --serial - Operate only on the specified serial numbers. This parameter can be specified multiple times, or multiple serials can be specified delimited by commas (without spaces between the serials). This option cannot be specified with --name or --owner.
  • -o, --owner - Operate only on queries owned by the specified owner.
  • -n, --name - Operate only on queries with the specified name.
  • -D, --includeNonDisplayable - Include non-displayable queries (such as the Import Helper and Revert Helper) queries in the operation. By default, these queries are not included in controller tool operations.
    • --includeNonDisplayable=true
  • -T, --includeTemporary - Include temporary queries in the operation. By default, temporary queries are not included in controller tool operations as the queries are typically used for one-time operations such as historical data imports.
    • --includeTemporary=true