This page provides an overview of query management in Deephaven. It focuses on managing PQs via the UI and programmatically, sourcing query code from Git, and modularizing queries with built-in utilities.

Persistent Queries (PQs) are one of the most powerful features Deephaven offers. They are queries that are scheduled to run on a regular basis, automating processes that must be done periodically and/or continually. PQs are commonly used in existing Enterprise systems to automate tasks such as:

  • Continuous data ingestion
  • Trading strategies
  • Data quality checks
  • Data transformation
  • Performance monitoring
  • And much more...

Persistent Query management

A Deephaven Persistent Query (PQ) contains either Python or Groovy/Java code that utilizes Deephaven's table API to analyze or ingest data, along with the details required to run this code, such as the type of query, the heap required to run it, and the users who can access it. Any given Deephaven Enterprise installation can have thousands of PQs written by different users, teams, or departments. Managing queries effectively will help your organization make the most of Deephaven.

Topics in query management include:

Types of PQs

Deephaven offers several types of PQs. Some are available to all users, others are available only to users with administrative privileges, and the rest are internal PQs necessary for the system to function. A basic understanding of the types, access, and functions of different PQs will help manage them.

Available to all users

  • Live Query (Script): A script that runs on a query server.
  • Live Query (ReplayScript): A script runs on a query server and replays historical data as if it were live and ticking.
  • Batch Query (RunAndDone): A query that runs on a query server, executes once, and then stops.

Available to users with elevated privileges

  • Live Query (Merge): A script that runs on a merge server.
  • Batch Query (ImportServer): A query that runs on a merge server, executes once, and then stops.
  • Import - JDBC: A query that imports data from a JDBC source.
  • Import - CSV: A query that imports data from a CSV file.
  • Import - XML: A query that imports data from an XML file.
  • Import - Deephaven Binary Logs: A query that imports data from Deephaven binary logs.
  • Import - Downsample: A query that imports data from one or more existing tables and downsamples it.
  • Data Merge: A query that merges intraday data into historical data.
  • Data Validation: A query that validates and deletes merged intraday data.

Internal

  • Web Client Data: Supports the web UI. If this is not running, the web UI is not accessible, and the only way to access any UI is via safe mode to restart this PQ.
  • Workspace Data Snapshot: Typically run nightly, it takes snapshots of the WorkspaceData table. This helps the web UI respond faster during loading when a user logs in.
  • Revert Helper: Used by the classic Swing UI to assist when a query is reverted to a previous version.
  • Import Helper: Used by the classic Swing UI to assist with import, merge, and validation queries.
  • Interactive Console: Run when a user creates a new Code Studio (temporary PQ) or when the Share Console option is enabled in the Swing UI.
  • Telemetry Helper: Used by the classic Swing UI to record the user-perceived delay of UI operations.