PersistentQueryStateLog

The Persistent Query Controller manages all Persistent Queries (PQs) in Deephaven, including starting, stopping, and monitoring the health of the PQs.

When a query worker is started or stopped, it goes through a series of state changes that represent the worker's status. Each state represents a specific condition, and every state change for every PQ is stored in the PersistentQueryStateLog. Current states are also visible in the status column in the Query Configuration Deephaven console panel.

The [Persistent Query Configuration Log] (./persistent-query-configuration-log. md) table stores PQ history (details about each PQ addition, modification, and deletion).

Columns

Column Name	Column Type	Description
`Date`	String	The date the state change occurred. This is the partitioning column.
`Owner`	String	The owner of the Persistent Query.
`Name`	String	The name of the Persistent Query.
`Timestamp`	DateTime	The timestamp for the state change.
`Status`	String	The new query state.
`ControllerHost`	String	The controller host that managed the query.
`DispatcherHost`	String	The host of the dispatcher that managed the worker.
`ServerHost`	String	The host on which the query ran.
`WorkerName`	String	The worker's name.
`ProcessInfoId`	String	The process info ID of the worker (a unique identifier for the process that can be used to cross-reference the worker with logs such as the query performance logs).
`WorkerPort`	int	The worker's port for connections.
`LastModifiedByAuthenticated`	String	The authenticated user who last started this query.
`LastModifiedByEffective`	String	The effective user who last started this query.
`SerialNumber`	long	The query's serial number (a system-generated identifier unique to each Persistent Query).
`VersionNumber`	long	The query's version number (a number that starts at 1 for each query and is incremented each time the query is updated).
`TypeSpecificState`	String	A type-specific state value; this will not apply to most queries.
`ExceptionMessage`	String	If applicable, the exception message for a failed query.
`ExceptionStackTrace`	String	If applicable, the full stack trace for a failed query.
`EngineVersion`	String	For Core+ workers, the version of the engine on which the worker ran, such as `0.39.0`.
`WorkerKind`	String	The type of worker, either `DeephavenEnterprise` or `DeephavenCommunity`.
`ScriptLoaderState`	Map	State of script loader infrastructure when logged. Currently, only "git" repository states are relevant, with the revision fingerprint of each, keyed by repo name, logged.
`DispatcherPort`	int	The dispatcher port that managed the worker.
`ReplicaSlot`	int	The replica slot of the worker.
`StatusDetails`	String	Further details on the PQ state, such as the worker pod state in Kubernetes.

Query states

State	Description
`Uninitialized`	The query has no status as it has not been run.
`Connecting`	The controller is connecting.
`Authenticating`	The query is authenticating with the authentication server.
`AcquiringWorker`	The dispatcher is creating a worker process for the query.
`Initializing`	The worker process for the query is initializing.
`Running`	The query worker is running (script query types only).
`Failed`	The query worker failed to start correctly; an exception should be visible.
`Error`	The query generated an error.
`Disconnected`	The query worker disconnected unexpectedly. This may occur, for example, if the JVM experiences an `OutOfMemoryError`, a Hotspot error (possibly caused by a buggy native module), or if the garbage collector pauses the JVM for longer than the heartbeat interval between the worker and dispatcher.
`Stopping`	The query is shutting down.
`Stopped`	The query stopped normally (Live Query (Script) query types only).
`Completed`	The query completed successfully (Batch Query (RunAndDone) and data-management (merge, import, and validation) query types only).
`Executing`	The query is executing (Batch Query (RunAndDone) and data-management (merge, import, and validation) query types only).

Example

To see the state changes of a specific Persistent Query (PQ), you can filter the PersistentQueryStateLog table by the query's name. The following example shows how to do this in both Core+ Python and Groovy, using today's data.

pqsl=db.liveTable("DbInternal", "PersistentQueryStateLog").where("Date=today()", "Name=`<PQ Name>`").sort("Timestamp")

pqsl = (
    db.live_table("DbInternal", "PersistentQueryStateLog")
    .where(["Date=today()", "Name=`<PQ Name>`"])
    .sort("Timestamp")
)

The following combines the PersistentQueryStateLog with the ProcessEventLog table to see the events for the most recent run of a PQ.

First, it queries the PersistentQueryStateLog to find the most recent entry for the specified PQ.
Then it queries the ProcessEventLog table for all events that match the ProcessInfoId of the most recent entry.
If you're using replicas or spares, you may want to restrict the PersistentQueryStateLog to a specific replica.

pqslLast=db.liveTable("DbInternal", "PersistentQueryStateLog").where("Date=today()", "Name=`<PQ Name>`")
        .sort("Timestamp")
        .tail(1)

pel=db.liveTable("DbInternal", "ProcessEventLog")
        .where("Date=today()")
        .whereIn(pqslLast, "ProcessInfoId")
        .sort("Timestamp")

pqslLast = (
    db.live_table("DbInternal", "PersistentQueryStateLog")
    .where(["Date=today()", "Name=`<PQ Name>`"])
    .sort("Timestamp")
    .tail(1)
)

pel = (
    db.live_table("DbInternal", "ProcessEventLog")
    .where("Date=today()")
    .where_in(pqslLast, "ProcessInfoId")
    .sort("Timestamp")
)

PersistentQueryStateLog

Columns

Query states

Example

Related documentation