Finding errors

Persistent query processes record their activity and state in the ProcessEventLog and the PersistentQueryStateLog tables. When errors or exceptions occur, these tables can be queried to find error messages or queries that have failed.

Note

See: Internal tables

Querying the PersistentQueryStateLog table

The PersistentQueryStateLog is updated every time a persistent query changes state, and can be used to see both the current state of any persistent query as well as all the historical states. The status of failed queries could be:

  • Failed: The persistent query failed before or during initialization.
  • Error: An error occurred after the query was initialized (e.g., processing incremental updates)
  • Disconnected: The worker process disconnected from the dispatcher. Often this results from a problem that terminates the JVM without giving the worker a chance to notify the dispatcher (e.g., OutOfMemoryError or hotspot errors in native code).

The following example shows all times a persistent query entered a failure state on the current date:

pqsAllFailedStates = db.liveTable("DbInternal","PersistentQueryStateLog").where("Date=today()", "Status in `Failed`, `Error`, `Disconnected`")

pqsAllFailedStates = db.live_table("DbInternal", "PersistentQueryStateLog").where(
    ["Date=today()", "Status in `Failed`, `Error`, `Disconnected`"]
)

Adding a lastBy call to the query shows the most recent state of any persistent query that attempted to start on the current date:

pqslAllCurrentStates = db.liveTable("DbInternal","PersistentQueryStateLog").where("Date=today()").lastBy("SerialNumber")

pqslAllCurrentStates = (
    db.live_table("DbInternal", "PersistentQueryStateLog")
    .where("Date=today()")
    .last_by("SerialNumber")
)

Combining the above queries indicates which persistent queries are in a failed state today:

pqslCurrentFailedStates = db.liveTable("DbInternal","PersistentQueryStateLog").where("Date=today()").lastBy("SerialNumber").where("Status in `Failed`, `Error`, `Disconnected`")

pqslCurrentFailedStates = (
    db.live_table("DbInternal", "PersistentQueryStateLog")
    .where("Date=today()")
    .last_by("SerialNumber")
    .where(["Status in `Failed`, `Error`, `Disconnected`"])
)

Querying the ProcessEventLog table

To find all FATAL Level messages in the ProcessEventLog, the following query can be used:

pel = db.liveTable("DbInternal","ProcessEventLog").where("Date=today()").where("Level=`FATAL`")

pel = (
    db.live_table("DbInternal", "ProcessEventLog")
    .where("Date=today()")
    .where("Level=`FATAL`")
)

To find specific errors or error messages, use the Process Info Id to query the ProcessEventLog:

t = db.liveTable("DbInternal","ProcessEventLog")
    .where("Date=today()")
    .where("ProcessInfoId=`<your_value>`")

t = (
    db.live_table("DbInternal", "ProcessEventLog")
    .where("Date=today()")
    .where("ProcessInfoId=`<your_value>`")
)

The Process Info ID can be found by hovering over the Info button in the console, and the worker's name includes a prefix of the ProcessInfoId:

img

Hold your mouse over the Log Entry to view the full exception message and stack trace from the results of the query.

img

To search for results in the LogEntry column, we recommend using the Go to feature, which you can access by right-clicking a cell or using the keyboard shortcut Cmd+G:

img

Viewing Persistent Query Exceptions in the Console

When a persistent query fails with a Java exception, you can view the exception message and stack trace in the Deephaven console.

To view the exception message, open the Query Config panel or Query Monitor. Then right-click on the failed query in the "Exception Details" column and select Show Exceptions in the drop-down menu:

img