Finding errors

Persistent query processes record their activity and state in the ProcessEventLog and the PersistentQueryStateLog tables. When errors or exceptions occur, these tables can be queried to find error messages or queries that have failed.

Note

See: Internal tables

Querying the PersistentQueryStateLog table

The PersistentQueryStateLog is updated every time a persistent query changes state, and can be used to see both the current state of any persistent query as well as all the historical states. The status of failed queries could be:

  • Failed: The persistent query failed before or during initialization.
  • Error: An error occurred after the query was initialized (e.g., processing incremental updates)
  • Disconnected: The worker process disconnected from the dispatcher. Often this results from a problem that terminates the JVM without giving the worker a chance to notify the dispatcher (e.g., OutOfMemoryError or hotspot errors in native code).

The following example shows all times a persistent query entered a failure state on the current date:

pqsAllFailedStates = db.i("DbInternal","PersistentQueryStateLog").where("Date=currentDateNy()", "Status in `Failed`, `Error`, `Disconnected`")

Adding a lastBy call to the query shows the most recent state of any persistent query that attempted to start on the current date:

pqslAllCurrentStates = db.i("DbInternal","PersistentQueryStateLog").where("Date=currentDateNy()").lastBy("SerialNumber")

Combining the above queries indicates which persistent queries have entered a failure state on the current date:

pqslCurrentFailedStates = db.i("DbInternal","PersistentQueryStateLog").where("Date=currentDateNy()").lastBy("SerialNumber").where("Status in `Failed`, `Error`, `Disconnected`")

Querying the ProcessEventLog table

To find all FATAL Level messages in the ProcessEventLog, the following query can be used:

t = db.i("DbInternal","ProcessEventLog").where("Date=currentDateNy()").where("Level==`FATAL`")

To find specific errors or error messages for a specific query worker, first query the PersistentQueryStateLog table to determine the worker name and worker host and use these values to query the ProcessEventLog. For example:

workerHost="hostname"
workerId="worker_123"
errorText="Problem maintaining subscription"

t = db.i("DbInternal","ProcessEventLog")
    .where("Date=currentDateNy()")
    .where("Host==workerHost")
    .where("Process==workerId")
    .where("LogEntry.contains(errorText)")

Hold your mouse over the Log Entry to view the full exception message and stack trace from the results of the query.

img

Viewing Persistent Query Exceptions in the Console

When a persistent query fails with a Java exception, you can view the exception message and stack trace in the Deephaven console.

To view the exception message, open the Query Config panel or Query Monitor. Then right-click on the failed query in the "Exception Details" column and select Show Exceptions in the drop-down menu:

img