Configuring heap dump on Assertion Failure
When the Deephaven query engine encounters inconsistent state, it may trigger an assertion using its internal assertion library. The assertion causes an Error to be thrown which contains a message that is intended to provide some information to the Deephaven developers about the cause of the error. The assertion message however only provides a small amount of information. This can be a useful point for attaching a debugger, but many errors in the engine are rare and thus trying to catch them with a debugger is impractical.
Heap dumps are a file containing all of the live objects that exist within Java virtual machine’s heap along with stack traces of each running thread. The tools for analyzing a heap dump are not as powerful as a debugger, but the heap dumps can be automatically triggered and then examined afterwards.
It is possible to initiate a heap dump on assertion or requirement failures, by setting the following properties:
assertion.heapDump=truerequire.heapDump=true
To configure, set -Dassertion.heapDump=true to the Extra JVM arguments of the query, or insert it into your global configuration files. These properties default to false. When set, workers (RemoteQueryProcessors) will initiate a heap dump that it can be analyzed to help identify a root cause with tools such as jvisualvm, JProfiler, or eclipse.
Heap dump files may be very large and are by default written to /var/log/deephaven/query_server (for the query server) or /var/log/deephaven/merge_server (for merge server worker heap dumps). You may select an alternative directory by setting the following property:
heapDump.path
The worker will check for the existence of this directory on startup; you must ensure that it exists and is writable by the dbquery (for db_query_server) and dbmerge (for db_merge_server) users. If the specified directory does not exist, then your workers will fail to start.