Troubleshoot Java Processes

Deephaven components and queries run as Java processes. To find and correct system errors, it is sometimes necessary to use Java tools to understand the state of the system.

Debugging process configuration

Use the sudo jinfo <pid> command to print Java configuration information for a specified Java process like the CLASSPATH and other useful info about a running Java process.

Debugging SSL/TLS Connections

Understanding SSL/TLS connection problems can sometimes be difficult, especially when it is not clear what messages are actually being sent and received.

You can get excellent certificate debugging by adding -Djavax.net.debug=all to your Java launch parameters to help troubleshoot or diagnose SSL/TLS connections. As this produces significant log output, it is not enabled by default.

For the Deephaven launcher, this is best accomplished with jvmargs.txt. For server processes, you must add EXTRA_ARGS parameters to the hostconfig file.

Thread dumps

There are times during incident management when the Deephaven Support team may ask you to perform a thread dump of a particular Deephaven process. This does not impact the system in any way, but records valuable diagnostic information into the logs to be used in troubleshooting.

Perform a thread dump with jstack

To perform a thread dump with jstack, simply run:

sudo jstack -F <pid> > <file-path>
  • pid is the process ID of the Deephaven process.
  • file-path is the file path where the thread dump will be written.

For example, to perform a thread dump of worker_1, use ps, grep, and awk to capture the process ID:

sudo jstack -F $(ps -ef | grep worker_1 | awk '{print $2}' | head -n 1) > /tmp/threadDumpWorker1.txt

Perform a thread dump with kill -3 (SIGQUIT)

In cases where you want the thread dump to go to the process logs, kill -3 can be used and the thread dump will be sent to the standard output stream of the process.

To perform a thread dump with kill -3, simply run:

sudo kill -3 <pid>
  • pid is the process ID of the Deephaven process.

For example, to perform a thread dump of worker_1, use ps, grep, and awk to capture the process ID:

sudo kill -3 $(ps -ef | grep worker_1 | awk '{print $2}' | head -n 1)

By default, worker logs are sent to the Deephaven database along with stderr and stdout, where stderr is redirected to the stdout for the process.

To view stdout logs for worker_d0498a61, the following query can be used:

t=db.liveTable("DbInternal", "ProcessEventLog").where("Date=today()", "Process=`worker_d0498a61`", "Level=`STDOUT`").sort("Timestamp")

t = (
    db.live_table("DbInternal", "ProcessEventLog")
    .where(["Date=today()", "Process=`worker_d0498a61`", "Level=`STDOUT`"])
    .sort("Timestamp")
)