Troubleshoot Java processes

Since Deephaven's core components and user queries operate as Java processes, a deep understanding of Java's diagnostic tools is essential for effective troubleshooting. This guide covers several standard Java utilities that allow you to inspect the state of a running process at a low level. The techniques described here are invaluable for diagnosing complex issues related to process configuration, SSL/TLS connectivity, gRPC connectivity, and performance bottlenecks by analyzing thread activity.

Debug process configuration

Use the sudo jinfo <pid> command to print Java configuration information for a specified Java process like the CLASSPATH and other useful info about a running Java process.

Debug SSL/TLS connections

Understanding SSL/TLS connection problems can sometimes be difficult, especially when it is not clear what messages are actually being sent and received.

You can get excellent certificate debugging by adding -Djavax.net.debug=all to your Java launch parameters to help troubleshoot or diagnose SSL/TLS connections. As this produces significant log output, it is not enabled by default.

For the Deephaven launcher, this is best accomplished with jvmargs.txt. For server processes, you must add EXTRA_ARGS parameters to the hostconfig file.

Debug gRPC connections

Deephaven services and command-line tools communicate over gRPC. When a gRPC connection fails, the process log often shows only the outcome and not the cause. For example, a Controller client that cannot reach the Controller reports a heartbeat failure like this:

ControllerClientGrpc|UUID=A3914F516EA91ED9C33901B528E64388: missed heartbeat, consecutiveLost=1, exception=StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 1.977090334s. Name resolution delay 0.000222292 seconds. [closed=[], open=[[wait_for_ready, buffered_nanos=2005769208, waiting_for_connection]]]

gRPC logs the details of the connection attempt, but Deephaven hides most of it by default. Two environment variables expose that detail. Setting either variable to any value, even false, enables it.

Environment variable	Effect
`DH_GRPC_DISABLE_LOGGING_EXCLUSIONS`	Removes the filter that server processes apply to the noisiest gRPC classes, so their low-level messages reach the process log. It does not change the gRPC log level, so it has no effect on command-line tools, which launch with gRPC logging set to warnings and above.
`DH_GRPC_STDERR_LOGGING`	Writes every gRPC log message synchronously to standard error, with no filtering and regardless of the process's gRPC log level. Use this with command-line tools such as `dhconfig`, so the diagnostics appear in the terminal as the command runs.

You cannot set these in a Deephaven properties file, because gRPC logging is configured before the process connects to the Configuration Server. Where you set the variable depends on the process:

Server processes on traditional deployments: add an export line to the stanza for that process in the hostconfig file, as shown in Java process launch configuration, then restart the process with dh_monit.
```
export DH_GRPC_DISABLE_LOGGING_EXCLUSIONS=1
```
Server processes on Podman deployments: the hostconfig file lives inside each container rather than on the host. Open a shell in the container, make the same change there, and restart the process with dh_monit. The change does not survive recreating the container.
```
podman exec -it <container name> bash
```
Server processes on Kubernetes: add the variable under the userEnv key in your Helm values and apply it with a Helm upgrade, as described in Kubernetes process launch configuration.
```
userEnv:
  common:
    DH_GRPC_DISABLE_LOGGING_EXCLUSIONS: "1"
```

Command-line tools: set the variable in the shell for a single invocation.

DH_GRPC_STDERR_LOGGING=1 /usr/illumon/latest/bin/dhconfig properties list

If the gRPC detail points at a TLS handshake, combine this with the SSL/TLS debugging described above.

As with -Djavax.net.debug=all, this produces significant log output and adds overhead to every connection. Enable it only while diagnosing a problem, and remove it afterward.

Heap dumps

A heap dump is a snapshot of the memory of a Java process. Heap dumps are crucial for diagnosing memory-related problems, such as memory leaks, garbage collection issues, and java.lang.OutOfMemoryError exceptions. The dump file can be analyzed with tools like Eclipse MAT or VisualVM to inspect the objects on the heap.

To create a heap dump, you can use the jmap utility:

sudo jmap -dump:format=b,file=<file-path> <pid>

file-path is the path where the heap dump file will be saved (e.g., /tmp/worker_heap.bin).
pid is the process ID of the Deephaven process.

For example, to create a heap dump of worker_1:

sudo jmap -dump:format=b,file=/tmp/worker_1.bin $(ps -ef | grep worker_1 | awk '{print $2}' | head -n 1)

Caution

Generating a heap dump will pause the Java process, sometimes for a significant amount of time depending on the heap size. This will make the process unresponsive. Perform this action with caution on production systems.

Thread dumps

There are times during incident management when the Deephaven Support team may ask you to perform a thread dump of a particular Deephaven process. This does not impact the system in any way, but records valuable diagnostic information into the logs to be used in troubleshooting.

Perform a thread dump with jstack

To perform a thread dump with jstack, simply run:

sudo jstack -F <pid> > <file-path>

pid is the process ID of the Deephaven process.
file-path is the file path where the thread dump will be written.

For example, to perform a thread dump of worker_1, use ps, grep, and awk to capture the process ID:

sudo jstack -F $(ps -ef | grep worker_1 | awk '{print $2}' | head -n 1) > /tmp/threadDumpWorker1.txt

Perform a thread dump with kill -3 (SIGQUIT)

In cases where you want the thread dump to go to the process logs, kill -3 can be used and the thread dump will be sent to the standard output stream of the process.

To perform a thread dump with kill -3, simply run:

sudo kill -3 <pid>

pid is the process ID of the Deephaven process.

For example, to perform a thread dump of worker_1, use ps, grep, and awk to capture the process ID:

sudo kill -3 $(ps -ef | grep worker_1 | awk '{print $2}' | head -n 1)

By default, worker logs are sent to the Deephaven database along with stderr and stdout, where stderr is redirected to the stdout for the process.

To view stdout logs for worker_d0498a61, the following query can be used:

t=db.liveTable("DbInternal", "ProcessEventLog").where("Date=today()", "Process=`worker_d0498a61`", "Level=`STDOUT`").sort("Timestamp")

t = (
    db.live_table("DbInternal", "ProcessEventLog")
    .where(["Date=today()", "Process=`worker_d0498a61`", "Level=`STDOUT`"])
    .sort("Timestamp")
)

Analyzing garbage collection

Analyzing garbage collection (GC) logs is a powerful technique for diagnosing performance issues, latency spikes, and memory pressure in a Java application. By enabling GC logging, you can get detailed insights into heap usage, pause times, and the frequency of collection events.

For comprehensive information on enabling GC logging, interpreting G1 log messages, and selecting JVM profiles, see the Garbage collection logging guide.

Modify Java launch parameters

Many of the troubleshooting techniques described in this guide, such as enabling detailed logging, require adding arguments to the Java command line that launches a process. The method for setting these parameters varies depending on whether you are targeting a service, a worker, or an interactive script.

For detailed instructions on how to modify launch parameters for each process type, see the Java process launch configuration guide.