Debugging Tools

Many tools are available to help debug the Deephaven process executing your query. These tools require knowledge of Java; knowledge of Deephaven's internals is helpful.

Connect a remote debugger

Remote debugging allows you to introspect and even manipulate a worker's state and can be applied to either Code Studios or Persistent Queries. You may want to debug a worker to examine the state of a Deephaven object or to examine and step through your own Java extensions.

To debug a Deephaven worker, you should create an IntelliJ project that includes the Deephaven JARs and any of your own components.

When debugging a Legacy worker, you must create a project with the Legacy JARs.
When debugging a Core+ worker, you must create a project with Deephaven Core+ and Deephaven Community Core JARs.

There are two modes of remote debugging available:

Attaching to a remote JVM. In this mode, you start the worker first and then start the debugger second.
Listening to a remote JVM. In this mode, you start the debugger first and then start the worker second.

Either way, you must add options to your worker's JVM arguments. From the Code Studio, set the arguments in the Extra JVM Arguments text box after expanding Show Advanced:

For a Persistent Query, set Extra JVM Arguments after expanding Show Advanced in the Settings tab:

IntelliJ run configuration

The Jetbrains remote debugging tutorial describes configuring an IntelliJ run configuration attaching or listening to a remote JVM. The configuration for "Remote JVM Debug" allows you to select the debugging mode, the port, and, if applicable, the remote hostname. Command Line Arguments for remote JVM: in this dialog serve as a suitable template for the extra JVM arguments to provide to your worker:

Attaching to a remote JVM

Typical worker JVM arguments to start a JVM that listens for connections from a remote debugger (i.e., the IntelliJ debugger attaches to a remote JVM):

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005

Note

Each listening JVM must have a unique port otherwise the worker will fail to start with an error of the form:

Unable to allocate console: Failed to get query processor from dispatcher DispatcherConnection-query-example-com:22013 for 5581d06a-f474-4f7e-be4f-ed156830a33c
caused by Received invalid/rejection dispatcher response for 5581d06a-f474-4f7e-be4f-ed156830a33c
caused by Cancelled worker due to failure to start:
ERROR: transport error 202: bind failed: Address already in use
ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [src/jdk.jdwp.agent/share/native/libjdwp/debugInit.c:734]

Warning

When attaching to a remote JVM, setting suspend=y is necessary to debug worker startup, but prevents the worker startup process from proceeding until you connect your debugger to the worker. If you do not connect a debugger quickly enough, then the worker startup times out and fails. Consider using a listening debugger instead. When debugging script code in a Code Studio, you can select suspend=n and attach the debugger before executing your script code.

Listening to a remote JVM

Typical worker JVM arguments to make it connect to a listening debugger (i.e., the IntelliJ debugger is listening to a remote JVM):

-agentlib:jdwp=transport=dt_socket,server=n,address=localhost:5005,suspend=y

Before starting your worker, make sure your listening debugger and any applicable SSH port forwarding are enabled.

SSH Port Forwarding

Remote debugging requires TCP connectivity between your local IDE and the worker on your selected port. Depending on your network, VPN, and firewall configuration, it may be necessary or easier to use SSH tunnels to establish the necessary connectivity.

For example, if you have a worker that is listening on port 5050 on the host worker-host.example.com and you want to connect to port 5005 from IntelliJ, you could execute the following SSH command to create a tunnel that forwards port 5005 on your local machine to port 5050 on the remote machine:

ssh -NL 5005:localhost:5050 worker-host.example.com

The -N option instructs SSH not to open an interactive shell, instead only the tunnel is opened. Adding the -v option can sometimes be useful to troubleshoot connectivity issues.

If your debugger is listening on port 5006, and you've configured a worker on worker-host.example.com to connect to 5060 on startup, the following SSH command to create the tunnel:

ssh -NR 5060:localhost:5006 worker-host.example.com

Heap Dumps

Heap Dumps write the entire state of the JVM heap to a file. You must have access to the server running your worker to retrieve this file and then use a tool like Java VisualVM to analyze it. You may also set the extra JVM argument -XX:+HeapDumpOnOutOfMemoryError.