How to remotely debug workers

Remote debugging allows you to introspect and even manipulate the state of a worker and can be applied to either Code Studios or Persistent Queries. You may want to debug a worker to examine the state of a Deephaven object, or to examine and step through your own Java extensions.

To debug a Deephaven worker, you should create an IntelliJ project that includes the Deephaven JARs and any of your own components.

• When debugging a Legacy worker, you must create a project with the Legacy JARs.
• When debugging a Core+ worker, you must create a project with Deephaven Core+ and Deephaven Community Core JARs.

There are two modes of remote debugging available:

1. Attaching to a remote JVM. In this mode, you start the worker first and then start the debugger second.
2. Listening to a remote JVM. In this mode, you start the debugger first and then start the worker second.

In either case, you must add additional options to your worker's JVM arguments. From the Code Studio, set the arguments in the Extra JVM Arguments text box after expanding Show Advanced:

For a Persistent Query, set Extra JVM Arguments after expanding Show Advanced in the Settings tab:

IntelliJ run configuration

The Jetbrains remote debugging tutorial describes how to configure an IntelliJ run configuration attaching or listening to a remote JVM. The configuration for "Remote JVM Debug" allows you to select the debugging mode, the port, and if applicable the remote hostname. Command Line Arguments for remote JVM: in this dialog serve as a suitable template for the extra JVM arguments to provide to your worker:

Attaching to a remote JVM

Typical worker JVM arguments to start a JVM that listens for connections from a remote debugger (i.e., the IntelliJ debugger attaches to a remote JVM):

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005

Unable to allocate console: Failed to get query processor from dispatcher DispatcherConnection-query-example-com:22013 for 5581d06a-f474-4f7e-be4f-ed156830a33c
caused by Received invalid/rejection dispatcher response for 5581d06a-f474-4f7e-be4f-ed156830a33c
caused by Cancelled worker due to failure to start:
ERROR: transport error 202: bind failed: Address already in use
ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [src/jdk.jdwp.agent/share/native/libjdwp/debugInit.c:734]

Listening to a remote JVM

Typical worker JVM arguments to make it connect to a listening debugger (i.e., the IntelliJ debugger is listening to a remote JVM):

-agentlib:jdwp=transport=dt_socket,server=n,address=localhost:5005,suspend=y

Make sure your listening debugger and any applicable SSH port forwarding is enabled before starting your worker.

SSH Port Forwarding

Remote debugging requires TCP connectivity between your local IDE and the worker on your selected port. Depending on your network, VPN, and firewall configuration it may be necessary or easier to use SSH tunnels to establish the necessary connectivity.

For example, if you have a worker that is listening on port 5050 on the host worker-host.example.com and you want to connect to port 5005 from IntelliJ, you could execute the following SSH command to create a tunnel that forwards port 5005 on your local machine to port 5050 on the remote machine:

ssh -NL 5005:localhost:5050 worker-host.example.com

The -N option instructs SSH not to open an interactive shell, instead only the tunnel is opened. Adding the -v option can sometimes be useful to troubleshoot connectivity issues.

If your debugger is listening on port 5006, and you've configured a worker on worker-host.example.com to connect to 5060 on startup, the following SSH command to create the tunnel:

ssh -NR 5060:localhost:5006 worker-host.example.com