Debug an embedded Deephaven server with PyCharm

The Deephaven Python embedded server (the deephaven-server package) starts the Deephaven engine directly inside your own Python process, rather than running it as a separate server you connect to. This guide shows you how to build the embedded server from source and debug it with PyCharm.

Note

If you only need to debug a pip-installed release of deephaven-server, see the pip debugging guide instead. This guide is for developers who build the embedded server from a local deephaven-core clone, for example to step into Deephaven's Python source or test local changes.

How this differs from other debugging setups

With the embedded server, your Python process embeds the JVM, so your script, the Deephaven engine, and PyCharm's debugger backend all run in one process. It's the same process PyCharm already runs when you launch a script. The setups compare as follows:

  • Docker and Gradle source build: The Deephaven server runs as its own process, in a container or from ./gradlew server-jetty-app:run, and its JVM embeds a Python interpreter. To debug it, you run PyCharm's remote debugging server, which is a paid feature. In the Deephaven console, you call settrace from pydevd, the debugger library PyCharm uses.
  • Embedded server: PyCharm launches the single Python process that runs your script and the engine. You debug it locally, the same way you'd debug a pip installation locally, with no remote debugging server. For code you type into the console, you still call pydevd.settrace, but with no host or port (step 6).

The build and project setup are similar to the Gradle source-build debugging guide: you build wheels from a local clone into a virtual environment, then open the clone as a PyCharm project.

Prerequisites

This guide assumes you have a local clone of Deephaven Core, as described in the guide on building Deephaven from source. On Windows, the clone, the virtual environment, and the build must live inside WSL 2, so run every command in this guide from your WSL distribution.

The embedded server starts a JVM inside your Python process, so set the JAVA_HOME environment variable to your JDK 21 installation before you run the launch script you write in step 3. Step 4 shows how to set it in PyCharm if PyCharm doesn't pick it up.

Building the wheels requires Docker, because the wheel build runs inside a container. Make sure Docker is installed and the Docker daemon is running before you run the ./gradlew commands in step 1. See the Docker prerequisite in the guide on building Deephaven from source for the required version.

You also need PyCharm. On macOS and Linux, the free tier of PyCharm works, because this setup uses only local debugging. Before 2025.1, the free tier was PyCharm Community Edition. On Windows, this setup uses a WSL interpreter, which requires PyCharm Pro.

Warning

Use PyCharm 2024 or later. Debugging Deephaven with PyCharm 2023 has known problems.

1. Build and install the wheels

From the root of the deephaven-core clone, create and activate a virtual environment:

The rest of this guide calls this virtual environment embedded-dh-venv. If you use a different name, substitute it.

Then, build the Python server wheel and the embedded server wheel:

Install both wheels into embedded-dh-venv with one command. The embedded server wheel, deephaven_server, requires exactly the same version of deephaven-core, which is the py-server wheel you just built. When you pass both files to one pip install, pip takes deephaven-core from your local build instead of PyPI:

Replace <version> with the version in the built wheel filenames. To see them, run ls py/server/build/wheel/ py/embedded-server/build/wheel/. Both wheels have the same version.

Also install pydevd. Step 6 uses it to set programmatic breakpoints:

2. Create a PyCharm project

Create a PyCharm project from the deephaven-core clone used to build the wheels in step 1. From PyCharm's launch window, click Open and navigate to the deephaven-core directory. Opening the clone lets you edit Deephaven's source alongside your launch script and rebuild the wheels from the same project.

Set the project interpreter to embedded-dh-venv. Open PyCharm > Settings on macOS, or File > Settings on Windows and Linux. Then go to Project: deephaven-core > Python Interpreter, and select the interpreter in <deephaven-core>/embedded-dh-venv/bin/python if it isn't already selected.

On Windows, add the WSL 2 interpreter with Add New Interpreter > On WSL, and select embedded-dh-venv/bin/python inside your WSL distribution.

3. Write a launch script

Create a script, such as script.py, that starts the embedded server:

The jvm_args list passes these options to the embedded JVM:

  • -Xmx4g sets the JVM's maximum heap to 4 GB. Raise or lower it to fit your data and your machine.
  • -DAuthHandlers=io.deephaven.auth.AnonymousAuthenticationHandler enables anonymous authentication, so you can open the Deephaven web IDE without a pre-shared key. Debugging doesn't depend on it.
  • -Dprocess.info.system-info.enabled=false turns off collection of operating system and hardware information at startup. Debugging doesn't depend on it, but on M2 Macs keep it, as the pip install guide recommends.

The empty_table line stands in for your own code. Set a breakpoint on it to check that the debugger stops in the launch script. Import from deephaven only after you create the Server, because the deephaven package needs the JVM that Server starts.

Anonymous authentication provides no security, so use it only on a server bound to localhost, as host="localhost" does here.

Without the AuthHandlers argument, the server uses pre-shared key authentication with a random key. A server started from a Python script doesn't print that key to the console, so you can't look it up. To use a pre-shared key you choose, replace the AuthHandlers argument with -Dauthentication.psk=<your-key>. Enter that key when you open the web IDE in step 6.

The while True loop at the end of the launch script keeps the Python process running, which keeps the embedded server and the debugger session alive. Without the loop, the process exits when the script reaches its last line.

The loop also matters when you debug a function called from a ticking table. Deephaven runs ticking-table updates on its own threads, so the process must still be running when an update reaches your code. See Ticking tables when your script starts the server.

4. Create a run configuration

  1. Go to Run > Edit Configurations.
  2. Click the + button and select Python.
  3. Name the configuration (e.g., "Embedded Server Debug").
  4. Set Script path to the launch script from step 3 (e.g., script.py), and set the interpreter to embedded-dh-venv.
  5. If PyCharm doesn't pick up JAVA_HOME from your environment, add JAVA_HOME under Environment variables, set to your JDK 21 installation.
  6. Click OK to save the configuration.

5. Debug the launch script

Set breakpoints in the launch script before you start the debugger, because its code runs as soon as the process starts. Because everything runs in the interpreter PyCharm is debugging, PyCharm stops at breakpoints in your own code and in Deephaven's Python source.

When you install the wheels, pip copies Deephaven's Python modules into the site-packages directory of embedded-dh-venv. That installed copy is the code that runs, so set breakpoints in Deephaven's source there. Open it under External Libraries in PyCharm's Project tool window, not py/server in the clone.

Then select the run configuration you created in step 4, such as "Embedded Server Debug", and click the Debug button. PyCharm launches the script in a Python process and attaches its debugger to that process before any Deephaven or JVM code runs.

If you change py/server or Deephaven's Java code, rebuild and reinstall the wheels (step 1) before you debug again. The embedded server wheel contains the server's Java JARs, so Java changes take effect only after you rebuild it.

6. Debug code from the Deephaven web console

Once the server starts, open the Deephaven web IDE at http://localhost:10000/ide/ and run code in its console. If you set a pre-shared key in step 3, enter it when the web IDE asks for it.

Because the embedded server runs in the same process as your launch script, code executed in the console runs in that same interpreter. Console commands run on Deephaven's server threads rather than on your script's main thread. Deephaven registers those threads with the debugger, so PyCharm stops at the breakpoints you set.

Code you type into the console, such as a user-defined function that a query string calls, isn't in a file open in PyCharm, so you can't click in the margin to set a breakpoint in it. Call pydevd.settrace in that code as a programmatic breakpoint instead:

The console and your launch script share the same debugged process, so PyCharm's debugger is already attached. That's why pydevd.settrace doesn't need a host or port here.

PyCharm pauses each time udf runs, once for each of the table's 10 rows. Click Resume Program to move to the next row.

While execution is paused in a command you ran from the console, that command holds the update graph's lock. Ticking tables stop updating, and other console commands wait, until you resume. The tables then catch up.

Deephaven has some debugging behaviors of its own, such as lazily evaluated table operations and ticking-table updates on other threads. Check out Common problems before you dig into a breakpoint that won't hit.

Debug Deephaven's Java engine

PyCharm's debugger reaches only Python code. Deephaven's query engine is Java, so PyCharm can't step into table operations or the update graph. To debug the Java engine in the same process, add a JDWP agent to jvm_args in your launch script:

The address=127.0.0.1:5005 setting makes the JVM listen for a debugger on port 5005 of your own machine only. Then attach a JVM debugger, such as an IntelliJ IDEA Remote JVM Debug configuration, to port 5005. Remote JVM Debug is available in every edition of IntelliJ IDEA, including the free one.

Troubleshooting

Gradle build fails

Problem: ./gradlew :py-embedded-server:assemble fails with errors.

Solutions:

  • Make sure Docker is installed and the Docker daemon is running. The wheel builds run in a container.
  • Verify you can build the full source-based server first. The embedded server wheel depends on the same build infrastructure.
  • Check that you're running the Gradle command from the deephaven-core repository root.

pip install fails or pulls the wrong version

Problem: Installing the deephaven_server wheel tries to download deephaven-core from PyPI instead of using your local build, or fails because no matching version is published.

Solutions:

  • Pass the locally built deephaven_core wheel from py/server/build/wheel/ and the deephaven_server wheel to the same pip install command, as shown in step 1.

Breakpoints in Deephaven source don't bind

Problem: Breakpoints set in files under py/server/deephaven appear grayed out or never hit.

Solutions:

  • Set breakpoints in Deephaven's Python source in the copy installed in embedded-dh-venv, not in py/server in the clone. Python runs the installed wheel from site-packages, not the source in the clone. PyCharm lists the installed copy under External Libraries in its Project tool window. See step 5.

PyCharm can't find Deephaven source files

Problem: When debugging, PyCharm shows "Source code not available" or can't find files.

Solutions:

Console code isn't hitting breakpoints

Problem: Breakpoints work in your launch script but not in code run from the console.

Solutions:

Breakpoints in a function called from a query never hit

Problem: A breakpoint in a function that a table operation calls is never reached.

Solutions: