Core+ Python Client

The Deephaven Core+ client allows you to connect to Persistent Queries using the Community Core engine in the Deephaven Enterprise system. You may either connect to a new temporary Persistent Query, or one of the Persistent Queries that you have access to on the server. This page provides a brief overview of using the Python client; more details are available in the reference documentation.

The Python client wheel is located on a Deephaven server in /usr/illumon/dnd/latest/py/wheel/, with a name of the form deephaven_coreplus_client-1.20231218.432-py3-none-any.whl. To use the wheel, use pip (e.g., pip install deephaven_coreplus_client-1.20231218.432-py3-none-any.whl). pip automatically installs dependencies referenced by the wheel, including the underlying pydeephaven client from Deephaven Community Core.

The first step in either case is to create a SessionManager, the Deephaven installation exposes a connection.json file that is the simplest way to create a session. Simply specify the URL from your Deephaven installation. Alternatively, you may provide a JSON string, using the json= named parameter.

from deephaven_enterprise.client.session_manager import SessionManager

connection_info = "https://deephaven-host:8000/iris/connection.json"
session_mgr: SessionManager = SessionManager(connection_info)

The host and port are the same ports used to configure launcher instances. Typically, this means port 8000 when your system is configured to use Envoy or port 8123 without Envoy.

pip3.8 install certifi
/Applications/Python\ 3.8/Install\ Certificates.command

After the session is created, you may authenticate with a password:

session_mgr.password("username", "password")

Or using a private key:

session_mgr.private_key("/path-to-private-key/priv-username.base64.txt")

At this point, you can then connect to an existing Persistent Query. The session object extends from the Deephaven Community pydeephaven.session.Session. The Deephaven Community Python API documentation provides more information on interacting with the worker session and tables.

session = session_mgr.connect_to_persistent_query("Community Python")
qc = session.open_table("tt")
print("Table size: ", qc.size)
session.close()

Instead of connecting to an existing Persistent Query, you can instead make a new temporary Persistent Query to host your worker.

session = session_mgr.connect_to_new_worker(name=None, heap_size_gb=1.0)

# We can run arbitrary script code on the worker, which creates the variable tt
session.run_script('from deephaven import time_table\n'
                   'tt = time_table("PT1s").update("Remote=false")')
# Code runs on the server, so you will not see "Hello, world" locally, rather you
# would find it in the ProcessEventLog table that captures the worker's stdout
session.run_script('print("Hello, world")')

# We can also create tables and execute calls by manipulating Session and table objects.
# Here session.time_table call creates a time_table (in the worker) with a period of one second.
# There is an intermediate Table, that we then pass into the update method, which again creates
# a table on the server with a column named "Remote".  We assign the result to a local variable,
# which references the server-side table.
table = session.time_table(period=1000_000_000).update(["Remote=true", "Value=1"])
# The bind_table call inserts the table referenced by our local variable into the server's
# Python scope, thus making it available from the UI (or other script calls).
session.bind_table("remote", table)

# We can operate on the remote table server side
session.run_script("remote_sum = remote.view([\"Value\"]).sum_by()")
# Retrieve results by name
sum_table = session.open_table("remote_sum")
# And convert them to arrow
print(sum_table.to_arrow())

session.close()

You can interact with the session through the Community Python API and view tables with the Deephaven Web UI. By default, the temporary queries are stopped and deleted after you close the session object.

In-Worker Use

The Deephaven Core+ Python client is installed into Python worker's virtual environment as of Deephaven 1.20231218.190. This enables you to create a SessionManager object and access the provided controller client to manage persistent queries. If no connection_info parameter is provided, then the SessionManager connects to and authenticates with the current cluster. For example, the following snippet prints the names of running queries on the current cluster:

from deephaven_enterprise.client.session_manager import SessionManager

session_mgr: SessionManager = SessionManager()
session_mgr.controller_client.map()

for x in session_mgr.controller_client.map().values():
        if session_mgr.controller_client.is_running(x.state.status):
		    print(x.config.name)

Jupyter

The Deephaven Python client can be used in conjunction with Jupyter. To do so, you must install the deephaven-ipywidgets package and the Deephaven client wheel file (e.g., deephaven_coreplus_client-1.20231218.432-py3-none-any.whl) into your environment. For example, to create a virtual environment for JupyterLab, you could run:

python -mvenv ./jupyter-venv
./jupyter-venv/bin/pip install ./deephaven_coreplus_client-1.20231218.432-py3-none-any.whl deephaven-ipywidgets jupyterlab

To start JupyterLab, then run:

./jupyter-venv/bin/jupyter lab

JupyterLab automatically launches a new tab in your web browser. For example, it may start a server on port 8888, which you can then access as "http://localhost:8888/lab/tree/Untitled.ipynb". The Python client's session manager and other objects work identically as outside JupyterLab, but the deephaven-ipywidgets library provides the DeephavenWidget function that displays the same responsive table grid as is available in the Deephaven Web UI. Most functionality, like sorting and filtering, and organizing columns is available. However, dashboarding features like linked tables require the Deephaven web UI.

For example, the following script loads the DbInternal.PersistentQueryStateLog table and displays it as a Grid:

from deephaven_enterprise.client.session_manager import SessionManager
from deephaven_ipywidgets import DeephavenWidget

connection_info = "https://hostname:8123/iris/connection.json"
session_mgr: SessionManager = SessionManager(connection_info)

session_mgr.private_key("/path-to-private-key/priv-username.base64.txt")

session = session_mgr.connect_to_new_worker(name=None, heap_size_gb=1.0)

session.run_script("""live_table = db.live_table("DbInternal", "PersistentQueryStateLog").where("Date=today()")""")

display(DeephavenWidget(session.open_table("live_table")))

# Note: executing the session.close() terminates the connection used by the widget.
session.close()

In your notebook, you may split operations into distinct cells. If you include session.close() in the same cell as creating the widget, then you cannot see the table, because the widget requires the worker to be running in order to display results. Executing the script as several cells produces a result like the following:

Displaying Server Side Logs in Jupyter

Any logging from the session.run_script commands exists on the server in the ProcessEventLog and can not be seen in the Jupyter client. The jupyter_logging package, available at /usr/illumon/dnd/latest/py/wheel/deephaven_jupyter_logging-1.20231218.432-py3-none-any.whl, contains the LoggedSession class, which fetches the STDOUT and STDERR server logs from the ProcessEventLog for the given commands.