Core+ Python Client

The Deephaven Core+ client allows you to connect to Persistent Queries using the Community Core engine in the Deephaven Enterprise system. You may either connect to a new temporary Persistent Query, or one of the Persistent Queries that you have access to on the server. This page provides a brief overview of using the Python client; more details are available in the reference documentation.

The Python client wheel is located on a Deephaven server in /usr/illumon/coreplus/latest/py/wheel/, with a name of the form deephaven_coreplus_client-1.20231218.432-py3-none-any.whl. To use the wheel, use pip (e.g., pip install deephaven_coreplus_client-1.20231218.432-py3-none-any.whl). pip automatically installs dependencies referenced by the wheel, including the underlying pydeephaven client from Deephaven Community Core.

The Core+ client communicates with the authentication server, Persistent Query controller, and Core+ workers. The underlying communications protocol is gRPC and Deephaven does not make incompatible controller or authentication changes within a major release. Grizzly (1.20240517) controller and authentication clients can connect to either Grizzly or Vermilion+ (1.20231218) clusters. Vermilion+ clients with Envoy can connect to a Grizzly cluster. A Vermilion+ client after 1.20231218.397 is required to connect to a Grizzly cluster without Envoy. Older clients may not have all the features that newer clients do (e.g., replicas are not supported by Vermilion+ clients communicating with a Grizzly cluster). To communicate with a worker, you must use a client version that corresponds to the Community Core version of the worker.

The first step in either case is to create a SessionManager, the Deephaven installation exposes a connection.json file that is the simplest way to create a session. Simply specify the URL from your Deephaven installation. Alternatively, you may provide a JSON string, using the json= named parameter.

from deephaven_enterprise.client.session_manager import SessionManager

connection_info = "https://deephaven-host:8000/iris/connection.json"
session_mgr: SessionManager = SessionManager(connection_info)

The host and port are the same ports used to configure launcher instances. Typically, this means port 8000 when your system is configured to use Envoy or port 8123 without Envoy.

pip3.8 install certifi
/Applications/Python\ 3.8/Install\ Certificates.command

After the session is created, you may authenticate with a password:

session_mgr.password("username", "password")

Or using a private key:

session_mgr.private_key("/path-to-private-key/priv-username.base64.txt")

At this point, you can then connect to an existing Persistent Query. The session object extends from the Deephaven Community pydeephaven.session.Session. The Deephaven Community Python API documentation provides more information on interacting with the worker session and tables.

session = session_mgr.connect_to_persistent_query("Community Python")
qc = session.open_table("tt")
print("Table size: ", qc.size)
session.close()

Instead of connecting to an existing Persistent Query, you can instead make a new temporary Persistent Query to host your worker.

session = session_mgr.connect_to_new_worker(name=None, heap_size_gb=1.0)

# We can run arbitrary script code on the worker, which creates the variable tt
session.run_script(
    "from deephaven import time_table\n"
    'tt = time_table("PT1s").update("Remote=false")'
)
# Code runs on the server, so you will not see "Hello, world" locally, rather you
# would find it in the ProcessEventLog table that captures the worker's stdout
session.run_script('print("Hello, world")')

# We can also create tables and execute calls by manipulating Session and table objects.
# Here session.time_table call creates a time_table (in the worker) with a period of one second.
# There is an intermediate Table, that we then pass into the update method, which again creates
# a table on the server with a column named "Remote".  We assign the result to a local variable,
# which references the server-side table.
table = session.time_table(period=1000_000_000).update(["Remote=true", "Value=1"])
# The bind_table call inserts the table referenced by our local variable into the server's
# Python scope, thus making it available from the UI (or other script calls).
session.bind_table("remote", table)

# We can operate on the remote table server side
session.run_script('remote_sum = remote.view(["Value"]).sum_by()')
# Retrieve results by name
sum_table = session.open_table("remote_sum")
# And convert them to arrow
print(sum_table.to_arrow())

session.close()

You can interact with the session through the Community Python API and view tables with the Deephaven Web UI. By default, the temporary queries are stopped and deleted after you close the session object.

In-Worker Use

The Deephaven Core+ Python client is installed into a Python worker's virtual environment. This enables you to create a SessionManager object and access the provided controller client to manage persistent queries. If no connection_info parameter is provided, then the SessionManager connects to and authenticates with the current cluster. For example, the following snippet prints the names of running queries on the current cluster:

from deephaven_enterprise.client.session_manager import SessionManager

session_mgr: SessionManager = SessionManager()
session_mgr.controller_client.map()

for x in session_mgr.controller_client.map().values():
    if session_mgr.controller_client.is_running(x.state.status):
        print(x.config.name)

Within a worker context, you may use a Core+ session to download local subscriptions or snapshots of table references over the Barrage protocol via barrage_subscribe and barrage_snapshot. The results from barrage_subscribe and barrage_snapshot can interact with other local tables in the worker. Additionally, you can get a reference to the Barrage session itself for custom control, as seen in Core documentation.

from deephaven_enterprise.client.session_manager import SessionManager

session_mgr = SessionManager()
session = session_mgr.connect_to_new_worker(name=None, heap_size_gb=1.0)
table_ref = session.live_table("DbInternal", "ProcessEventLog").where("Date=today()")
table_local_subscribe = session.barrage_subscribe(table_ref)
table_local_snapshot = session.barrage_snapshot(table_ref)

barrage_session = session.barrage_session()

Related documentation

• Installing Python
• Use Deephaven Core+ from Jupyter