Core+ Python Client in Jupyter

Deephaven Enterprise Core+ with Jupyter allows you to use Deephaven and view tables and plots directly in Jupyter notebooks. To connect Enterprise Core+ and Jupyter, use the Core+ Python client from Jupyter to connect to a running Deephaven server.

Installation

Installing and running the Deephaven Python client in conjunction with Jupyter is a simple process. You will need to:

Install Jupyter
Install the Deephaven client wheel file in your Jupyter virtual environment

Install Jupyter

Note

We recommend using a Python virtual environment to decouple and isolate Python installs and associated packages.

Deephaven can operate in JupyterLab or Jupyter Notebook - the choice is yours! Both are installed with pip:

# install JupyterLab
pip install jupyterlab
# or install Jupyter Notebook
pip install notebook

Locate and install the Deephaven client wheel file

Every Deephaven server hosts the Python client wheel file that you must install in your Jupyter virtual environment to connect to the server from Jupyter. Find the wheel at /usr/illumon/coreplus/latest/py/wheel/, with a name of the form deephaven_coreplus_client-<version_labels>-py3-none-any-whl.

To find the wheel file on the server you want to connect to, SSH to the server:

ssh -i path/to/private_key *server_name*

Then, navigate to the wheel file location:

cd /usr/illumon/coreplus/latest/py/wheel/

Secure copy the file to your machine:

scp *user*@*server_name*:/usr/illumon/coreplus/latest/py/wheel/*file*.whl *user*@*destination_server_name*:/path/to/file

Install the Deephaven client wheel file in your Jupyter virtual environment

Now that you have the correct wheel file for your Deephaven Core+ client, you can install it in your Jupyter virtual environment:

python -mvenv ./jupyter-venv
./jupyter-venv/bin/pip install *wheel_file* deephaven-ipywidgets jupyterlab

Start Jupyter

You're ready to get up and running! Start Jupyter in your virtual environment:

./jupyter-venv/bin/jupyter lab

JupyterLab automatically launches a new tab in your web browser. You can access it at "http://localhost:8888", which will redirect you to your JupyterLab's homepage.

Usage

The Python client's session manager and other objects work identically as outside JupyterLab, but the deephaven-ipywidgets library provides the DeephavenWidget function that displays the same responsive table grid as is available in the Deephaven Web UI. Most functionality, like sorting, filtering, and organizing columns is available. However, dashboarding features like linked tables require the Deephaven web UI.

Regardless of what you want to do with Deephaven in Jupyter, the first step is to connect to a worker.

Connect to a Deephaven worker

To connect to Deephaven from Jupyter, you need to create a SessionManager object and connect to the server. The SessionManager object is the primary interface for interacting with Deephaven from Jupyter. Open a new notebook and run:

# Import the necessary modules
from deephaven_enterprise.client.session_manager import SessionManager
from deephaven_ipywidgets import DeephavenWidget

# Create a SessionManager
session_mgr: SessionManager = SessionManager(
    "https://<server-name>:8000/iris/connection.json"
)

# Use the `password` method to authenticate with the server
session_mgr.password("<username>", "<password>")

# Connect to a new worker
session = session_mgr.connect_to_new_worker(
    name=None, heap_size_gb=1.0
)  # `name=None` assigns an autogenerated name to the worker based on the time it was created

You're connected! Now you can run scripts on the server, fetch tables from the connected worker or from Persistent Queries, and create new tables - and of course, display all of the above in the notebook. See the Python client guide for more information on what the Python client can do.

Run a script

You can use the SessionManager's run_script method to execute Deephaven Core+ scripts on the server. Then, use display to show the results in the notebook. For example, to display a historical table from the LearnDeephaven database:

session.run_script(
    """historical_table = db.historical_table("LearnDeephaven", "StockTrades").where("Date=`2017-08-25`")"""
)

display(DeephavenWidget(session.open_table("historical_table")))

The Deephaven widget displayed in a Juypterlab notebook after running run_script

Finally, you can close the connection to the server when you're done:

# Note: executing the session.close() terminates the connection used by the widget.
session.close()

Caution

Running session.close() in the same cell as creating the widget will prevent you from seeing the table, as the widget requires the worker to be running to display results.

Executing the script as several cells produces a result like the following:

Several cells in a Jupyterlab notebook display code, while the final block creates a Deephaven widget with a table

Fetch a table from a namespace

You can use the SessionManager's catalog method to list all of your connected worker's accessible namespaces and tables:

from deephaven_enterprise.client.session_manager import SessionManager
from deephaven_enterprise.client.session_manager import DndSession
from deephaven_ipywidgets import DeephavenWidget

connection_info = "https://qa-grizzly-java17-cluster-infra-1.int.illumon.com:8000/iris/connection.json"
session_mgr: SessionManager = SessionManager(connection_info)

session_mgr.password("demoUser", "demo")

# Connect to a new worker
session = session_mgr.connect_to_new_worker(
    name=None, heap_size_gb=1.0
)  # `name=None` assigns an autogenerated name to the worker based on the time it was created

catalog = session.catalog_table()

display(DeephavenWidget(catalog))

The Deephaven widget displays the catalog of all the connected worker's accessible namespaces and tables

You can use historical_table or live_table to fetch a table from the namespace. For example, to fetch StockTrades from the LearnDeephaven namespace:

table = session.historical_table("LearnDeephaven", "StockQuotes").where(
    "Date = `2017-08-25`"
)

display(DeephavenWidget(table))

The "StockQuotes" table from the "LearnDeephaven" namespace

Create and manipulate tables

You can create empty tables and time tables with the SessionManager's empty_table and time_table methods, and use all of the Deephaven methods you are familiar with to manipulate data:

from deephaven_enterprise.client.session_manager import SessionManager
from deephaven_ipywidgets import DeephavenWidget

connection_info = "https://<server-name>:8000/iris/connection.json"
session_mgr: SessionManager = SessionManager(connection_info)

session_mgr.password("demoUser", "demo")

session = session_mgr.connect_to_new_worker(name=None, heap_size_gb=1.0)

table = session.empty_table(4).update("X = i")

time_table = session.time_table("PT1S").update("X = i")

display(DeephavenWidget(table))
display(DeephavenWidget(time_table))

# Note: executing the session.close() terminates the connection used by the widget.
session.close()

A static table and a time table displayed in the Deephaven widget

Connect to a Persistent Query

A SessionManager can fetch a Persistent Query via the connect_to_persistent_query method. You can then use open_table to open tables:

from deephaven_enterprise.client.session_manager import SessionManager
from deephaven_enterprise.client.session_manager import DndSession
from deephaven_ipywidgets import DeephavenWidget

connection_info = "https://qa-grizzly-java17-cluster-infra-1.int.illumon.com:8000/iris/connection.json"
session_mgr: SessionManager = SessionManager(connection_info)

session_mgr.password("demoUser", "demo")

session = session_mgr.connect_to_persistent_query("test_pq")

table_from_pq = session.open_table("crypto_table")

display(DeephavenWidget(table_from_pq))

# Note: executing the session.close() terminates the connection used by the widget.
session.close()

A table retrieved from a PQ in the Deephaven widget