Python environment

Deephaven seamlessly integrates with Python, enabling users to harness Python's capabilities for data analysis, manipulation, automation, and much more.

There are several ways to manage custom Python virtual environments (venvs) in Deephaven. This guide demonstrates how to:

  • Programmatically inspect the Python venv of the worker process running the current Code Studio or Persistent Query (PQ), and how to install additional packages within it.
  • Customize the default Python venv used for Code Studios and PQs.

Note

This guide does not cover customizing the Python venv when starting a new Code Studio or PQ. For more details on those, see:

Manage venvs programmatically

You can manage venvs programmatically through the deephaven_enterprise.venv module.

Get the path of the current virtual environment

To retrieve the path of the current venv, call get_venv_path:

from deephaven_enterprise.venv import get_venv_path

print(get_venv_path())

The output will be a string representing the path to the virtual environment, similar to:

/db/TempFiles/dbquery/db_query_server/cache/classes/RemoteQueryDispatcher/clients/qa-grizzly-java17-cluster-infra-1.c.illumon-eng-170715.internal_45/workers/worker_39081bbe_classes/coreplus-venv-39081bbe-0468-434e-a07c-c55686dde6ee

Install additional packages

You can install Python packages at runtime using the install function from the deephaven_enterprise.venv module. Here is an example:

import deephaven_enterprise.venv

try:
    deephaven_enterprise.venv.install(["pandas-ta"])
    print("pandas-ta installed successfully.")
except Exception as e:
    print(f"An error occurred: {e}")

This script attempts to install the pandas-ta package and prints a success message if the installation is successful. If an error occurs during the installation, it catches the exception and prints an error message.

To install packages, the worker must be running in a temporary virtual environment or on a Kubernetes cluster. If you need to install a package incompatible with the default packages, uncheck the Include default packages checkbox when starting a new Code Studio session or configuring a new PQ. Ensure that you include alternatives to many of the default packages for the worker to function properly. In particular, the Deephaven Enterprise worker wheel and the deephaven-core package are essential:

  • The worker wheel is located at /usr/illumon/coreplus/latest/py/wheel/deephaven_coreplus_worker-1.20240517.344-py3-none-any.whl.
  • The list of standard requirements for an installation is located at /usr/illumon/coreplus/latest/py/resources/requirements.txt.

If you have a running worker, you can also retrieve them by running the following code in the console.

import subprocess
from deephaven_enterprise.venv import get_venv_path

cmd = get_venv_path() + "/bin/pip"
result = subprocess.run([cmd, "freeze"], capture_output=True, text=True)
print(result.stdout)

Install custom Python virtual environments

If DH_DND_PYTHON is set to true in your cluster.cnf file, then a default Python virtual environment is configured for Core+ workers. You can also create alternative Python virtual environments and assign them to your own custom worker kinds.

To install a Python environment for the Core+ worker, you must create a virtual environment and configure it properly for the installed Community version. Deephaven recommends placing them in /usr/illumon/coreplus/venv/. In the following example, we create a default virtual environment in the /usr/illumon/coreplus/venv/0.37.4-custom directory. The requirements.txt file provides a list of packages and versions that Deephaven has tested with this distribution. You should use it as a starting point for adding, removing, or replacing packages. Deephaven does not test our integrations with package versions other than those listed in requirements.txt. In addition to the requirements.txt file which references packages available via pip, you must also install the Deephaven Core+ worker wheel and it is recommended that you install the Deephaven Core+ client wheel.

Note

In the snippets below, be sure to change the path to match your Core+ install and select the proper Community pip package version.

The following snippet installs a custom Python virtual environment with pip:

# Replace the paths below where you wish to a) install the virtual environment,
# b) take the requirements.txt file (which specifies Python dependencies) from,
# and c) source the appropriate Enterprise Core+ wheels.
#
# While not required, Deephaven recommends that you install the virtual environment
# inside the Core+ package directory for each specific version.
venvPath=/usr/illumon/coreplus/venv/0.37.4-custom
sudo -u irisadmin python3.10 -m venv $venvPath
sudo -u irisadmin /bin/bash -c "source $venvPath/bin/activate && \
pip3.10 install -r /usr/illumon/coreplus/latest/py/resources/requirements.txt && \
pip3.10 install /usr/illumon/coreplus/latest/py/wheel/deephaven_coreplus_worker-1.20240517.344-py3-none-any.whl && \
pip3.10 install /usr/illumon/coreplus/latest/py/wheel/deephaven_coreplus_client-1.20240517.344-py3-none-any.whl"

If, instead, you prefer to use conda, refer to the snippet below to install a custom Python virtual environment:

export PYTHONNOUSERSITE=True
conda create --prefix /usr/illumon/coreplus/venv/corePlusConda python=3.10
conda activate /usr/illumon/coreplus/venv/corePlusConda
/usr/illumon/coreplus/venv/corePlusConda/bin/pip3 install -r /usr/illumon/coreplus/latest/py/resources/requirements.txt
/usr/illumon/coreplus/venv/corePlusConda/bin/pip3 install /usr/illumon/coreplus/latest/py/wheel/deephaven_coreplus_worker-1.20240517.344-py3-none-any.whl
/usr/illumon/coreplus/venv/corePlusConda/bin/pip3 install /usr/illumon/coreplus/latest/py/wheel/deephaven_coreplus_client-1.20240517.344-py3-none-any.whl

You must create the virtual environment on each query server.