Package a Deephaven Python project

Python packaging turns a project containing Python code into a distribution that can be installed with pip. A distribution can provide an importable library, command-line tools, or both. This guide shows how to package Python projects that use Deephaven. It follows the conventions of the Python Packaging User Guide and uses setuptools as the build backend.

A packaged project uses Deephaven in one of three ways. The choice determines which Deephaven package it depends on:

Inside a running server: The package is installed into a Deephaven server's own Python environment, such as a custom Docker image or the virtual environment of a pip-installed server, and imported from the server's console or scripts. Depend on deephaven-core, which provides the server-side deephaven API without a server of its own.
Inside an embedded server: deephaven-server starts a Deephaven server and its JVM inside the Python process. It includes deephaven-core. The server-side deephaven API is available in that same process once a Server object has been created, which starts the JVM.
Through a remote client: pydeephaven connects a Python process to a Deephaven server that is already running, such as one started with Docker or pip-installed deephaven-server in another process. The client does not start a server and does not provide the server-side deephaven API; it works with tables through a Session.

The packaging tooling is the same in all three cases. What differs is the dependency you declare and, for the server-side deephaven API, the order in which modules are imported. This guide shows a library and a command-line package for the embedded-server and remote-client models, and a library for the running-server model.

The code in this guide is condensed from the deephaven-python-packaging repository, which contains complete, runnable versions of each package along with sample data and a README for each. The repository versions add input validation and error handling; the snippets here leave those out to stay focused on packaging. Clone it to try the examples or to use one as a starting point:

git clone https://github.com/deephaven-examples/deephaven-python-packaging.git
cd deephaven-python-packaging

Python packaging terms

A Python project is the source code and configuration you develop. Building it produces a distribution package, usually a wheel (.whl) or source distribution, which users install with pip. An import package is the code users import in Python. Its name can differ from the distribution name: for example, a project distributed as my-dh-cli can provide the import package my_dh_cli. The Python Packaging Glossary defines these and other packaging terms.

The standard project configuration file is pyproject.toml. Its [project] table is defined by the pyproject.toml specification, and the Python Packaging User Guide's Writing your pyproject.toml walks through each field. Setuptools-specific options, such as package discovery, are documented in the setuptools pyproject.toml configuration guide. The examples use the src layout, which keeps importable code in src/:

my_project/
├── src/
│   └── my_package/
│       ├── __init__.py
│       └── module.py
├── pyproject.toml
└── README.md

Package a library for a running server

A library package provides functions for another program to import. It does not start a server itself; that is the caller's job. Depend on deephaven-core, which provides the deephaven module without a server of its own. A Deephaven server already has it, so installing the library into a server adds nothing else and does not change the server's Deephaven version. The full example is my_dh_library in the example repository.

The following configuration uses setuptools and its automatic package discovery to find the import package under src/. The [build-system] and [tool.setuptools.packages.find] tables are the same for every example in this guide, so later listings show only the tables that differ:

[build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"

[project]
name = "my-dh-library"
version = "0.1.0"
description = "Reusable Deephaven query functions"
readme = "README.md"
requires-python = ">=3.9"
dependencies = [
  "deephaven-core>=42.7",
]

[tool.setuptools.packages.find]
where = ["src"]

Set the Python version and dependency constraints to versions you test and support. A lower bound such as deephaven-core>=42.7 declares compatibility; it does not lock the environment to a repeatable set of versions. If you need that, use a lock file or a pinned requirements file alongside the package. The syntax for these constraints is defined by the dependency specifiers and version specifiers specifications.

Because a library is only imported inside a running server or after the calling program has created a Server, its modules can import deephaven at the top level. For example, src/my_dh_library/queries.py might contain:

from deephaven.table import Table


def filter_by_threshold(table: Table, column: str, threshold: float) -> Table:
    """Keep rows where column > threshold."""
    return table.where([f"{column} > {threshold}"])

Install the library into the same environment as a Deephaven server, then use it from the server's console or scripts. There is no Server to create because the code already runs inside one:

from my_dh_library.queries import filter_by_threshold
from deephaven import read_csv

data = read_csv("data/sample.csv")
filtered = filter_by_threshold(data, "Score", 75.0)

The same library also works from a standalone Python process that starts its own embedded server with deephaven-server. The calling program creates the Server first:

from deephaven_server import Server

server = Server(port=10001, jvm_args=["-Xmx4g"])
server.start()

from deephaven import read_csv
from my_dh_library.queries import filter_by_threshold

scores = read_csv("data/sample.csv")
high_scores = filter_by_threshold(scores, "Score", 75.0)

Package code for an embedded server

Use deephaven-server when the Python program needs to start a Deephaven server itself. This is the right fit for a self-contained tool, a batch job, or a script that should not depend on a separately running Deephaven server.

The embedded server runs a JVM, so every machine that runs the package needs Java 17 or later installed, with JAVA_HOME set. pip cannot install Java, and pyproject.toml has no way to declare it as a dependency, so state this requirement in your package's README. See the pip install prerequisites for details.

One rule shapes every embedded-server package: deephaven cannot be imported until a deephaven_server.Server has been constructed in the same process. Constructing Server starts the JVM, and deephaven checks for a running JVM when it is imported, so an import that happens any earlier fails. Calling server.start() is a separate step that starts the server itself; the import does not depend on it. In practice this means that a package that defines commands must keep deephaven out of module scope in __init__.py and in the command modules themselves, and import it inside the function, after the Server is constructed.

Embedded-server command

A command-line program that uses the embedded server is a complete, standalone application: when a user runs it, it starts a server, does its work, and exits. The full example is my_dh_cli in the example repository. The layout is:

my_dh_cli/
├── src/
│   └── my_dh_cli/
│       ├── __init__.py
│       ├── __main__.py
│       └── cli.py
└── pyproject.toml

The package metadata declares the runtime dependencies and the installed command. The examples use Click for argument parsing, so it is listed alongside deephaven-server. The [project.scripts] table defines a console script entry point; pip generates an executable with the given name when the package is installed. The entry point must name a callable that exists in the package:

[project]
name = "my-dh-cli"
version = "0.1.0"
description = "A command-line Deephaven CSV query"
requires-python = ">=3.9"
dependencies = ["deephaven-server>=42.7", "click"]

[project.scripts]
my-dh-cli-query = "my_dh_cli.cli:main"

Python imports the package before it resolves the console-script entry point, so __init__.py and cli.py must not import deephaven at module scope; if they did, every command in the package would fail at launch. Keep __init__.py empty or limited to imports that do not depend on Deephaven, and in cli.py import Server and deephaven inside main(), after argument parsing:

import click


@click.command()
@click.argument("input_file", type=click.Path(exists=True, dir_okay=False))
@click.option(
    "--port", default=10001, show_default=True, help="Port for the embedded server"
)
def main(input_file: str, port: int) -> None:
    """Read a CSV file and add a computed column."""
    from deephaven_server import Server

    server = Server(port=port, jvm_args=["-Xmx4g"])
    server.start()

    from deephaven import read_csv

    table = read_csv(input_file)
    result = table.update(["DoubleScore = Score * 2"])
    click.echo(f"Processed {result.size} rows")

Server itself can be imported at any time; only deephaven has to wait until a Server has been constructed. The example calls server.start() before the import because the command needs a running server anyway, not because the import requires it. The process owns the server and JVM for as long as it runs. If you want a command that works against a server you already have running, see Package a remote Deephaven client instead.

Note

A package that provides both a library and commands has to be more careful, because the library's own modules import deephaven. The repository's my_dh_toolkit shows that case: its __init__.py deliberately imports nothing that requires Deephaven, its commands import the library modules only after creating the server, and Python users import from the submodules (my_dh_toolkit.queries) rather than from the package.

Add a __main__.py if you also want to support python -m my_dh_cli. This is useful when the package is installed but its console script is not on the PATH, or when you want to be explicit about which Python interpreter runs the command:

from my_dh_cli.cli import main

if __name__ == "__main__":
    main()

Once the package is installed, both launchers run the same main() function and accept the same arguments:

my-dh-cli-query data/sample.csv
python -m my_dh_cli data/sample.csv

Each process that runs this command starts its own embedded server, which means the command carries JVM startup time and memory cost on every invocation. If another process already uses the selected port, such as a Deephaven server running in Docker, the embedded server fails to start because it cannot bind that port; pass a free port with --port.

Package a remote Deephaven client

Use pydeephaven when the program should connect to a Deephaven server that is already running. This is a regular Python client application: it does not need deephaven-server, a local JVM, or a startup step before imports. The server runs elsewhere, so many client processes can connect to it at once, and the client process stays lightweight.

Because the client has no server to start, the embedded-server import rule does not apply. pydeephaven can be imported at module scope, including in __init__.py, and the package can be structured like any other Python project that talks to a network service.

The trade-off is that the client API is not the same as the server-side deephaven API. Client code works with table handles that refer to tables on the server, and operations on those handles are sent to the server for execution. Most common table operations are available, but code written against deephaven does not run unchanged against pydeephaven. One difference comes up in the examples below: the server cannot read files from the client's machine, so local data is uploaded as a pyarrow table. Others are smaller, such as reading column names from table.schema.names rather than table.column_names.

The examples connect with anonymous authentication. A Deephaven server started with Docker or pip uses a pre-shared key by default; either start it with anonymous authentication enabled, or pass the server's key to Session as shown in the client command below. Where possible, keep pydeephaven on the same version as the server it connects to.

Client library

A client library packages functions that operate on tables through a Session. Pass the session in from the caller rather than creating one inside the library; this leaves connection details, authentication, and the session's lifetime under the calling program's control, and it lets the same functions be used against different servers. The full example is my_dh_client_library in the example repository.

The [project] table declares pydeephaven as the dependency. It also declares pyarrow, which the library imports directly to read CSV files. pydeephaven depends on pyarrow already, but a package should declare what it imports rather than rely on a transitive dependency:

[project]
name = "my-dh-client-library"
version = "0.1.0"
description = "Reusable functions for a remote Deephaven server"
requires-python = ">=3.9"
dependencies = ["pydeephaven>=42.7", "pyarrow"]

The library's functions take and return client-side Table handles. src/my_dh_client_library/utils.py holds upload_csv, which gets local data onto the server:

import pyarrow.csv as pacsv
from pydeephaven import Session, Table


def upload_csv(session: Session, path: str) -> Table:
    """Read a local CSV file and upload it to the server as a table."""
    return session.import_table(pacsv.read_csv(path))

src/my_dh_client_library/queries.py holds the query functions. The query string in filter_by_threshold is the same one the server-side library used, but here the operation runs on the server and only the handle is returned to the client. publish gives a table a name on the server:

from pydeephaven import Session, Table


def filter_by_threshold(table: Table, column: str, threshold: float) -> Table:
    """Keep rows where column > threshold."""
    return table.where([f"{column} > {threshold}"])


def publish(session: Session, name: str, table: Table) -> None:
    """Bind a table under a name so it is visible in the server's IDE and to other clients."""
    session.bind_table(name, table)

Because pydeephaven can be imported at module scope, __init__.py can re-export the public functions so that callers import them directly from the package name:

from my_dh_client_library.queries import filter_by_threshold, publish
from my_dh_client_library.utils import upload_csv

__all__ = ["filter_by_threshold", "publish", "upload_csv"]

A calling program creates the Session, passes it to the library functions, and closes it when finished. Nothing has to start before the imports; the only requirement is a server to connect to:

from pydeephaven import Session

from my_dh_client_library import filter_by_threshold, publish, upload_csv

with Session(host="localhost", port=10000) as session:
    scores = upload_csv(session, "data/sample.csv")
    high_scores = filter_by_threshold(scores, "Score", 75.0)
    publish(session, "high_scores", high_scores)

high_scores is now bound on the server, so it appears in the IDE and other clients can open it with session.open_table("high_scores"). The server's script scope holds the binding, so the table stays available after the session that created it closes, until the name is reassigned or deleted on the server. To bring rows back into the client process, call table.to_arrow() on a table handle while the session is still open.

Client command

A client package can expose a command in the same way as an embedded-server package, but it depends on pydeephaven and uses pydeephaven.Session. The full example is my_dh_client in the example repository. The layout is the same as the embedded-server command's, and __main__.py is the same two lines with the package name changed. The command starts quickly because there is no JVM to launch, which makes the client model a good fit for small utilities that are run often:

my_dh_client/
├── src/
│   └── my_dh_client/
│       ├── __init__.py
│       ├── __main__.py
│       └── cli.py
└── pyproject.toml

The pyproject.toml declares the dependencies and the console script in the same way as the embedded-server command:

[project]
name = "my-dh-client"
version = "0.1.0"
description = "A command-line Deephaven client"
requires-python = ">=3.9"
dependencies = ["pydeephaven>=42.7", "pyarrow", "click"]

[project.scripts]
my-dh-client-query = "my_dh_client.cli:main"

The command below does the same work as the embedded-server command, but on a server it connects to: it uploads a CSV file, adds a column, and binds the result under a name. The name defaults to the file's stem and must be a valid Python identifier. The command checks the name before it connects and exits with a usage error if the name is invalid, so a file such as my-scores.csv needs --name to choose a different one. Unlike the embedded-server command, it imports pydeephaven at module scope and has no startup step. It uses anonymous authentication by default. For another authentication method, pass its name with --auth-type; provide any required token through the DH_AUTH_TOKEN environment variable rather than a command-line argument so it does not appear in shell history or process listings.

import keyword
import os
from pathlib import Path
from typing import Optional

import click
import pyarrow.csv as pacsv
from pydeephaven import Session


@click.command()
@click.argument("input_file", type=click.Path(exists=True, dir_okay=False))
@click.option(
    "--host", default="localhost", show_default=True, help="Deephaven server host"
)
@click.option("--port", default=10000, show_default=True, help="Deephaven server port")
@click.option(
    "--auth-type", default="Anonymous", show_default=True, help="Authentication type"
)
@click.option(
    "--name", default=None, help="Name to bind the result under [default: file stem]"
)
def main(
    input_file: str, host: str, port: int, auth_type: str, name: Optional[str]
) -> None:
    """Upload a CSV file to a running Deephaven server and process it there."""
    name = name or Path(input_file).stem
    if not name.isidentifier() or keyword.iskeyword(name):
        raise click.BadParameter(
            f"'{name}' is not a valid Python identifier", param_hint="--name"
        )

    with Session(
        host=host,
        port=port,
        auth_type=auth_type,
        auth_token=os.environ.get("DH_AUTH_TOKEN", ""),
    ) as session:
        table = session.import_table(pacsv.read_csv(input_file))
        result = table.update(["DoubleScore = Score * 2"])
        session.bind_table(name, result)

    click.echo(f"Bound result table '{name}' on {host}:{port}")

Session is a context manager, so the client connection closes when the command finishes. The result table lives on the remote server, not in the client process, and because it is bound in the server's script scope, it remains available under its name after the command exits. Running the command several times does not start additional servers, so it does not need a separate free local server port.

Once installed, run it against a server on the default host and port, or point it elsewhere:

my-dh-client-query data/sample.csv
my-dh-client-query data/sample.csv --host dh.example.com --port 10000

For a server that uses a pre-shared key, set DH_AUTH_TOKEN in the environment without typing the key into a command. For example, read it from a file that is not checked into version control, or use read -s to enter it at a hidden prompt. Then pass the handler's class name:

read -s DH_AUTH_TOKEN && export DH_AUTH_TOKEN
my-dh-client-query data/sample.csv --auth-type io.deephaven.authentication.psk.PskAuthenticationHandler

The install and distribution steps that follow apply to all of the packages in this guide. Nothing about the build process depends on which Deephaven package the project uses.

Install and distribute a project

The commands in this section are run from the project directory, the one that contains pyproject.toml. They work the same way for every example in this guide.

During development, install from the project directory in editable mode. This makes changes in the source tree available without reinstalling, which is convenient while iterating on a command or library:

python -m pip install -e .

To install a project normally from its source directory, use python -m pip install .. This copies the package into the environment's site-packages, so later source changes are not picked up until you reinstall.

Projects often have dependencies that only some users need, such as plotting libraries or test tooling. Rather than requiring them for everyone, declare them as extras in [project.optional-dependencies] in pyproject.toml:

[project.optional-dependencies]
visualization = ["matplotlib>=3.7"]
dev = ["pytest>=7"]

Users then request extras by name in square brackets after the project. When installing from the source directory, the project is ., with or without -e. Quote the requirement so shells such as zsh do not treat the square brackets as filename patterns:

python -m pip install ".[visualization]"
python -m pip install -e ".[visualization,dev]"

The same syntax works for a published project, for example python -m pip install "my-dh-library[visualization]", in which case pip resolves the distribution from its configured package index rather than the local directory.

To distribute the project, use build to produce a wheel and source distribution from the project directory. The first command installs build itself; the second runs it:

python -m pip install --upgrade build
python -m build

The generated files are placed in dist/. A wheel can be installed directly with python -m pip install dist/my_project-0.1.0-py3-none-any.whl, copied to another machine, or hosted on an internal package index. For many teams, sharing wheels this way is all that is needed. Publishing to PyPI requires additional account, package-name, and upload configuration; follow the Python Packaging User Guide's tutorial on packaging and distributing projects for those steps.