Deephaven's Python package

The Deephaven Python package (deephaven) is the server-side Python API for writing queries that run inside the Deephaven engine. Queries you write with it work the same way on static tables and on real-time, ticking tables, and the results update as new data arrives.

To connect to a Deephaven server from a separate Python process, use the Python client (pydeephaven) instead.

Deephaven's Python API provides functions for creating, ingesting, transforming, and exporting tables. This guide gives a brief overview in two parts. The first part shows common tasks with short examples. The second part lists each submodule in the package, with a link to its Pydoc and related guides.

What can you do with deephaven?

In short: a lot. Let's look at a few common tasks that only scratch the surface.

Create tables from scratch

You can create tables from scratch in several ways. The code below uses three of them:

  • empty_table creates a table with the number of rows you ask for. update then adds columns to it.
  • new_table builds a table from columns you define with column functions such as int_col, from the column module.
  • time_table creates a ticking table, which is a table whose rows change as new data arrives. This one adds a row every second.

The table_factory module defines all three functions. The top-level deephaven package re-exports them, so the example imports them from deephaven.

Ingest data from external sources

Ingest data into tables from external sources such as CSV, Apache Parquet, Kafka, and SQL.

The top-level package re-exports read_csv from the csv module. The first example reads a CSV file with read_csv and a Parquet file with the parquet module's read.

The next example consumes a Kafka topic with consume. It needs a running Kafka broker at redpanda:9092.

The last example runs a SQL query against a PostgreSQL database with read_sql. It reads the connection details from environment variables. To run it, you need a PostgreSQL database and the connectorx package. Deephaven does not install it by default, so install it before you run the example.

Filter data

You can filter tables by conditions or by row position, such as the rows at the top or bottom of a table. The following example filters with where, using query strings, including one that calls a Python function. It then keeps the first or last rows with head and tail.

Calculate grouped, cumulative, and rolling aggregations

The following example computes per-symbol averages with agg_by, which applies combined aggregations such as avg and returns one row per group.

The example then uses update_by. Unlike agg_by, it keeps every row and adds the results as new columns. It adds two per-symbol cumulative and rolling aggregations:

  • cum_sum adds a running total of X.
  • rolling_avg_tick averages X and Y over a window of rows: the current row and the four rows before it.

Rolling aggregations can also use a time window instead of a row window.

Join tables together

Deephaven offers a wide variety of ways to join tables together. Joins can be exact or relational or inexact or time-series.

Even ticking tables can be joined with no extra work. Joins on ticking tables use the same calls as joins on static tables, and the result updates as the inputs change. The exception is range_join, which accepts only static tables.

The following example uses natural_join to add each employee's department details from t_right to the matching rows of t_left.

Rogers has no DeptID, so the department columns in that row are null. No employee belongs to Marketing, so it does not appear in the result.

Convert to and from NumPy and pandas

NumPy and pandas are two of Python's most popular packages, and Deephaven Python queries often use them. The numpy and pandas modules convert tables to and from these libraries' data structures.

The following example converts a table to a NumPy array with to_numpy and to a DataFrame with to_pandas. It then converts each one back with the NumPy to_table and the pandas to_table. For more, see Use NumPy in Deephaven queries and Convert between Deephaven tables and pandas DataFrames.

Export tables to Parquet, CSV, Kafka, and more

You can export table data to many formats, including Parquet, CSV, Kafka, and HTML. The following code block writes a table to Parquet with the parquet module's write and to CSV with write_csv.

To share a table with other Deephaven servers without exporting it, use Uniform Resource Identifiers (URIs).

What's in the deephaven package

Each of the following subsections, in alphabetical order, describes one submodule of the deephaven package and links to its Pydoc.

agg

The agg submodule defines the combined aggregations that table operations such as agg_by apply.

appmode

The appmode submodule supports writing Deephaven application mode Python scripts.

arrow

The arrow submodule supports conversions to and from PyArrow tables and Deephaven tables. See Deephaven and PyArrow.

barrage

The barrage submodule provides functions for accessing resources on remote Deephaven servers over Barrage. See Capture Python client tables with Barrage to use it with shared tickets.

calendar

The calendar submodule defines functions for working with business calendars.

column

The column submodule implements columns in tables. You use it mainly to create new tables.

concurrency_control

The concurrency_control submodule defines the Barrier class and the ConcurrencyControl protocol. They give you explicit control over concurrency when select, update, and where evaluate formulas and filters. See the parallelization guide.

configuration

The configuration submodule provides read access to Deephaven configuration properties as strings, numbers, or booleans.

constants

The constants submodule defines the global constants, including Deephaven's special numerical values. Values include the maximum, minimum, NaN, null, and infinity values for different data types.

csv

The csv submodule supports reading an external CSV file into a Deephaven table and writing a table to a CSV file. The top-level deephaven package re-exports its read and write functions as read_csv and write_csv.

dbc

The dbc submodule lets you connect to and execute queries on external databases from Deephaven.

dherror

The dherror submodule defines a custom exception for the Deephaven Python package. See the triage errors guide for more information.

dtypes

The dtypes submodule defines the data types supported by the Deephaven engine.

execution_context

The execution_context submodule lets you manage the Deephaven query execution context on threads.

experimental

The experimental submodule contains experimental Deephaven features. Their interfaces can change between releases. Examples include left outer joins, full outer joins, AWS S3 support, and Iceberg integration.

filters

The filters submodule implements filters that you can use in filtering operations.

html

The html submodule supports exporting Deephaven data in HTML format.

jcompat

The jcompat submodule provides Java compatibility support and convenience functions to create Java data structures from corresponding Python ones.

json

The json submodule describes the structure of JSON values declaratively. For example, to read a Kafka topic in JSON format, build a Jackson provider from a JSON value description with jackson.provider. Then pass the provider to object_processor_spec.

learn

The learn submodule provides utilities for efficient data transfer between tables and Python objects. It serves as a framework for using machine learning libraries in Deephaven Python queries.

liveness_scope

The liveness_scope submodule uses liveness scopes to control when the engine cleans up unreferenced nodes in the query update graph, instead of relying only on garbage collection.

numpy

The numpy submodule supports the conversion between Deephaven tables and NumPy arrays. See Use NumPy in Deephaven queries.

pandas

The pandas submodule supports the conversion between Deephaven tables and pandas DataFrames. See Convert between Deephaven tables and pandas DataFrames.

pandasplugin

The pandasplugin submodule displays pandas DataFrames as Deephaven tables in the UI.

parquet

The parquet submodule supports reading external Parquet files into Deephaven tables and writing Deephaven tables out as Parquet files.

perfmon

The perfmon submodule contains tools to analyze performance of the Deephaven system and queries.

plot

The plot submodule contains the framework for Deephaven's built-in plotting API. That API is no longer under active development. For most plotting tasks, use Deephaven Express (deephaven.plot.express), which the deephaven-plugin-plotly-express package provides.

plugin

The plugin submodule contains the framework for using plugins and creating your own plugins in Deephaven. It comes from the deephaven-plugin package, which deephaven-core installs as a dependency.

plugin_authorization

The plugin_authorization submodule provides authorization helpers for Python plugins that send server objects, such as tables, to clients.

query_library

The query_library submodule lets you import Java classes or packages into the query library so you can use them in queries.

replay

The replay submodule provides support for replaying historical data.

server

The server submodule provides the executors module, which lets Python code submit tasks to named JVM server executors. The two executors are serial and concurrent.

stream

The stream submodule contains Deephaven's Apache Kafka integration, table publishers, and functions for converting between table types, such as blink_to_append_only.

systemic_obj_tracker

The systemic_obj_tracker submodule controls systemic object marking, which determines the objects whose failures the engine reports to the server's error handler.

table

The table submodule implements these objects:

table_factory

The table_factory submodule provides various ways to create Deephaven tables, including:

table_listener

The table_listener submodule implements Deephaven's table listeners.

time

The time submodule defines functions for handling Deephaven date-time data.

update_graph

The update_graph submodule provides access to the locks of the update graph. By default, Deephaven acquires the shared lock for you when a table operation on a refreshing table needs it. Hold a lock yourself when you need a consistent view of ticking data across several operations.

updateby

The updateby submodule builds the cumulative and rolling aggregations that the update_by table operation uses.

uri

The uri submodule implements tools for resolving Uniform Resource Identifiers (URIs) into objects.