Use NumPy in Deephaven queries

NumPy is an open-source Python module that includes a library of powerful numerical capabilities. These capabilities include multi-dimensional arrays, mathematical functions implemented in compiled C for speed, and a C API for writing extensions. It is one of the most popular and widely used Python modules.

NumPy is a dependency of the Deephaven Python package, so installing Deephaven also installs NumPy.

This guide shows three ways to use NumPy with Deephaven tables:

  • Convert tables and business calendars to NumPy objects, and NumPy arrays back to tables, with deephaven.numpy. The array is a one-time copy that does not update when the table ticks (receives new data).
  • Call NumPy from query strings, one row or one grouped array at a time. The result column updates as the table ticks.
  • Pass table data to a NumPy-based function in batches and write the results back to a table with deephaven.learn. When the source table is add-only or blink, the output table updates as it ticks.

deephaven.numpy

deephaven.numpy is a submodule of the Deephaven Python package that provides three functions:

Convert between tables and arrays

The following code block converts a Deephaven table to a NumPy array and back:

Important

  • to_numpy produces a single 2D array, so all columns it converts must have the same data type. Otherwise, it raises an error. Use the cols argument to choose same-typed columns.
  • to_numpy copies the selected data into memory, every column by default. To copy less, pass cols and filter the table first.
  • to_numpy does not convert null values. See Handle nulls below.
  • If the table is refreshing (its data changes over time), to_numpy converts a snapshot of its current data. The array does not update when the table ticks.

Handle nulls

When to_numpy converts a table, it does not replace nulls with numpy.nan:

  • Nulls in primitive columns appear as Deephaven's null constants.
  • Nulls in Boolean columns appear as True.
  • Nulls in Instant columns appear as NaT.
  • Nulls in String columns appear as the string 'None', and nulls in other object columns appear as None.

Use dh_null_to_nan from deephaven.jcompat to convert null constants to numpy.nan. For integer, Boolean, or character arrays, pass type_promotion=True, which returns a new float64 array. Without it, dh_null_to_nan raises an error for those arrays.

The following code block converts an int column that contains nulls and replaces the nulls with numpy.nan:

Convert a business calendar

The following code block gets one of Deephaven's example business calendars by name with calendar and converts it to a NumPy busdaycalendar with to_np_busdaycalendar. It then prints the converted calendar's weekmask and its first five holidays:

The weekmask marks Monday through Friday as business days, and the holidays come from the Deephaven calendar.

NumPy in query strings

You can use NumPy in query strings like any other Python module. Two rules apply:

The following code block converts sensor voltages to temperatures by passing a scalar column to a NumPy-based function, one row at a time. The function uses np.interp to look up each voltage on a fixed calibration curve. The query language has no built-in equivalent, so this is a good job for NumPy:

Voltages between two points on the curve get a linearly interpolated temperature. A voltage past the end of the curve, such as 3.4, gets the temperature at that end (60.0).

NumPy functions also work on whole arrays, which is useful when each result depends on more than one row. The following code block:

  1. Uses the row index i to build four values of X.
  2. Uses group_by to collect them into one row, so X becomes a vector column.
  3. Passes that array to np.gradient in a query string. np.gradient has no built-in equivalent.
  4. Uses ungroup to expand the result back to one value per row.

Each value of Y depends on its neighbors in X. The first and last rows use one-sided differences (8 − 1 = 7), and the middle rows use central differences ((27 − 1) / 2 = 13).

deephaven.learn

deephaven.learn is a submodule of the Deephaven Python package that moves data between Deephaven tables and Python objects, such as the NumPy arrays its gather module produces. AI/ML applications use it most often, but it works for other batch computations too. It follows the gather-compute-scatter paradigm:

  • Gather: A gather function copies data from table columns into a Python object, such as a NumPy array.
  • Compute: Your function computes on that object.
  • Scatter: A scatter function writes the results back into a table.

to_numpy copies a one-time snapshot of a table, even a refreshing one. deephaven.learn instead adds its results as columns of a new table, which updates as the source table ticks. That works only when the source table is one of these types:

Because learn identifies rows by their row keys, it raises an error on other refreshing tables. For those, see AI/ML workflows.

The code below uses deephaven.learn to calculate the sine of a set of values.

The code defines three functions:

  • table_to_numpy gathers rows and columns from a table into a NumPy array.
  • compute_sin takes the sine of the gathered data.
  • numpy_to_table scatters the results back into a table.

At the end, the code calls the learn function with these arguments:

  • table: source, the input table.
  • model_func: compute_sin, the compute function.
  • inputs: a learn.Input that pairs the input column X with the gather function table_to_numpy.
  • outputs: a learn.Output that pairs the output column SinX with the scatter function numpy_to_table.
  • batch_size: 101, so that learn processes all 101 rows in one call.

For a deeper dive on this subject, see the deephaven.learn guide.