Use NumPy in Deephaven queries
NumPy is an open-source Python module that includes a library of powerful numerical capabilities. These capabilities include multi-dimensional arrays, mathematical functions implemented in compiled C for speed, and a C API for writing extensions. It is one of the most popular and widely used Python modules.
NumPy is a dependency of the Deephaven Python package, so installing Deephaven also installs NumPy.
This guide shows three ways to use NumPy with Deephaven tables:
- Convert tables and business calendars to NumPy objects, and NumPy arrays back to tables, with
deephaven.numpy. The array is a one-time copy that does not update when the table ticks (receives new data). - Call NumPy from query strings, one row or one grouped array at a time. The result column updates as the table ticks.
- Pass table data to a NumPy-based function in batches and write the results back to a table with
deephaven.learn. When the source table is add-only or blink, the output table updates as it ticks.
deephaven.numpy
deephaven.numpy is a submodule of the Deephaven Python package that provides three functions:
to_numpy: Converts a Deephaven table to a NumPy array.to_table: Converts a NumPy array to a Deephaven table.to_np_busdaycalendar: Converts a DeephavenBusinessCalendarto a NumPybusdaycalendar.
Convert between tables and arrays
The following code block converts a Deephaven table to a NumPy array and back:
Important
to_numpyproduces a single 2D array, so all columns it converts must have the same data type. Otherwise, it raises an error. Use thecolsargument to choose same-typed columns.to_numpycopies the selected data into memory, every column by default. To copy less, passcolsand filter the table first.to_numpydoes not convert null values. See Handle nulls below.- If the table is refreshing (its data changes over time),
to_numpyconverts a snapshot of its current data. The array does not update when the table ticks.
Handle nulls
When to_numpy converts a table, it does not replace nulls with numpy.nan:
- Nulls in primitive columns appear as Deephaven's null constants.
- Nulls in Boolean columns appear as
True. - Nulls in
Instantcolumns appear asNaT. - Nulls in
Stringcolumns appear as the string'None', and nulls in other object columns appear asNone.
Use dh_null_to_nan from deephaven.jcompat to convert null constants to numpy.nan. For integer, Boolean, or character arrays, pass type_promotion=True, which returns a new float64 array. Without it, dh_null_to_nan raises an error for those arrays.
The following code block converts an int column that contains nulls and replaces the nulls with numpy.nan:
Convert a business calendar
The following code block gets one of Deephaven's example business calendars by name with calendar and converts it to a NumPy busdaycalendar with to_np_busdaycalendar. It then prints the converted calendar's weekmask and its first five holidays:
The weekmask marks Monday through Friday as business days, and the holidays come from the Deephaven calendar.
NumPy in query strings
You can use NumPy in query strings like any other Python module. Two rules apply:
- Wrap NumPy calls in a Python function with type hints, so the resulting column has the correct data type.
- Prefer a built-in query language function when one exists. Python functions are usually slower because each call crosses the Python-Java boundary.
The following code block converts sensor voltages to temperatures by passing a scalar column to a NumPy-based function, one row at a time. The function uses np.interp to look up each voltage on a fixed calibration curve. The query language has no built-in equivalent, so this is a good job for NumPy:
Voltages between two points on the curve get a linearly interpolated temperature. A voltage past the end of the curve, such as 3.4, gets the temperature at that end (60.0).
NumPy functions also work on whole arrays, which is useful when each result depends on more than one row. The following code block:
- Uses the row index
ito build four values ofX. - Uses
group_byto collect them into one row, soXbecomes a vector column. - Passes that array to
np.gradientin a query string.np.gradienthas no built-in equivalent. - Uses
ungroupto expand the result back to one value per row.
Each value of Y depends on its neighbors in X. The first and last rows use one-sided differences (8 − 1 = 7), and the middle rows use central differences ((27 − 1) / 2 = 13).
deephaven.learn
deephaven.learn is a submodule of the Deephaven Python package that moves data between Deephaven tables and Python objects, such as the NumPy arrays its gather module produces. AI/ML applications use it most often, but it works for other batch computations too. It follows the gather-compute-scatter paradigm:
- Gather: A gather function copies data from table columns into a Python object, such as a NumPy array.
- Compute: Your function computes on that object.
- Scatter: A scatter function writes the results back into a table.
to_numpy copies a one-time snapshot of a table, even a refreshing one. deephaven.learn instead adds its results as columns of a new table, which updates as the source table ticks. That works only when the source table is one of these types:
- Static
- Add-only, which includes append-only
- Blink
Because learn identifies rows by their row keys, it raises an error on other refreshing tables. For those, see AI/ML workflows.
The code below uses deephaven.learn to calculate the sine of a set of values.
The code defines three functions:
table_to_numpygathers rows and columns from a table into a NumPy array.compute_sintakes the sine of the gathered data.numpy_to_tablescatters the results back into a table.
At the end, the code calls the learn function with these arguments:
table:source, the input table.model_func:compute_sin, the compute function.inputs: alearn.Inputthat pairs the input columnXwith the gather functiontable_to_numpy.outputs: alearn.Outputthat pairs the output columnSinXwith the scatter functionnumpy_to_table.batch_size:101, so thatlearnprocesses all 101 rows in one call.
For a deeper dive on this subject, see the deephaven.learn guide.
Related documentation
- Incremental update model
- Table types
- Python-Java boundary
deephaven.learn- AI/ML workflows
- Business calendars
- Filters
- Query strings
- Arrays
- Convert between Deephaven tables and pandas DataFrames
- Install and use Python packages
- Python functions in query strings
- Built-in query language functions
- Auto-imported functions
calendardh_null_to_nandouble_colempty_tablegroup_bynew_table- Nulls
- Special variables
to_np_busdaycalendarto_numpyto_tableungroupupdatedeephaven.numpyPydocdeephaven.learnPydocdeephaven.learn.gatherPydocdeephaven.jcompatPydoc