AI/ML workflows

This guide explores several AI/ML workflows in Deephaven. It shows how to train a model on historical data with a table iterator, then apply it to live data in real time. It covers two ways to apply the model:

deephaven.learn is an alternative to these workflows. It works on static tables and on add-only or blink ticking tables. Append-only tables are add-only, so they qualify too.

Train a model

Training data in AI/ML applications is almost always historical and thus inherently static. You can train a model on historical data with a table iterator. Deephaven provides four table iterators:

Model training typically uses batch data, so a chunked iterator is usually the best choice. Use iter_chunk_dict or iter_chunk_tuple. The choice between the dict-based and tuple-based iterators is a matter of personal preference.

Consider the following code, which fits a cubic polynomial model to a table of noisy static data:

The printed polynomial is the model. Its coefficients approximate the 0.1, 0.3, -0.7, and -11 used to generate the data. They don't match exactly, because of the noise and because the model averages two separate fits.

Real-time inference

You can apply a trained model to a ticking table in two ways:

Table operations are simpler, while the listener and publisher give more control. Workflow trade-offs compares them.

The examples in this section apply the model fitted in Train a model, whose coefficients are stored in coeffs. Run that example first.

Table operations

One way to apply a model to a table in real time is to use one of the following table operations:

These operations create new columns from existing columns, so you can use them to add a column of model outputs. They differ in when they compute results and whether they store them in memory. For more information, see Choose the right column selection method.

The following code block uses update to apply the model. update performs the calculations immediately and stores the results in memory.

A live line plot of the noisy Y values and the model's YFitted values over time

Table listener and publisher

A combination of a table listener and table publisher can also evaluate AI/ML models in real time. The listener runs in each update cycle in which the source table changes. It receives the rows that were added, modified, and removed, and passes the ones you choose to the model. The publisher writes the model's outputs to a new table.

A diagram showing a table listener passing added, modified, and removed rows from a source table to an AI model, and a table publisher writing the model's outputs to a new table

The diagram shows all three kinds of row change reaching the model. This guide's example passes only the added and modified rows to it.

The example below works in five steps:

  • The table listener applies the polynomial model to the rows added or modified in each update cycle.
  • The table publisher writes the outputs to preds, a blink table that holds only the current cycle's outputs.
  • The blink_to_append_only call converts preds to an append-only table that keeps the full prediction history.
  • The last_by call keeps only the latest prediction for each Timestamp.
  • The natural_join call joins those latest predictions back to the source table so the plot can show them next to the input data.

A live line plot of the noisy Y values and the YFitted predictions joined from the table publisher

In on_update, the listener handles each kind of row change differently:

  • Added rows: The listener applies the model and publishes one prediction per row.
  • Modified rows: The listener applies the model again and publishes another prediction with the same Timestamp. last_by keeps only the latest prediction for each timestamp before the natural_join.
  • Removed rows: The listener ignores them. Their old predictions stay in the prediction history. The natural_join drops them from the result, because it keeps only rows that are still in source_live.

The time table in this example only adds rows, so the modified and removed cases never happen here. They matter when you use the same listener on a table that modifies or removes rows.

The last_by step works because each row of the time table has its own Timestamp. When several rows can share a timestamp, publish and join on the column or columns that identify each row.

The example registers the listener with do_replay=True. Its first call also receives the rows already in source_live, so every row gets a prediction. For more information, see Replay data.

Use a thread pool

Deephaven calls a listener function inside the update cycle, while the update graph holds its exclusive lock. That lock gives the listener a consistent view of the data, but the cycle can't finish until the listener returns. A slow listener therefore makes every table in the update graph fall behind. To avoid this, run slow calculations on other threads.

Worker threads keep the model out of the update cycle. On a Python build with the GIL (global interpreter lock), they run Python code one at a time. They speed up a model only when its heavy work runs in native code, such as NumPy, PyTorch, or TensorFlow.

The following code uses a thread pool from Python's concurrent.futures module to illustrate the concept. It builds on the previous example, with these changes:

  • It creates a ThreadPoolExecutor with four worker threads.
  • compute_and_publish no longer takes the publisher as an argument.
  • The listener calls executor.submit(compute_and_publish, timestamps, data) instead of calling compute_and_publish directly. The model runs on a worker thread, and the listener returns right away.
  • A report_error callback prints any exception a worker thread raises.

A live line plot of the noisy Y values and the YFitted predictions published from worker threads

With more than one worker, calculations can finish out of order. If the source table modifies a row twice in quick succession, the older prediction can be published last. last_by keeps whichever prediction was published last, so the joined value can come from the row's older inputs.

To keep predictions in order, set max_workers=1. A single worker runs calculations in the order the listener submits them, and the model still runs outside the update cycle.

executor.submit returns a Future that holds any exception compute_and_publish raises. The example's report_error callback prints it, so errors on worker threads aren't lost.

To stop the example, call handle.stop(), then executor.shutdown().

Workflow trade-offs

So, you have a trained AI/ML model and aren't sure which workflow to use. Here are some things to consider: