Write data to an in-memory, real-time table

This guide covers two ways to publish data to in-memory ticking tables:

table_publisher publishes data to a blink table, while DynamicTableWriter writes data to an append-only table. Both can ingest data from external live sources, such as WebSockets. We recommend table_publisher in most cases: it has a newer, more flexible, and more performant API, and it supports blink tables natively. DynamicTableWriter can be more convenient when you add very few rows (for example, one) at a time and want a simple interface.

Table publisher

The table_publisher factory function creates a TablePublisher and its linked blink table. From there:

  • Add data to the blink table with add.
  • (Optionally) Store data history in a downstream table.
  • (Optionally) Shut the publisher down when finished.

More sophisticated use cases add steps but follow the same basic formula.

Example: getting started with a table publisher

The first four arguments of table_publisher are:

  • name: the publisher's name.
  • col_defs: the blink table's column definitions, such as a dictionary that maps column names to data types. See TableDefinitionLike for all accepted forms.
  • on_flush_callback: an optional function that runs at the start of each update cycle.
  • on_shutdown_callback: an optional function that runs once when the publisher shuts down.

This example passes an on-shutdown callback and no on-flush callback. table_publisher returns the linked blink table and the TablePublisher, in that order.

The following code creates a blink table with three columns (X, Y, and Z). The columns contain no data until you call add, which the example's add_table function wraps.

Each call to add_table adds rows to my_table. The rows appear on the next update cycle after the command that adds them finishes. For details, see Data appears on the next update cycle.

The my_table blink table after data has been added

To stop the TablePublisher, call publish_failure. This notifies the blink table's listeners of the failure, so downstream tables show an error. It also invokes the on-shutdown callback, and later calls to add return without publishing. In this example, the when_done function calls publish_failure.

Example: threading

The getting started example adds data to its blink table only when you call add_table by hand. In most real-world use cases, you want to add data automatically at a regular interval. Python's threading module can do this. The following example adds between 5 and 10 rows of data to my_table via empty_table every second for 5 seconds.

Important

Table operations that run in a separate thread and compile query strings, such as update, need an execution context. Without one, they raise an exception. Capture the current context with get_exec_ctx and open it in the thread with a with statement.

The above table

Data history

Table publishers create blink tables. Deephaven processes new data in periodic update cycles. A blink table keeps only the rows added in the current cycle and drops them when the next cycle starts, so it stores no data history. In most use cases, you want to store some or all of the rows written during previous update cycles. There are two common ways to do this:

See the table types user guide for more information on these table types, including which one is best suited for your application.

The following code block builds on the threading example. It reuses that example's my_table and thread_func, creates a downstream ring table and append-only table from my_table, and then starts a new thread that adds more data to my_table.

The above my_table, my_ring_table, and my_append_only_table tables

Example: asyncio

The following code block pulls cryptocurrency trade data from Coinbase's WebSocket feed. It ingests the data asynchronously on an asyncio event loop that runs in a background thread, so a single thread can serve several WebSocket subscriptions without blocking the console.

The code collects incoming WebSocket messages in the my_matches list instead of adding them to the publisher one at a time. The on_flush_callback passed to table_publisher runs once at the beginning of each update cycle, so on_flush adds everything collected since the previous cycle as one table. Keep the flush callback fast, because it blocks the update cycle while it runs.

The on_shutdown_callback cancels the WebSocket task when the publisher shuts down.

Note

Install the websockets package before you run the code below.

The above t1 and t2 tables

DynamicTableWriter

DynamicTableWriter writes data into a live, in-memory table whose column names and data types you define. To use it:

Example: getting started with DynamicTableWriter

The following example creates a table with two columns (A and B). The columns contain randomly generated integers and strings, respectively. A separate thread adds a new row every second for ten seconds. When the thread finishes writing, it closes the writer with close.

Example: trig functions

The following example writes rows with an X column and its sine, cosine, and tangent in the SinX, CosX, and TanX columns, and plots the sine and cosine with Deephaven Express as the table updates.

Data appears on the next update cycle

Neither a table publisher nor a DynamicTableWriter adds rows to its table immediately. A table publisher's add and a writer's write_row both queue the new rows. The queued rows reach the table when the table's update source next refreshes, which happens once per cycle of the Update Graph (UG). The UG is the engine component that processes table updates. Rows queued from a background thread can appear in a cycle that is already running, if they arrive before that refresh. Rows added from a table publisher's on-flush callback, which runs at the start of a cycle, appear in that same cycle.

The Python script session holds the exclusive UG lock while a command executes, so that cycle cannot run until the command finishes. As a result, new rows do not appear in output tables while the command that wrote them is still running.

The following example shows this with a DynamicTableWriter, whose append-only table keeps the rows so you can inspect them afterward. What would you expect the print statement below to produce?

The print statement prints True: it runs in the same command as the write_row calls. The Python script session holds the exclusive UpdateGraph lock while a command executes, so the UG cycle that adds the queued rows to result can't run until the command finishes.

A second command can still start before that cycle runs, so wait for the table to update if it is still empty. Then the same print statement prints False.

In a standard Deephaven server, the Periodic Update Graph drives these update cycles. To learn how update cycles propagate changes through your queries, see Deephaven's table update model.