Generate tables with Python functions

This guide covers function-generated tables, which create a table from a Python function. With a trigger table or a refresh interval, the result is a ticking table with rows that update over time. The function runs once when the table is created, and again whenever:

  • One or more trigger tables tick.
  • A refresh interval is reached.

Use a function-generated table to ingest data from external sources into a ticking table. A refresh that produces a new table replaces the whole result, so when the input is already a Deephaven table, use regular table operations, many of which update incrementally.

Usage pattern

Function-generated tables follow this basic usage pattern:

  • Define a Python function that returns a table.
  • Create a function-generated table by calling function_generated_table with that function and at most one kind of trigger:
    • One or more trigger tables, passed as source_tables. Every trigger table must be a refreshing (ticking) table.
    • A refresh interval, passed as refresh_interval_ms.

If you supply neither trigger, or a refresh_interval_ms that is zero or negative, the function runs once and the result is a static table.

Define the generator and call function_generated_table

Define the table generator function as you would any other Python function. It must return one of the following:

  • A table with the same column names and types as the result. The result takes its columns from the first generated table, or from the table_definition when you supply one.
  • None, to keep the previous result.

The function can read data from its source tables, but don't run further table operations on them inside the function:

  • Deephaven can return a cached (memoized) result for such an operation.
  • The function-generated table doesn't track that cached table as a dependency.
  • The function can therefore run before the cached table finishes updating in the same update cycle, and read inconsistent data.

List every ticking table the function depends on in source_tables.

The following code block defines make_table, which returns a five-row table with a column of random integers and a column of random doubles. It uses make_table as the table generator function and calls function_generated_table twice:

  • Once with a trigger table.
  • Once with a refresh interval.

Execution context

Function-generated tables require an execution context to run in. Pass a context with the exec_ctx parameter. If you omit it, function_generated_table captures the execution context that is current when you call it, and uses that context for every invocation of the table_generator. In a console script, that is the systemic execution context. The examples on this page do not pass exec_ctx, so they use the systemic execution context.

Example: pull data from a web API

The following example pulls the hourly forecast for Denver, Colorado, from the National Weather Service's free-to-use Weather API. Each row is one forecast hour, in time order, and PctChanceRain holds the forecast probability of any precipitation. The function re-runs once per minute (refresh_interval_ms=60_000). It runs during an update cycle, so a slow request delays every table update until it returns, and timeout=10 bounds that wait. If the request fails, the table stops updating. For production use, fetch the data on a separate thread and have the function return the latest result.

The above denver_weather table

How the result updates

Every refresh in which the table_generator produces a table replaces the result in full. Each table update:

  • Removes all of the previous rows.
  • Adds all of the newly generated rows.
  • Contains no modified rows and no shifts, even when the generated data is identical to the previous cycle's.

Downstream operations therefore reprocess the entire result on every refresh that produces a table, which is why this guide recommends regular table operations when the input is already a Deephaven table.

A refresh in which the table_generator returns None produces no update, unless the result is a blink table. See Retain the previous result.

Additional options

Beyond the trigger, the table_generator can decline to produce a new table. function_generated_table also accepts several optional parameters that control how the result is produced and shaped. Two of them refine how the result updates, independently of each other:

  • copy_data controls where the result's data lives and what its row keys look like.
  • blink_table controls how downstream operations interpret each update.

Retain the previous result

When new data is not always available, the table_generator function can return None to decline producing a new table on a given update cycle. The result then keeps the previous cycle's table. A blink table result is cleared instead.

If the first invocation returns None, supply a table_definition so the result's columns are known before the first table exists.

Specify the table definition

You can supply a table_definition, such as a dictionary that maps column names to deephaven.dtypes types. The definition is authoritative. It defines the result's columns and their order. Every table the table_generator produces must be compatible with it. A definition is required when the first invocation returns None. See Retain the previous result.

Copy data or delegate to the generated table

By default (copy_data=True), the generated rows are copied into the result's own column sources, and the result uses a flat, contiguous row set with row keys 0 through size - 1. The generated table itself is not retained.

With copy_data=False, the result skips the copy and delegates directly to the generated table. It:

  • Uses the generated table's column sources.
  • Adopts the generated table's row set as-is.
  • Reports the generated table's row set as each update's added rows and the previous cycle's row set as its removed rows.

Because the result holds the generated column sources across cycles, a refreshing generated table must expose immutable column sources. A ticking table keeps each column's previous-cycle values so downstream operations can see what changed. A generated table that changes values in place would corrupt those values, so function_generated_table rejects a refreshing generated table if any of its column sources is not immutable. A static table produced fresh on each refresh, such as one produced by snapshot, always meets this requirement.

Set blink_table=True to present the result as a blink table, so downstream operations see only the rows generated during the current cycle. Each update is still a full replacement, as described in How the result updates. The blink setting changes only how downstream operations interpret that update, not how the rows are copied or delegated.

A blink result behaves as follows:

  • It requires a refresh trigger.
  • Rows generated in one update cycle are removed on the next cycle, whether or not the table_generator runs again.
  • With a refresh interval longer than one cycle, the result is empty between refreshes.

Pass arguments to the table generator

Use args (a tuple) and kwargs (a dictionary) to pass positional and keyword arguments to the table_generator. function_generated_table passes the same arguments on every invocation.