Table data service

This guide covers using the Table Data Service API to integrate custom, partitioned, lazily loaded data from an external service into Deephaven workflows.

To use the Table Data Service API, you implement a backend that supplies your data, and then construct a Table Data Service from it. The API provides classes with abstract methods that serve as a template for your implementation. This guide covers each of the components you implement, along with an example implementation.

Note

This feature is currently experimental. The API and its characteristics are subject to change.

How the API fits together

The Python TableDataService API lets you serve data from an external service or database to Deephaven as PyArrow tables, which can back Deephaven tables in both static and refreshing contexts.

You implement three classes and use a fourth:

Refreshing tables from the Python API are add-only. A backend can add new table locations and append rows to existing ones, but it can't remove a location or remove rows from one.

The following sections list the methods each class requires and give an example implementation of the three classes you implement. The Example section uses those implementations to build static and refreshing tables.

TableKey

A TableKey is a unique identifier for a table. An implementation must have the following methods:

  • __hash__: Create a hash of the key.
  • __eq__: Compare two keys for equality. If key1 == key2 is true, then hash(key1) == hash(key2) must also be true.

The following code block contains an example TableKey implementation named TableKeyImpl. It implements both of the required methods, along with a __str__ method so the key is readable in log and error messages.

TableLocationKey

A TableLocationKey is a unique identifier for a table location. A table location is one partition of a table, and it holds a subset of the table's rows. If the table has partitioning columns, the location also holds the partitioning values that identify it. A location's size is its row count.

An implementation requires the same methods as a TableKey implementation:

  • __hash__: Create a hash of the key.
  • __eq__: Compare two keys for equality. If key1 == key2 is true, then hash(key1) == hash(key2) must also be true.

The following code block contains an example TableLocationKey implementation named TableLocationKeyImpl. It implements both of the required methods, along with a __str__ method so the key is readable in log and error messages.

TableDataServiceBackend

A TableDataServiceBackend is the interface you implement to supply data to a TableDataService. You must implement each of its abstract methods:

The TableDataService calls table_locations and table_location_size for static tables (refreshing=False in make_table or make_partitioned_table). It calls subscribe_to_table_locations and subscribe_to_table_location_size for refreshing tables (refreshing=True).

Backend methods don't return their results. Each one calls the callbacks that the service passes to it:

MethodDelivers its result toThen calls
table_schemaschema_cb(data_schema, partitioning_schema)Nothing
table_locationslocation_cb(location_key, partitioning_values), once per locationsuccess_cb
table_location_sizesize_cb(row_count)Nothing
column_valuesvalues_cb(pyarrow_table)Nothing
subscribe_to_table_locationslocation_cb, for existing and future locationssuccess_cb after the existing locations
subscribe_to_table_location_sizesize_cb, for the current and future sizessuccess_cb after the current size

Each method reports a failure by calling its failure_cb callback with an Exception. The two subscribe_* methods keep the callbacks for future changes and return a function that the service calls to unsubscribe.

When the service calls table_schema, table_locations, table_location_size, or column_values, it waits until the method calls one of its callbacks, so every code path in those methods must call either the result callback or failure_cb. A backend can call the callbacks from another thread, for example when new data arrives. However, a callback called from another thread while the backend method is still running can block until that method returns, so a backend method shouldn't wait for another thread to call its callbacks.

The callbacks take these arguments:

  • data_schema: The PyArrow schema of the data columns. It doesn't need to include the partitioning columns.
  • partitioning_schema: The PyArrow schema of the partitioning columns, or None if the table has none.
  • partitioning_values: A one-row PyArrow table that matches partitioning_schema and holds the location's partitioning values, or None if the table has no partitioning columns. Any other row count is an error.
  • The table passed to values_cb: One column, with at least min_rows and at most max_rows rows. Any other row count is an error. min_rows is never larger than the service's page_size.

The following code blocks contain two example classes:

  • TestTable is an in-memory data store that holds the backend's data. It is a plain Python class, not a Deephaven table. Its add_table_location and append_table_location methods add a location and append rows to an existing one.
  • TestBackend implements all six required methods of TableDataServiceBackend, plus an add_table method that registers a table with the backend.
An in-memory data store for the backend
A TableDataServiceBackend implementation

TableDataService

The TableDataService is the class you interact with. To create one, pass your TableDataServiceBackend implementation to its constructor, along with up to three optional keyword-only parameters:

  • chunk_reader_factory: The Barrage chunk reader factory used to decode the PyArrow data the backend returns. Most backends can omit it.
  • stream_reader_options: The Barrage options the service uses when it reads the PyArrow data. If omitted, the service uses Barrage's default snapshot options. Most backends can omit it.
  • page_size: The page size, in rows, that the service uses when it requests column values. If omitted, the service uses the PythonTableDataService.defaultPageSize configuration property, which defaults to 65,536 rows (64K).

The TableDataService class has two methods:

  • make_table

    Creates a table backed by the data service backend. Pass it the TableKey that identifies the table, and set refreshing to choose a static or refreshing table.

  • make_partitioned_table

    Creates a partitioned table backed by the data service backend. Pass it the TableKey that identifies the table, and set refreshing to choose a static or refreshing table.

Example

This example combines the TableKeyImpl, TableLocationKeyImpl, TestTable, and TestBackend classes from the TableKey, TableLocationKey, and TableDataServiceBackend sections into a working TableDataService. It shows how to:

  • Create a custom table and add it to the backend.
  • Manually add data to the backend's table locations.
  • Fetch a table from that backend in both static and refreshing contexts.

The code below creates two tables from the same backend:

  • t_static is a static table (refreshing=False). It holds the rows the backend had when the engine first read the table. In this example, that read happens when the IDE displays t_static.
  • t is a refreshing table (refreshing=True). It keeps updating while something holds a reference to it.

The code creates t inside a liveness scope, which manages the table and its subscriptions to the backend.

  • The example doesn't release the scope, so t keeps updating after the with block ends.
  • When you no longer need t, call the scope's release method to drop the scope's reference to the table.
  • Once nothing else holds the table, the engine cancels its subscriptions to the backend. For example, the engine cancels them after you release the scope and close the table in the IDE.

While the tables are open in the IDE, call add_ticker_data again to append more data. The refreshing table t shows the new rows, and the static table t_static does not change.

Each location occupies its own range of row keys, so the rows appended to MSFT:BZX and MSFT:BZY appear in t before the rows of GOOG:BZX, the location added later. Because rows don't always arrive at the end, the table is add-only but not append-only.