Table data service
This guide covers using the Table Data Service API to integrate custom, partitioned, lazily loaded data from an external service into Deephaven workflows.
To use the Table Data Service API, you implement a backend that supplies your data, and then construct a Table Data Service from it. The API provides classes with abstract methods that serve as a template for your implementation. This guide covers each of the components you implement, along with an example implementation.
Note
This feature is currently experimental. The API and its characteristics are subject to change.
How the API fits together
The Python TableDataService API lets you serve data from an external service or database to Deephaven as PyArrow tables, which can back Deephaven tables in both static and refreshing contexts.
You implement three classes and use a fourth:
TableKeyidentifies a table.TableLocationKeyidentifies one location (partition) of a table.TableDataServiceBackendsupplies schemas, locations, sizes, and column values for those keys.TableDataServicetakes your backend and creates Deephaven tables from it.
Refreshing tables from the Python API are add-only. A backend can add new table locations and append rows to existing ones, but it can't remove a location or remove rows from one.
The following sections list the methods each class requires and give an example implementation of the three classes you implement. The Example section uses those implementations to build static and refreshing tables.
TableKey
A TableKey is a unique identifier for a table. An implementation must have the following methods:
__hash__: Create a hash of the key.__eq__: Compare two keys for equality. Ifkey1 == key2is true, thenhash(key1) == hash(key2)must also be true.
The following code block contains an example TableKey implementation named TableKeyImpl. It implements both of the required methods, along with a __str__ method so the key is readable in log and error messages.
TableLocationKey
A TableLocationKey is a unique identifier for a table location. A table location is one partition of a table, and it holds a subset of the table's rows. If the table has partitioning columns, the location also holds the partitioning values that identify it. A location's size is its row count.
An implementation requires the same methods as a TableKey implementation:
__hash__: Create a hash of the key.__eq__: Compare two keys for equality. Ifkey1 == key2is true, thenhash(key1) == hash(key2)must also be true.
The following code block contains an example TableLocationKey implementation named TableLocationKeyImpl. It implements both of the required methods, along with a __str__ method so the key is readable in log and error messages.
TableDataServiceBackend
A TableDataServiceBackend is the interface you implement to supply data to a TableDataService. You must implement each of its abstract methods:
table_schema: Fetch the schema of a table.table_locations: Fetch the locations of a table.table_location_size: Fetch the size of a table location.column_values: Fetch the values of one column at one location, starting at rowoffset.subscribe_to_table_locations: Subscribe to existing and future table locations.subscribe_to_table_location_size: Subscribe to existing and future table location sizes.
The TableDataService calls table_locations and table_location_size for static tables (refreshing=False in make_table or make_partitioned_table). It calls subscribe_to_table_locations and subscribe_to_table_location_size for refreshing tables (refreshing=True).
Backend methods don't return their results. Each one calls the callbacks that the service passes to it:
| Method | Delivers its result to | Then calls |
|---|---|---|
table_schema | schema_cb(data_schema, partitioning_schema) | Nothing |
table_locations | location_cb(location_key, partitioning_values), once per location | success_cb |
table_location_size | size_cb(row_count) | Nothing |
column_values | values_cb(pyarrow_table) | Nothing |
subscribe_to_table_locations | location_cb, for existing and future locations | success_cb after the existing locations |
subscribe_to_table_location_size | size_cb, for the current and future sizes | success_cb after the current size |
Each method reports a failure by calling its failure_cb callback with an Exception. The two subscribe_* methods keep the callbacks for future changes and return a function that the service calls to unsubscribe.
When the service calls table_schema, table_locations, table_location_size, or column_values, it waits until the method calls one of its callbacks, so every code path in those methods must call either the result callback or failure_cb. A backend can call the callbacks from another thread, for example when new data arrives. However, a callback called from another thread while the backend method is still running can block until that method returns, so a backend method shouldn't wait for another thread to call its callbacks.
The callbacks take these arguments:
data_schema: The PyArrow schema of the data columns. It doesn't need to include the partitioning columns.partitioning_schema: The PyArrow schema of the partitioning columns, orNoneif the table has none.partitioning_values: A one-row PyArrow table that matchespartitioning_schemaand holds the location's partitioning values, orNoneif the table has no partitioning columns. Any other row count is an error.- The table passed to
values_cb: One column, with at leastmin_rowsand at mostmax_rowsrows. Any other row count is an error.min_rowsis never larger than the service'spage_size.
The following code blocks contain two example classes:
TestTableis an in-memory data store that holds the backend's data. It is a plain Python class, not a Deephaven table. Itsadd_table_locationandappend_table_locationmethods add a location and append rows to an existing one.TestBackendimplements all six required methods ofTableDataServiceBackend, plus anadd_tablemethod that registers a table with the backend.
An in-memory data store for the backend
A TableDataServiceBackend implementation
TableDataService
The TableDataService is the class you interact with. To create one, pass your TableDataServiceBackend implementation to its constructor, along with up to three optional keyword-only parameters:
chunk_reader_factory: The Barrage chunk reader factory used to decode the PyArrow data the backend returns. Most backends can omit it.stream_reader_options: The Barrage options the service uses when it reads the PyArrow data. If omitted, the service uses Barrage's default snapshot options. Most backends can omit it.page_size: The page size, in rows, that the service uses when it requests column values. If omitted, the service uses thePythonTableDataService.defaultPageSizeconfiguration property, which defaults to 65,536 rows (64K).
The TableDataService class has two methods:
-
Creates a table backed by the data service backend. Pass it the
TableKeythat identifies the table, and setrefreshingto choose a static or refreshing table. -
Creates a partitioned table backed by the data service backend. Pass it the
TableKeythat identifies the table, and setrefreshingto choose a static or refreshing table.
Example
This example combines the TableKeyImpl, TableLocationKeyImpl, TestTable, and TestBackend classes from the TableKey, TableLocationKey, and TableDataServiceBackend sections into a working TableDataService. It shows how to:
- Create a custom table and add it to the backend.
- Manually add data to the backend's table locations.
- Fetch a table from that backend in both static and refreshing contexts.
The code below creates two tables from the same backend:
t_staticis a static table (refreshing=False). It holds the rows the backend had when the engine first read the table. In this example, that read happens when the IDE displayst_static.tis a refreshing table (refreshing=True). It keeps updating while something holds a reference to it.
The code creates t inside a liveness scope, which manages the table and its subscriptions to the backend.
- The example doesn't release the scope, so
tkeeps updating after thewithblock ends. - When you no longer need
t, call the scope'sreleasemethod to drop the scope's reference to the table. - Once nothing else holds the table, the engine cancels its subscriptions to the backend. For example, the engine cancels them after you release the scope and close the table in the IDE.
While the tables are open in the IDE, call add_ticker_data again to append more data. The refreshing table t shows the new rows, and the static table t_static does not change.
Each location occupies its own range of row keys, so the rows appended to MSFT:BZX and MSFT:BZY appear in t before the rows of GOOG:BZX, the location added later. Because rows don't always arrive at the end, the table is add-only but not append-only.