Synchronize multiple tables

When working with multiple related tables in Deephaven, you may encounter situations where tables receive updates at different rates. This guide shows how to use SyncTableFilter and LeaderTableFilter to coordinate updates across multiple tables.

The synchronization problem

Deephaven does not provide cross-table or cross-partition transactions. The system processes each partition independently to maximize throughput. This means that a row created later in one partition may appear in your query before a row created earlier in a different partition.

This independence can cause consistency issues when you have multiple tables that contain correlated data. For example:

  • A trading system might have separate tables for orders, executions, and messages that all share a common transaction ID.
  • A data pipeline might split events across multiple tables, each tagged with a sequence number.
  • A multi-source system might need to wait until all sources report data for a given timestamp.

Both SyncTableFilter and LeaderTableFilter solve this problem by ensuring that only coordinated rows appear in the filtered results.

When to use each utility

Choose the synchronization utility based on your table relationships:

  • Use SyncTableFilter when all tables are peers. Each table contributes equally to determining which rows to show. The filter passes through rows where all tables have matching ID values.

  • Use LeaderTableFilter when one table should control synchronization. The leader table contains ID values that dictate which rows from follower tables to show. This is useful when one table acts as a coordination log or contains the authoritative sequence of events.

Requirements

Both utilities require:

  • Add-only tables: Tables must not modify, shift, or remove rows. If you filter an add-only source table, the result remains add-only.
  • Monotonically increasing IDs: ID values must increase for each key. IDs cannot decrease or repeat.
  • Atomic updates: All rows for a given ID of a given table must appear in the same update.
  • Shared keys: Tables must have common key columns for grouping.

SyncTableFilter

SyncTableFilter synchronizes multiple peer tables by showing, for each key, only the rows at the highest ID that all tables currently share.

How it works

For each key, the filter finds the highest ID for which every input table has a matching row, and passes through only the rows at that ID. When the tables receive new data and reach a higher commonly available ID, the filter removes the previous ID's rows and adds the new ID's rows.

Example

This example synchronizes three tables that share Symbol as a key and use SeqNum as the ID:

In this example:

  • For AAPL, priceData has SeqNum 1, 2, and 3, but volumeData and bidAskData only go up to SeqNum 2. The highest ID common to all three is 2, so only the SeqNum 2 rows appear in the synchronized results.
  • For GOOGL, priceData and volumeData have SeqNum 1 and 2, but bidAskData only has SeqNum 1. The highest common ID is 1, so only the SeqNum 1 rows appear.
  • When bidAskData receives SeqNum 2 for GOOGL, the filter advances to show those rows instead, replacing the SeqNum 1 rows.

API

Create a builder with the ID column name and key column names:

Add each table with a unique name:

Build and retrieve the synchronized tables:

LeaderTableFilter

LeaderTableFilter synchronizes multiple tables using a leader-follower pattern. The leader table contains ID columns that specify which rows from each follower table to show.

How it works

The leader table contains one ID column for each follower table. For each key, the filter shows the rows from each follower table that match the IDs in the leader's most recent row for that key, once every follower's ID is satisfied. An ID is satisfied either by a matching row in that follower table, or by a null, which is always treated as satisfied but yields no rows for that follower. An earlier leader row for that key is superseded once a later one is fully satisfied.

Example

This example uses a synchronization log as the leader table:

In this example:

  • The syncLog leader table controls which trades and messages appear. Only the most recent leader row per key is shown once its IDs are matched in every follower table.
  • For ClientA/S1, the leader has two rows: (TradeId 100, MessageId 1) and (TradeId 101, MessageId 2). Both are fully matched by tradeLog and messageLog. However, only the most recent match — TradeId 101 and MessageId 2 — appears in the synchronized results.
  • Even though tradeLog has Id 102 and messageLog has MsgId 3, they don't appear because the leader hasn't referenced them yet.
  • For ClientB/S2, only trade 200 and message 5 appear.

API

Create a builder with the leader table and key columns:

Add each follower table with:

  • A unique name
  • The table reference
  • ID column mapping (format: "leaderIdColumn=followerIdColumn")
  • Key columns in the follower table (must match leader key columns in type)

Build and retrieve the synchronized tables:

Partitioned table variant

LeaderTableFilter.PartitionedTableBuilder works with partitioned tables:

Requirements:

  • All partitioned tables have the same number of key columns.
  • Key columns have compatible types.
  • Key columns are joined in order.
  • Constituent tables within each partition are add-only.