Add a dedicated DIS for user tables

Deephaven provides intraday user tables written by a Data Import Server (Centrally Managed User Tables (Legacy) or Live Partitioned User Tables (Core+)). While historical user tables are directly written to disk by the worker, DIS-written user tables are written by the worker passing binary log rows to the Log Aggregator Server (LAS). The LAS then writes them to disk, where a tailer picks them up and streams them to a DIS. Besides processing the binary log rows into table rows on disk, the DIS also serves these updates to subscribers, which allows for ticking user tables and multiple appends to user tables.

When to use this guide

In environments with significant use of DIS-written user tables, it may be desirable to add a dedicated Data Import Server to:

  • Avoid resource contention between system table log file processing and intraday user table operations.
  • Isolate workloads for better performance and capacity planning.
  • Scale independently as user table usage grows.

This is only relevant for Centrally Managed User Tables (which use appendCentral and related commands) or Live Partitioned User Tables (which use appendLiveTable and related commands).

Why namespace set filtering is required

The simplest way to route data is using claims, which allow a DIS to claim specific namespaces or tables. However, claims do not support namespace sets (User vs System). Since user tables and system tables are distinguished by namespace set rather than specific namespace names, you must configure data routing manually to split User and System tables between different DIS instances.

Configuration steps

Step 1: Create the DIS

Follow the instructions in Add a dedicated Data Import Server to create a new DIS named db_rta on your chosen server. This will:

  1. Add or activate the DIS process on the server.
  2. Create the basic DIS configuration in the routing system.
  3. Configure the DIS process to load the correct configuration.

Note

When creating the DIS configuration in Step 2 of that guide, name it db_rta instead of using a namespace-based name. You'll configure the filters and TDCP routing in the steps below. Steps 2 and 3 below are modifications to the routing configuration that can be performed during Step 1, or separately afterward.

Step 2: Configure the User filter

The db_rta DIS needs a filter to specify that it handles User namespace set tables.

Edit the routing configuration to remove the claims section and add a filter to the db_rta entry:

This filter ensures that the db_rta DIS will only process tables from the User namespace set.

Step 3: Configure TDCP routing

The Table Data Cache Proxy (TDCP) needs to be configured to route User tables to db_rta and System tables to the other data import servers.

Edit the routing configuration to update the db_tdcp entry in the tableDataServices section:

This configuration tells the TDCP:

  • Route System namespace set tables to all data import servers except db_rta (as directed by their filters and claims)
  • Route User namespace set tables to db_rta

This is the key difference from using claims - the TDCP sources must be configured explicitly to handle namespace set routing.