Query table configuration

This guide discusses how to control various QueryTable features that affect your Deephaven tables' latency and throughput.

Query Table

QueryTable is Deephaven's primary implementation of our Table API.

The QueryTable has the following user-configurable properties:

Each property is described below, roughly categorized by similarity.

Memoization

Deephaven utilizes memoization for many table operations to improve performance by eliminating duplicate work. See Query Memoization for more details.

It can be beneficial to disable memoization when benchmarking or testing, as memoized results can hide true operational costs and skew performance metrics.

Redirection

Deephaven Tables maintain a 63-bit keyspace that maps a logical row in row-key space to its data. Many of Deephaven's column sources use a multi-level data layout to avoid allocating more resources than necessary to fulfill operational requirements. See selection method properties for more details.

Redirection is a mapping between a parent column source and the resulting column source for a given operation. A sorted column, for example, is redirected from the original to present the rows in the targeted sort order. Redirection may also flatten from a sparse keyspace to a flat and dense keyspace.

DataIndex

A Deephaven DataIndex is an index that can improve the speed of filtering operations.

Pushdown predicates with where

Pushdown predicates refer to the mechanism whereby filtering conditions are applied as early as possible, ideally at the data source (e.g., Parquet or other columnar formats), before loading data into the system. By annotating source reads with predicates, the engine pulls in only the rows that satisfy the conditions, significantly reducing I/O and improving performance.

Parallel processing with where

Parallelism for where operations is not enabled until the parent's size exceeds QueryTable.parallelWhereRowsPerSegment rows. This avoids the overhead of using threads for small operations. For tables larger than this threshold, the where operation uses a fixed number of parallel segments defined by QueryTable.parallelWhereSegments. These parameters can be tuned to avoid unnecessary parallelism when the overhead exceeds potential gains.

Parallel processing with select

The QueryTable operations select and update have performance enhancements that try to take advantage of parallelism during two separate phases of each table operation invocation. The first opportunity for parallelism is on the initial creation of the table. The engine will parallelize the initial computation of the resulting table state. The second opportunity for parallelism is when the operation's parent-table listener is notified that the parent was updated.

Parallelism for select operations is not enabled until the parent's size exceeds QueryTable.minimumParallelSelectRows rows. This can be tuned to avoid unnecessary parallelism (e.g., when the overhead exceeds potential gains).

Parallel snapshotting

Barrage clients, including our JavaScript implementation used on the web, fulfill subscription requests by snapshotting the required rows and columns in addition to listening for relevant changes when the table is refreshing. Parallel snapshotting is a feature that parallelizes this process across columns. If those columns are slow to access then parallel snapshotting will greatly reduce latency. However, parallel snapshotting may open many file handles to the same data source.

Parallel snapshotting is not enabled until the snapshot size exceeds QueryTable.minimumParallelSnapshotRows rows. This can be tuned to avoid unnecessary parallelism when the overhead exceeds potential gains.

Stateless by default (experimental)

Anticipated in a future release of Deephaven, the flag(s) in this category will flip from a default of false to a default of true. These flags enable the engine to assume more often that a given formula, filter, or selectable can be optimized (unless otherwise noted by the user via API usages).

This is experimental; more details can be learned by reading the Javadoc on io.deephaven.api.ConcurrencyControl.

Related documentation

• QueryTable JavaDocs
• Table API
• Incremental update model
• Query Memoization
• Data indexes
• Parallelizing queries