Understanding the Table API

How table operations work in Deephaven

Deephaven's Table API lets you filter, transform, aggregate, and join data through one consistent set of operations. The same operations work whether the data is static or updating in real time. Joins come in two families: exact and relational joins and time-series and range joins.

This guide explains the core concepts behind the API, helping you write more effective queries and choose between the overlapping ways to add columns.

Note

New to Deephaven? Start with How Deephaven works: A mental model for the conceptual foundation this guide builds on. This page assumes you already know why tables don't copy data and why formulas run in the engine. It focuses on the API itself and the tradeoffs you'll actually choose between.

For a quick-reference index of available operations and the reference page for each, see Table operations overview.

Tables are immutable

Table operations never modify their source, so the original is always safe to keep using. A live table's rows still change as new data arrives, but no operation changes the table you pass to it.

Most operations return a distinct new table. As an optimization, a no-op transformation can return the same object. Either way, the operation doesn't change the source:

This differs from dataframe libraries such as pandas, where assigning to a column modifies the DataFrame in place. In Deephaven, you build up results by chaining operations, and each step leaves its input untouched.

Each step's output usually isn't a full copy, though. Tables are recipes, not data explains how the engine shares unchanged columns instead of duplicating them.

Why immutability matters:

  • Debugging: Intermediate tables remain available for inspection.
  • Reuse: The same source can feed multiple downstream operations without interference.
  • Live updates: When source data changes, the engine can safely propagate those changes through the dependency graph of tables built from it.

Operations build a dependency graph

When you chain operations, you create a directed acyclic graph (DAG) of table dependencies:

Each table depends on the one before it. If source were live, filtered, enriched, and summary would all update whenever it changed.

As Static vs. live tables in the mental model guide shows, the same operations work on both static and live tables. Here is what that means for the graph:

The code is identical for static and live tables, but the graph behaves differently. The engine computes staticResult once, and it never changes. For a static table, the graph is simply a convenient way to structure your code.

For liveResult, the graph stays active. When liveTable gets new rows:

  • Updates propagate automatically through every dependent table.
  • You don't re-run your code. The engine handles the incremental updates.
  • Each downstream table sees a consistent view of the data.

Deephaven processes real-time data efficiently because it typically recomputes only what changed, not the entire result.

To check which kind you have, use isRefreshing:

Formulas are strings

You write column expressions, called formulas, as strings:

Deephaven's Java engine parses and executes these strings, so the syntax is Java-like:

  • Call Java methods such as Math.sqrt directly.
  • Use backticks for string literals: `hello`. Java double quotes also work, but you must escape them inside a double-quoted Groovy string.
  • Use the ternary operator for conditionals: X > 0 ? X : -X.

What's available inside formulas covers the other things a formula can use, including Deephaven's built-in functions.

What's available inside formulas

Inside a formula string, you have access to:

Column values, referenced by name:

Special variables:

  • i — Row position as int (0, 1, 2, ...)
  • ii — Row position as long (for tables with more than 2 billion rows)
  • k — Internal row key, which isn't necessarily the same as row position

Live tables restrict where you can use these variables. See Special variables on live tables.

Built-in functions, such as math, parsing, and time functions:

Java methods on column values and Java classes such as Math:

Your own Groovy closures, at a performance cost:

The engine calls your closure once per row. For details, see Calling Groovy from formulas. For performance-critical code, prefer built-in functions.

Query scope variables:

Variables you assign at the top level of a script, without def or a type, are automatically in Deephaven's query scope, so formula strings can use them. These variables are not in the query scope:

  • Variables declared with def or a type.
  • Variables declared inside a method or closure.

To use one of those in a formula, add it to the query scope explicitly.

Special variables on live tables

On a static table, formulas in update, select, view, and where can use i, ii, and k without restriction.

On a live table, row positions and keys can shift. By default, the engine rejects a formula that uses these variables unless the table is one of these types:

A time table is append-only, which is why liveTable in the dependency graph example can use ii.

Some operations, such as formula aggregations, reject these variables on every table, static or live.

See special variables for the error message and array-access rules, and Table types for what each table type is.

Choose how to add columns: memory vs. computation

The Table API offers several ways to add columns, each with different performance characteristics:

OperationStores valuesRecomputes on accessKeeps other columnsBest for
updateYesNoYesExpensive formulas, values accessed repeatedly
selectYesNoNoExpensive formulas, values accessed repeatedly
viewNoYesNoSimple formulas, memory-constrained environments
updateViewNoYesYesSimple formulas, memory-constrained environments
lazyUpdateCachedWhen cache missesYesFew unique input values, expensive computation
  • update computes values up front and stores them in memory.
  • select is like update but only includes the columns you list.
  • view computes values on demand instead of storing them, and only includes the columns you list.
  • updateView is like view but keeps all existing columns.

The following example adds the same Total column with update, select, and view. The Total values are identical in all three results. The difference is which columns each result keeps: resultUpdate keeps Price and Quantity, while resultSelect and resultView keep only Symbol and Total.

lazyUpdate computes a value the first time it sees a given set of inputs and caches the result. A later row with the same inputs reuses the cached value, so lazyUpdate pays off only when the inputs have few distinct values.

For live tables, choosing between stored and on-demand operations also decides when the work happens:

  • update and select recompute and store values for changed rows on every update cycle, even if nothing reads them.
  • view and updateView don't evaluate their formulas when the source changes. They compute values only when something reads them.
  • lazyUpdate also waits until a value is read, then caches it.