Understanding the Table API

How table operations work in Deephaven

Deephaven's Table API lets you filter, transform, aggregate, and join data through one consistent set of operations. The same operations work whether the data is static or updating in real time. Joins come in two families: exact and relational joins and time-series and range joins.

This guide explains the core concepts behind the API, helping you write more effective queries and choose between the overlapping ways to add columns.

Note

New to Deephaven? Start with How Deephaven works: A mental model for the conceptual foundation this guide builds on. This page assumes you already know why tables don't copy data and why formulas run in the engine. It focuses on the API itself and the tradeoffs you'll actually choose between.

For a quick-reference index of available operations and the reference page for each, see Table operations overview.

Tables are immutable

Table operations never modify their source, so the original is always safe to keep using. A live table's rows still change as new data arrives, but no operation changes the table you pass to it.

Most operations return a distinct new table. As an optimization, a no-op transformation can return the same object. Either way, the operation doesn't change the source:

This differs from libraries like pandas, where assigning to a column (for example, df["x"] = ...) modifies the DataFrame in place. In Deephaven, you build up results by chaining operations, and each step leaves its input untouched.

Each step's output usually isn't a full copy, though. Tables are recipes, not data explains how the engine shares unchanged columns instead of duplicating them.

Why immutability matters:

  • Debugging: Intermediate tables remain available for inspection.
  • Reuse: The same source can feed multiple downstream operations without interference.
  • Live updates: When source data changes, the engine can safely propagate those changes through the dependency graph of tables built from it.

Operations build a dependency graph

When you chain operations, you create a directed acyclic graph (DAG) of table dependencies:

Each table depends on the one before it. If source were live, filtered, enriched, and summary would all update whenever it changed.

As Static vs. live tables in the mental model guide shows, the same operations work on both static and live tables. Here is what that means for the graph:

The code is identical for static and live tables, but the graph behaves differently. The engine computes static_result once, and it never changes. For a static table, the graph is simply a convenient way to structure your code.

For live_result, the graph stays active. When live_table gets new rows:

  • Updates propagate automatically through every dependent table.
  • You don't re-run your code. The engine handles the incremental updates.
  • Each downstream table sees a consistent view of the data.

Deephaven processes real-time data efficiently because it typically recomputes only what changed, not the entire result.

To check which kind you have, use is_refreshing:

Formulas are strings

You write column expressions, called formulas, as strings, not native Python code:

Deephaven's Java engine, not Python, parses and executes these strings, so the syntax is Java-like:

  • Call Java methods such as Math.sqrt, not Python's math.sqrt.
  • Use backticks for string literals: `hello`. Java double quotes also work, but you must escape them inside a double-quoted Python string.
  • Use the ternary operator for conditionals: X > 0 ? X : -X.

What's available inside formulas covers the other things a formula can use, including Deephaven's built-in functions.

What's available inside formulas

Inside a formula string, you have access to:

Column values, referenced by name:

Special variables:

  • i — Row position as int (0, 1, 2, ...)
  • ii — Row position as long (for tables with more than 2 billion rows)
  • k — Internal row key, which isn't necessarily the same as row position

Live tables restrict where you can use these variables. See Special variables on live tables.

Built-in functions, such as math, parsing, and time functions:

Java methods on column values and Java classes such as Math:

Your own Python functions, at a performance cost:

Each call from Java into Python adds overhead. Calling Python from formulas explains when the engine batches those calls and when it makes one call per row. Either way, for performance-critical code, prefer built-in functions.

Query scope variables:

Python variables in the local or global scope are automatically available in formula strings through Deephaven's query scope.

Special variables on live tables

On a static table, formulas in update, select, view, and where can use i, ii, and k without restriction.

On a live table, row positions and keys can shift. By default, the engine rejects a formula that uses these variables unless the table is one of these types:

A time table is append-only, which is why live_table in the dependency graph example can use ii.

Some operations, such as formula aggregations, reject these variables on every table, static or live.

See special variables for the error message and array-access rules, and Table types for what each table type is.

Choose how to add columns: memory vs. computation

The Table API offers several ways to add columns, each with different performance characteristics:

OperationStores valuesRecomputes on accessKeeps other columnsBest for
updateYesNoYesExpensive formulas, values accessed repeatedly
selectYesNoNoExpensive formulas, values accessed repeatedly
viewNoYesNoSimple formulas, memory-constrained environments
update_viewNoYesYesSimple formulas, memory-constrained environments
lazy_updateCachedWhen cache missesYesFew unique input values, expensive computation
  • update computes values up front and stores them in memory.
  • select is like update but only includes the columns you list.
  • view computes values on demand instead of storing them, and only includes the columns you list.
  • update_view is like view but keeps all existing columns.

The following example adds the same Total column with update, select, and view. The Total values are identical in all three results. The difference is which columns each result keeps: result_update keeps Price and Quantity, while result_select and result_view keep only Symbol and Total.

lazy_update computes a value the first time it sees a given set of inputs and caches the result. A later row with the same inputs reuses the cached value, so lazy_update pays off only when the inputs have few distinct values.

If the formula is only a call to a Python function that the engine auto-vectorizes, lazy_update doesn't cache it. The engine evaluates that call in batches instead of row by row.

For live tables, choosing between stored and on-demand operations also decides when the work happens:

  • update and select recompute and store values for changed rows on every update cycle, even if nothing reads them.
  • view and update_view don't evaluate their formulas when the source changes. They compute values only when something reads them.
  • lazy_update also waits until a value is read, then caches it.