How Deephaven works: A mental model and patterns of use

Deephaven works differently from SQL databases and from Java collections, and some of its behavior might surprise you. Code that looks straightforward can produce unexpected results. This guide explains how to think about Deephaven so you can write effective queries and avoid common pitfalls.

This guide isn't a deep technical dive. For that, see Deephaven's design. Instead, this guide builds the mental model you need to work productively with Deephaven from day one. The second half of the page puts that model to work: common query patterns, what you can build on the engine, and a quick reference.

For hands-on learning, see the Crash course.

Tables are recipes, not data

In traditional programming, a data structure is a container holding your data. When you filter or transform it, you get a new container with different data inside.

Deephaven tables work differently. When you call where or update, Deephaven doesn't copy the source data. Instead, it builds a new table that shares the parent's unchanged columns. The new table computes only what's new, such as the set of rows that pass a filter or a new computed column. It also keeps a live dependency on its parent, so it recomputes automatically when the source changes, without you rerunning any code.

source is a live table: its data changes while your program runs. It adds one row for each second that elapses. A slow or delayed update cycle can add several of those rows at once.

ii is a built-in special variable that holds each row's position as a long, so X numbers the rows. The special variable i holds the same position as an int.

filtered holds the rows of source that currently satisfy X > 2, and doubled adds a Y column to every row of source. Both tables update automatically as new rows arrive in source.

Why this matters:

• You don't need to rerun your filter when data changes, because the filtered table updates itself.
• Multiple transformations can share the same source without duplicating data.
• Operations are typically much faster than copying entire datasets.

view also keeps a dependency on its parent, but it doesn't compute values up front. It stores the formula and evaluates it when values are accessed, which saves memory for columns you rarely read. See Use view for lightweight derived columns.

Static vs. live tables

Deephaven tables come in two flavors:

• Static tables: Data that doesn't change. Usually loaded from files, created with emptyTable or newTable, or snapshots of live data.
• Live tables: Data that updates continuously. Deephaven also calls these refreshing tables. Created with timeTable or connected to Kafka streams and other real-time sources.

The key insight: Most transformations on live tables produce live results. A table derived from a live table, like filtered in Tables are recipes, not data, updates automatically. You don't need to poll for changes or rerun queries, because the engine handles propagation for you. snapshot is a deliberate exception. It takes a live table and returns a static copy of it at a single point in time.

Formulas run in the engine

When you write a formula string like "Y = X * 2", that code runs inside Deephaven's Java-based engine, which is optimized for processing millions of rows efficiently.

The engine parses the string "Y = Math.sqrt(X * X + 1)" and compiles it to Java, so Groovy never evaluates the formula itself.

This has important implications:

• Java methods are available: Use Math.sqrt, String methods like toUpperCase, and other Java standard library methods.
• Groovy variables are available: Script variables assigned without def are available inside formulas through the query scope.
• Groovy closures are callable: Formulas can call closures, at a performance cost described in Calling Groovy from formulas.
• Engine-native formulas run fast: Formulas that use only operators, Java methods, and Deephaven's built-in functions are engine-native. The engine compiles them to Java and reads their input columns in bulk chunks, so each row costs a compiled Java call rather than a closure call.

Calling Groovy from formulas

You can call Groovy closures from formulas:

The engine calls a closure once per row, which is slower than engine-native formulas. For performance-critical code, prefer engine-native formulas.

Write stateless formulas

The engine doesn't promise to evaluate rows in order, or one at a time. This is the most common surprise for new users. Consider this code:

You might expect Value to be [1, 2, 3, 4, 5], but nothing guarantees it. Two things can break a counter like this one:

• Parallel evaluation: When an operation processes many rows at once, the engine can evaluate a formula's rows on multiple threads, so values can come out of order and differ between runs.
• Live re-evaluation: On a live table, the engine re-evaluates a formula only for the rows that change, so the counter's values no longer follow the row order.

On a small static table like this one, the values usually come back in order.

The rule: Formulas should be stateless. The result for row N should depend only on the input values for row N, not on what happened when processing other rows. Immutable variables, such as configuration values, are fine. Shared mutable state, such as a counter that every row updates, and logic that depends on the order rows are processed in are not.

Replacing counters with row positions

If you need deterministic row numbering on a table whose rows never change, or only ever get added at the end, use ii (the row position) instead of a counter:

Replacing running totals with updateBy

A running total is also stateful, because each row's value depends on the rows before it. Instead of accumulating it in a Groovy variable, use an updateBy cumulative operation such as CumSum. The engine computes it in table order and keeps it correct as the table changes:

For more complex cases involving state, see Controlling concurrency, which explains how to make the engine evaluate a formula serially.

Moving data between Groovy and Deephaven

Data lives in two places: Groovy variables and Deephaven tables. Understanding when data moves between them helps you write efficient code.

From Groovy to Deephaven

When you create a table from Groovy data, the engine copies that data:

From Deephaven to Groovy

When you extract data back to Groovy, you can copy column values into an array. The array is a copy, not a live view, so later changes to the table don't affect it. The following example copies the X column into an array:

For live tables, this copy holds the data from one moment in time. The table keeps updating, but your copy doesn't change.

Performance tips

Moving data between Groovy and Deephaven takes time. For large datasets, avoid extracting data in loops. Keep the work in the engine instead, as the first pattern below shows.

Common patterns

Pattern: Prefer engine operations over loops

You don't need a Groovy loop to compute per-group values. An aggregation such as aggBy runs inside the engine over whole batches of rows. On a live table, it updates as rows arrive:

See aggBy for all aggregation options.

Pattern: Use view for lightweight derived columns

Use view when you want derived columns without storing them. Use update when you need the results cached for repeated access or downstream operations.

Pattern: Compose queries step by step

Build complex analytics by chaining simple operations. Each step produces a table you can inspect, reuse, or build on:

Each intermediate table (cleaned, enriched) is a first-class object you can display, join, or use as input to further operations.

Pattern: Same code for batch and streaming

Write your logic once, and run the same table operations on files and on live streams:

You don't need separate codebases for batch and streaming data.

Pattern: Partition large datasets

A partitioned table splits a table into subtables, one per key value, so you can process each one independently. Partitioned tables let you process subtables in parallel and retrieve a subtable by key directly, instead of re-running a filter for each key. partitionBy creates one, and transform applies a closure to every subtable:

Here, transformFunc adds an AvgPrice column to each symbol's subtable: a rolling average of Price over that symbol's most recent 10 rows.

When you call transform from a script, the closure doesn't inherit the script's execution context, which table operations need to use the query scope or compile formulas. So the closure opens the script's context before it calls updateBy.

What you can build

Deephaven is more than a table engine. It's a platform for building data applications.

Data sources and sinks

Deephaven reads data from files, streams, and your own code, and writes results back out:

Client-server architecture

Deephaven runs as a server. Multiple clients can connect simultaneously:

• Web UI: For interactive work in the built-in console and grids
• Python client (pydeephaven): For Python applications outside the server
• JavaScript client: For web applications
• Java/C++/R/Go clients: For integrations in those languages

Clients that subscribe to the same table see consistent, live updates. You can share tables between sessions with shared tickets.

Quick reference

Related documentation

• Quickstart
• Crash course
• Table types
• Deephaven's live DAG
• Create tables
• Select, view, and update