How Deephaven works: A mental model and patterns of use

Deephaven works differently from tools like SQL or traditional Java data structures — and some of Deephaven's behavior might surprise you. Code that looks straightforward can produce unexpected results. This guide explains how to think about Deephaven so you can write effective queries and avoid common pitfalls.

This isn't a deep technical dive — for that, see Deephaven's design. Instead, this guide builds the mental model you need to work productively with Deephaven from day one.

For technical details, see Deephaven's design. For hands-on learning, see the Crash course.

Tables are recipes, not data

In traditional programming, a data structure is a container holding your data. When you filter or transform it, you get a new container with different data inside.

Deephaven tables work differently. When you call where or update, Deephaven doesn't copy the source data — it builds a new table that shares the parent's unchanged columns and computes only what's new (a filtered RowSet, or a computed column). That new table also keeps a live dependency on its parent, so it recomputes automatically when the source changes, without you rerunning any code.

source is a live table — it adds one row for each one-second period that elapses, so it grows by roughly a row per second (a slow or delayed update cycle can add several rows at once). filtered immediately reflects whichever rows currently satisfy X > 2, and it maintains a dependency on source: as new rows arrive, filtered and doubled automatically recompute and reflect the change, without you rerunning any code.

Why this matters:

• You don't need to re-run your filter when data changes — it happens automatically.
• Multiple transformations can share the same source without duplicating data.
• Operations are typically much faster than copying entire datasets.

When you call a table operation, Deephaven establishes the dependency and computes the initial result. That dependency remains active — if the source data changes, downstream tables update automatically without you re-running code. Operations like view are an exception: they store the formula but defer evaluation until values are actually accessed, which saves memory for columns you rarely read.

Formulas run in the engine

When you write a formula string like "Y = X * 2", that code runs inside Deephaven's Java-based engine, which is optimized for processing millions of rows efficiently.

The engine parses and executes the string "Y = Math.sqrt(X * X + 1)".

This has important implications:

• Java methods available: Use Math.sqrt, String methods like toUpperCase, and other Java standard library methods.
• Groovy variables accessible: Variables from your Groovy script are available inside formulas via the query scope.
• Much faster: The engine processes data in optimized batches, not one row at a time.

Calling Groovy from formulas

You can call Groovy methods and closures from formulas:

Closures assigned to top-level variables are available in query strings. The engine calls your closure once per row, which is slower than pure-engine formulas. For performance-critical code, prefer engine-native expressions.

Your code doesn't run row-by-row

This is the most common surprise for new users. Consider this code:

You might expect Value to be [1, 2, 3, 4, 5]. But the engine can evaluate rows in any order, potentially in parallel across multiple threads. You might get [3, 1, 4, 2, 5] or something else entirely — and results may differ between runs.

The rule: Formulas should be stateless — the result for row N should depend only on the input values for row N, not on what happened when processing other rows. Immutable variables (like configuration values) are fine; mutable state and order-dependent logic are not.

Replacing counters with row positions

If you need deterministic row numbering on static or append-only tables, use ii (the row position) instead of a counter:

For more complex cases involving state, see parallelization and the serial execution options. For details on ii and other special variables, see the reference documentation.

Static vs. live: understanding mutability

Deephaven tables come in two flavors:

• Static tables: Data that doesn't change. Loaded from files, created with emptyTable or newTable, or snapshots of live data.
• Refreshing (live) tables: Data that updates continuously. Created with timeTable, connected to streams, or other real-time sources.

The key insight: Most transformations on live tables produce live results — filter a live table, and the filtered result updates automatically. snapshot is a deliberate exception: it's a transformation that takes a live table and returns a static copy at that single point in time.

You don't need to poll for changes or re-run queries — the engine handles propagation automatically.

Moving data between Groovy and Deephaven

Data lives in two places: Groovy variables and Deephaven tables. Understanding when data moves between them helps you write efficient code.

From Groovy to Deephaven

When you create a table from Groovy data, the engine copies that data:

From Deephaven to Groovy

When you extract data back to Groovy, you can copy column values into an array. This is a snapshot, not a live view — changes to the table won't affect the array:

For live tables, consider using snapshot to get a static copy at a specific moment in time.

Performance tips

Moving data between Groovy and Deephaven takes time. For large datasets:

• Keep data in Deephaven tables and use engine operations (fast).
• Avoid repeatedly extracting data in loops (slow).
• Use snapshots strategically, not in tight loops.

What you can build

Deephaven isn't just a table engine — it's a platform for building data applications.

Data sources and sinks

Client-server architecture

Deephaven runs as a server. Multiple clients can connect simultaneously:

• Web UI: Built-in interactive console and grids
• Python client: from pydeephaven import Session
• JavaScript client: For web applications
• Java/C++ clients: For high-performance integrations

Clients that subscribe to the same table see consistent, live updates. You can share tables between sessions with shared tickets.

Common patterns

Pattern: Prefer engine operations over loops

See aggBy for all aggregation options.

Pattern: Use ii and i instead of counters

Pattern: Use view for lightweight derived columns

Use view when you want derived columns without storing them. Use update when you need the results cached for repeated access or downstream operations.

Pattern: Compose queries step by step

Build complex analytics by chaining simple operations. Each step produces a table you can inspect, reuse, or build on:

Each intermediate table (cleaned, enriched) is a first-class object you can display, join, or use as input to further operations.

Pattern: Same code for batch and streaming

Write your logic once — it works identically on files and live streams:

No need for separate batch and streaming codebases.

Pattern: Partition large datasets

Split data by key and process each partition efficiently:

Partitioned tables let you parallelize processing, quickly retrieve subtables by key, and improve filter performance in loops. See partitionBy and Partitioned tables for details.

Pitfalls to avoid

Pitfall: Stateful functions in formulas

Pitfall: Extracting data unnecessarily

Quick reference

Related documentation

• Quickstart — Get Deephaven running
• Crash course — Hands-on tutorial
• Table types
• Deephaven's live DAG
• Create tables
• Select, view, and update