Measure processing latency in queries

You may want to calculate timestamps in the middle of a query to track when Deephaven processes changes and measure latency. This guide explains how to force a formula to re-evaluate every time a row changes, even when the formula has no input dependencies that changed.

The problem: now() is not re-evaluated

Deephaven optimizes formula evaluation by only recomputing columns when their input dependencies change. This means a formula like ProcessTime = now() with no column dependencies will not be re-evaluated when existing rows are modified.

Consider this example:

When rows in source are modified, the ProcessTime column retains its original value because now() has no column dependencies that trigger re-evaluation.

Solution: Force formula re-evaluation with SelectColumnFactory.ofAlwaysUpdate

To force a formula to re-evaluate every time a row is modified, use SelectColumnFactory.ofAlwaysUpdate. This method creates a SelectColumn that bypasses the modified column set optimization and always re-evaluates when the engine sees a modification to the row.

Note

In Python, there is no native wrapper for SelectColumnFactory.ofAlwaysUpdate. Access the Java API via jpy interop as shown above.

Calculate end-to-end latency

A common use case is calculating the latency between when data originated (e.g., a quote timestamp from an exchange) and when Deephaven processed it.

How it works

When you use a regular formula like ProcessTime = now(), Deephaven tracks which columns the formula depends on. During an update cycle, the engine only re-evaluates formulas whose dependencies appear in the modified column set for that cycle.

SelectColumnFactory.ofAlwaysUpdate creates a SelectColumn with the alwaysEvaluate flag set to true. This tells the engine to always include this column in the re-evaluation set whenever the row is modified, regardless of whether the formula's dependencies changed.

Performance considerations

  • Use sparingly: The ofAlwaysUpdate mechanism bypasses an important optimization. Only use it when you genuinely need to capture the processing timestamp for every modification.
  • Combine operations: If you need multiple always-evaluate columns, pass multiple SelectColumn instances to a single update call rather than chaining multiple updates.
  • Monitor impact: Use the Query Operation Performance Log to monitor the performance impact of your queries.