Track processing time and measure latency

This guide explains how to track when Deephaven processes row modifications, which is useful for measuring end-to-end latency in real-time data pipelines.

Problem: formulas skip re-evaluation

When a row is modified, update and select recompute a formula column only if one of the formula's input columns changed. They learn which columns changed from the update's modified column set, a bitset that records which columns were modified in each update cycle.

Consider this example where we want to record when Deephaven processes a row. The time table ticks every second, adding a new row each time, and lastBy keeps a single row that changes on each tick:

Open result in the IDE: SourceTime and Value change every second, but ProcessTime keeps the value it got when the row was created.

A formula like ProcessTime = now(), which calls the built-in now function, has no column dependencies. It evaluates once when the row is created. It doesn't re-evaluate when Value changes on later ticks, because none of its input columns changed.

In real-world pipelines, this matters when you receive data from external sources, like Kafka, that include a source timestamp. You want to compare that timestamp against the time Deephaven processed the update. A plain ProcessTime = now() formula can't give you that, because it doesn't re-evaluate when the row is modified.

Solution: force re-evaluation with withRecomputeOnModifiedRow

To force a formula to re-evaluate every time a row is modified, use SelectColumnFactory.getExpression with withRecomputeOnModifiedRow:

With withRecomputeOnModifiedRow applied, ProcessTime updates on every tick that modifies the row, capturing when Deephaven processes each tick.

This approach requires three steps:

  1. Create a SelectColumn using SelectColumnFactory.getExpression. A SelectColumn is the object form of a column formula. Unlike a string formula passed to update, it can be marked to re-evaluate on every row modification.
  2. Wrap it with withRecomputeOnModifiedRow to bypass the modified column set optimization.
  3. Pass it as a Collection using Arrays.asList. The update method that accepts SelectColumn objects requires a Collection, not individual arguments.

A column marked with withRecomputeOnModifiedRow works only with update and select. view, updateView, and lazyUpdate reject such a column with an IllegalArgumentException.

Example: measure end-to-end latency

Because ProcessTime updates on every tick, you can compare it against a source timestamp, such as an exchange's trade timestamp, to measure end-to-end latency. This helps identify bottlenecks in your data pipeline. In this example, the time table's Timestamp column stands in for a source timestamp. The result shows how long after each scheduled tick Deephaven processed it. With real data, use the timestamp your source provides, such as the KafkaTimestamp column that Kafka ingestion adds.

Using result from the solution above, subtract SourceTime from ProcessTime to calculate the latency:

LatencyNanos is the time, in nanoseconds, between the source timestamp and the moment Deephaven processed the row. LatencyMs converts it to whole milliseconds with the built-in nanosToMillis function. The conversion truncates toward zero, so a latency under one millisecond shows as 0.

Performance considerations

Using withRecomputeOnModifiedRow bypasses an important optimization. The modified column set allows the engine to skip formula evaluation when inputs haven't changed, which can significantly reduce CPU usage for complex formulas.

Use this feature only when a formula must re-evaluate on every row modification even though its input columns didn't change, such as recording processing time. For most use cases, the default behavior of evaluating formulas only when their inputs change is correct and more efficient.