Track processing time and measure latency

This guide explains how to track when Deephaven processes row modifications, which is useful for measuring end-to-end latency in real-time data pipelines.

Problem: formulas skip re-evaluation

When a row is modified, update and select recompute a formula column only if one of the formula's input columns changed. They learn which columns changed from the update's modified column set, a bitset that records which columns were modified in each update cycle.

Consider this example where we want to record when Deephaven processes a row. The time table ticks every second, adding a new row each time, and last_by keeps a single row that changes on each tick:

Open result in the IDE: SourceTime and Value change every second, but ProcessTime keeps the value it got when the row was created.

A formula like ProcessTime = now(), which calls the built-in now function, has no column dependencies. It evaluates once when the row is created. It doesn't re-evaluate when Value changes on later ticks, because none of its input columns changed.

In real-world pipelines, this matters when you receive data from external sources, like Kafka, that include a source timestamp. You want to compare that timestamp against the time Deephaven processed the update. A plain ProcessTime = now() formula can't give you that, because it doesn't re-evaluate when the row is modified.

Solution: force re-evaluation with withRecomputeOnModifiedRow

To force a formula to re-evaluate every time a row is modified, create the column with SelectColumnFactory.getExpression and mark it with withRecomputeOnModifiedRow. The Python API has no wrapper for these Java methods, so use jpy to call them directly:

With withRecomputeOnModifiedRow applied, ProcessTime updates on every tick that modifies the row, capturing when Deephaven processes each tick.

This approach requires three steps:

  1. Get the Java class. Use jpy.get_type to load SelectColumnFactory.
  2. Create and mark the column. getExpression returns a Java SelectColumn, the object form of a column formula. withRecomputeOnModifiedRow marks it to re-evaluate on every row modification.
  3. Call the Java update method on source.j_table. The Python update method doesn't recognize a SelectColumn created through jpy. The Java update takes a Collection of Selectable objects, and a SelectColumn is a Selectable. Two helpers handle the conversion:
    • j_array_list wraps the column in a Java list.
    • Table wraps the Java result in a Python table.

A column marked with withRecomputeOnModifiedRow works only with update and select. view, update_view, and lazy_update accept only formula strings. The engine also rejects such a column in those operations, even when you pass it through the Java API.

Example: measure end-to-end latency

Because ProcessTime updates on every tick, you can compare it against a source timestamp, such as an exchange's trade timestamp, to measure end-to-end latency. This helps identify bottlenecks in your data pipeline. In this example, the time table's Timestamp column stands in for a source timestamp. The result shows how long after each scheduled tick Deephaven processed it. With real data, use the timestamp your source provides, such as the KafkaTimestamp column that Kafka ingestion adds.

Using result from the solution above, subtract SourceTime from ProcessTime to calculate the latency:

LatencyNanos is the time, in nanoseconds, between the source timestamp and the moment Deephaven processed the row. LatencyMs converts it to whole milliseconds with the built-in nanosToMillis function. The conversion truncates toward zero, so a latency under one millisecond shows as 0.

Performance considerations

Using withRecomputeOnModifiedRow bypasses an important optimization. The modified column set allows the engine to skip formula evaluation when inputs haven't changed, which can significantly reduce CPU usage for complex formulas.

Use this feature only when a formula must re-evaluate on every row modification even though its input columns didn't change, such as recording processing time. For most use cases, the default behavior of evaluating formulas only when their inputs change is correct and more efficient.