Working Live vs. Static Data

Syntax Differences

Live tables and static tables in Deephaven support identical functionality. Every table operation that can be applied to a static table can also be applied to a live table. The only difference in syntax is when initially retrieving the data.

  • db.t("<namespace>", "<table>") - Retrieves a static table. The result table will never change.
  • db.i("<namespace>", "<table>") - Retrieves a live table. As new rows are imported into the system, they are appended to the end of the result table.

Tip

The examples above do not include a filter on the partitioning column. However, you should always filter on the partitioning column when pulling up a table.

Performance Considerations

When working with live data, the query engine must update the results of query operations as data changes in each operation's source tables. The results of all query operations update to reflect changing data, including:

  • Filters (used with where())
  • Formulas (such as used with update() or updateView())
  • Aggregations (such as sumBy(), avgBy(), etc.)
  • Joins (such as naturalJoin() and aj(), etc.)
  • Sorts (with sort() and sortDescending())

When running a complex query on fast-changing data, the amount of processing required to update the results of each operation may overwhelm the query server's hardware.

An overwhelmed query processor results in markedly decreased performance. A worker struggling to keep up with changing data will present clear signs in the form of a degraded user experience such as the following:

  • Sorting and filtering tables will take longer than expected.
  • New queries run from an interactive console will take longer than expected.
  • Tables will refresh infrequently (far less often than once per second). As a result, the data in tables may not be acceptably current.

Note

To learn more about best practices and methods for working with live data, see Common Causes of Performance Issues with Real-Time Data.

WindowCheck Utility

The WindowCheck utility adds a Boolean column that automatically ticks when a new value in a Timestamp column passes a defined threshold. This allows users to flag old data in a way that minimizes tick expansion. For example, this feature could be used to filter out old or new data, or format the table based on time (e.g., format the color of old items).

When the WindowCheck utility is applied to a column:

  • the content of the column would show true if the difference since the previous Timestamp value is less than the defined threshold;
  • the WindowCheck column would show false if the difference was greater than the threshold.

The syntax follows:

.addTimeWindow(t, timestampColumn, windowNanos, inWindowColumn)

This query defines four arguments within the .addTimeWindow method:

  • t - the source table
  • timestampColumn - specifies the timestamp column to monitor in table
  • windowNanos - how many nanoseconds in the past a timestamp can be before it is out of the window
  • inWindowColumn - the name of the new Boolean column
tt = timeTable("00:00:01")
windowCheck = com.illumon.iris.db.tables.utils.WindowCheck.addTimeWindow(tt, "Timestamp", 1*SECOND, "InWindow")

from deephaven import *

tt = timeTable("00:00:01")
windowCheck = WindowCheck.addTimeWindow(tt, "Timestamp", 1 * dbtu.SECOND, "InWindow")

The resultant table (windowCheck) ticks whenever the timetable (tt) ticks:

img