Table iterators

This guide shows you how to iterate over table data in Python queries with Deephaven's built-in table iteration methods.

Note

These methods are for extracting data from Deephaven to Python, not for transforming table data. To transform table data, use table operations such as update to add columns, where to filter rows, or update_by for rolling and cumulative calculations. See Recipes, not loops for why table operations outperform Python loops.

Iteration methods

Deephaven offers the following table methods to iterate over table data:

These methods all return generators that yield Python data structures containing table data. Generators are efficient because they minimize copies of data and only load data into memory when needed. You can iterate over a generator only once. After that, it's exhausted.

The methods differ in two ways:

Row-based and chunk-based methods both copy data into Python in chunks of rows. The row-based methods then build a Python dictionary or tuple for every row, while the chunk-based methods build one per chunk. Chunk-based methods do less per-row Python work, but your code must work with arrays instead of single values.

One row at a time

iter_dict and iter_tuple return generators that yield one row at a time. iter_dict yields a dictionary with column names as keys. iter_tuple yields a named tuple with column names as attributes.

The following example iterates over a table one row at a time and prints its values:

The third loop unpacks each row into x and y by position, so it depends on the table's column order. See Column order and unpacking before you use it in a query whose columns can change.

The row-based methods also let you choose a chunk size with the chunk_size argument. This chunk size is the number of rows Deephaven copies from the table into Python at a time. The default chunk size is 2048 rows. The loop still receives one row at a time.

One chunk of rows at a time

iter_chunk_dict and iter_chunk_tuple return generators that yield a chunk of rows at a time. iter_chunk_dict yields a dictionary with column names as keys. iter_chunk_tuple yields a named tuple with column names as attributes. Each value in the dictionary or named tuple is a NumPy array that holds that column's values for the chunk.

Set the number of rows in each chunk with the chunk_size argument. The default is 2048 rows.

The following example iterates over a table one chunk of rows at a time and prints its values. The table has 10 rows and the chunk size is 5, so each loop body runs twice, once per chunk:

Null values

Null values in primitive columns arrive as Deephaven's null constants, not as None or NaN. For example, a null int value arrives as -2147483648, the value of NULL_INT. This applies to both the row-based and chunk-based methods.

Null values in object columns arrive differently in the two kinds of method. The row-based methods return None. The chunk-based methods convert each column to a NumPy array, so a null String arrives as the string "None", a null Instant arrives as NaT, and a null Boolean arrives as True. If a String or Boolean column can contain nulls, use a row-based method, or filter out the nulls with where before you iterate.

The following example iterates over a table that has a null in its second row, first one row at a time and then one chunk at a time:

Iterate over specific columns

iter_dict, iter_tuple, iter_chunk_dict, and iter_chunk_tuple each take an optional column name or list of column names as their first argument. Pass it to iterate over only those columns.

The values always arrive in the table's column order, whatever order you list the columns in. iter_dict and iter_chunk_dict key each value by column name, so the list order doesn't matter for them. With iter_tuple and iter_chunk_tuple, list the columns in the same order as the table. Column order and unpacking explains what goes wrong otherwise.

The following example only iterates over the X and Z columns in the source table:

Column order and unpacking

Unpacking values in the for statement, as in for x, y in source.iter_tuple(), assigns them to variables in the table's column order. The tuple methods also depend on column order when you pass them a column list. Column order causes problems in these cases:

  • If a query reorders the columns, unpacking silently assigns values to the wrong variables.
  • If a query adds or removes columns, unpacking fails with a ValueError.
  • If you pass a column list to iter_tuple or iter_chunk_tuple in a different order from the table, the tuple fields are named in your list's order, but the values arrive in the table's order. Accessing values by name then reads the wrong columns.

There are two ways to guard against these problems:

In the following example, reordered has the same columns as source, but in the order Y, X. Accessing values by name gives the right values, positional unpacking assigns the Y values to x, and calling view first restores the correct order:

Iterate over refreshing tables

The update graph processes table updates in cycles. When you iterate over a refreshing table, Deephaven can lock the update graph for the whole iteration. The lock stops the table from updating, so all of the values come from one consistent view of the table. Iteration runs under an update graph lock in the following cases:

  • Code you run in the console or in a script already holds the update graph's exclusive lock, so no table can update until the code finishes.
  • On a thread you start yourself, the iterator acquires the update graph's shared lock and holds it until the generator is exhausted, closed, or destroyed. It does this only when auto_locking is enabled, which it is by default, and the thread isn't already marked as safe to run table operations without a lock.
  • Code on your own thread already holds a lock it acquired explicitly, for example with update_graph.shared_lock. The iterator uses that lock instead of acquiring another one.

Code that runs as part of an update cycle, such as a table listener callback, doesn't acquire a lock, because the update graph is already processing that cycle. A listener can safely iterate over the table it listens to and any table it registers as a dependency. Other refreshing tables may not have finished updating in that cycle.

While the update graph is locked, it can't start a new update cycle. That stops every refreshing table from updating, including tables you aren't iterating over. Keep loop bodies short so the update graph can start its next cycle sooner.

Important

A generator releases the lock it acquired and the resources it holds when it is exhausted, closed, or destroyed. A for loop that runs to completion, such as for row in table.iter_dict():, does this automatically. Close the generator yourself when you:

  • Store it in a variable and leave the loop early with break.
  • Consume it without a for loop, for example by calling next on it.

To close it, call its close method, or release the last reference to it with del or by setting it to None.

The following example reads only the first row, so it closes the generator itself: