Extract table values

Deephaven provides several ways to extract values from tables into Python. Most queries don't need this. It's useful for debugging, for logging, and for code that reads table data directly, such as table listeners.

Choose a method based on how you need to find the values:

  • To read a value by its position in the table, use to_numpy. It copies the selected columns into a NumPy array, so it suits small tables or a few columns.
  • To loop over every value in a column, or over several columns of different types, use a for loop over a NumPy array or a table iterator. Table iterators read the table in chunks instead of copying whole columns into memory.
  • To read a value by the engine's internal row key, use a ColumnSource. This advanced option calls the Java API directly, so use it only when you need row keys rather than positions.

Note

These methods are for extracting values from Deephaven to Python, not for transforming table data. To create or modify columns, use operations like update. See Recipes, not loops! for more details.

to_numpy (positional index access)

A row's positional index is its position in the table, counting from 0. The recommended way to extract values by positional index is to_numpy. It converts the selected columns into a 2D NumPy array with one array column per table column. You then read values from the array by index.

To extract a single value by positional index, convert the column to a NumPy array, select its only column as a one-dimensional array, and read the value by index:

You can also use array slicing to extract multiple values:

Warning

to_numpy copies every row of the selected columns into memory, and for a refreshing (ticking) table it first takes a snapshot of the whole table. For large tables, keep only the columns you need with view, and reduce the rows with an operation such as where or head, before converting. Because the snapshot is a table operation, don't call to_numpy on a refreshing table inside a table listener. Use a table iterator, such as iter_tuple, instead.

Important

All columns that to_numpy converts must have the same data type. If you omit cols, it converts every column in the table, so every column must have the same type. For mixed types, convert columns separately, or loop over rows with a table iterator. Iterate with for loops shows both.

Iterate with for loops

To loop over a column's values, convert the column to a NumPy array with to_numpy, then use a standard Python for loop. This example loops over integers_array, created in to_numpy (positional index access):

To loop over several columns at once, one row at a time, the next examples use a table with two int columns and one String column:

to_numpy can convert columns of the same data type together into one 2D array. Convert columns of different data types separately:

To avoid copying whole columns into memory at once, or to loop over columns of different data types together, use a table iterator such as iter_tuple or iter_dict. Table iterators read the table in chunks of rows. See Table iterators for details.

Advanced: ColumnSource (row key access)

For advanced use cases, you can access the underlying Java ColumnSource through the table's j_object attribute, which holds the Java table that the Python Table wraps. This bypasses the Python API and isn't idiomatic Python, so use to_numpy for most use cases.

A ColumnSource provides access by row key, not positional index. A row key is the internal identifier the engine assigns to a row. It can differ from the row's positional index, especially in filtered tables or tables whose rows have been added, removed, or shifted. To learn how row keys and positional indices change as tables update, see the row keys and positional indices FAQ.

To find the row key for a position, look it up in the table's RowSet, the ordered set of row keys the table contains. Then read the value with a type-specific method:

Type-specific ColumnSource methods:

  • getInt(rowKey) for int columns
  • getLong(rowKey) for long columns
  • getDouble(rowKey) for double columns
  • getFloat(rowKey) for float columns
  • getShort(rowKey) for short columns
  • getByte(rowKey) for byte columns
  • getChar(rowKey) for char columns
  • getBoolean(rowKey) for Boolean columns
  • get(rowKey) for object columns

get also works on primitive columns, but it wraps each value in a Java object, so prefer the type-specific methods for primitives.

Note

Unlike to_numpy, which takes a snapshot of a refreshing table before copying it, a ColumnSource reads the table's live data. For a refreshing table, values read this way are consistent only while no update can change them: inside a listener that listens to the table or registers it as a dependency, or inside a shared_lock or exclusive_lock block. Console scripts already hold the exclusive lock, so this matters mainly for code that runs on other threads. See Engine locking.

Note

Inside a listener, use the getPrev* methods, such as getPrevInt, to read previous values, meaning each row's values before the current update cycle's changes. These methods take a previous row key, which you look up in the table's previous RowSet, for example result.j_object.getRowSet().prev().get(2).