Access table metadata

This guide shows how to use the properties and methods of Deephaven's Table class to access the metadata for your table, and how to use the table_diff function to compare two tables.

A table's metadata describes the table itself: its table type and size, whether it is refreshing, and the data type of each column. You might use it to confirm that a column is an int or a double, or to check whether a column is a partitioning column. A partitioning column gets its values from how the data is laid out in storage, not from the data files. For example, the key=value directory names of a key-value partitioned Parquet directory become partitioning columns.

meta_table

The meta_table property creates a new table that contains the table's metadata. Specifically, this table contains one row for every column in the original table, with four columns: Name, DataType, ColumnType, and IsPartitioning. ColumnType is Normal for a normal column or Partitioning for a partitioning column, and IsPartitioning gives the same information as a true/false value.

Let's create a table of weather data for Miami, Florida.

We can access the metadata as follows:

The meta table has one row for each of the Month, Temp, and Rain columns of miami. Its DataType column shows that Temp is an int and Rain is a double. Metadata is most useful for a table you're unfamiliar with, but even here it confirms each column's type at a glance.

Metadata also shows how an operation changes column types. Let's create a table of weather data for Miami over three dates in January, then average the high and low temperatures by day.

The Temp column starts as an int column, but avg_by turns it into a double column. We can see this by hovering over the column header in the UI, and also by accessing the table's metadata.

Table properties

Like meta_table, the following are properties, so access them without parentheses. Each example prints the value the property returns.

columns

For a full list of a table's column definitions, use columns:

column_names

For a list of a table's column names, use column_names:

definition

To get a table's TableDefinition, a mapping from each column name to its column definition, use definition:

To see whether a table is a blink table, use is_blink. The static miami table is not a blink table. A time table created with blink_table=True is, because it keeps only the rows added in the current update cycle:

is_failed

To see whether a table is in a failure state and no longer usable, use is_failed:

is_flat

To see whether a table is guaranteed to be flat, meaning its row keys run contiguously from 0 to the number of rows minus 1, use is_flat:

is_refreshing

To see whether a table is refreshing, meaning its rows can change as new data arrives, use is_refreshing. The miami table never changes, so it is not refreshing. A time table adds a row as time passes, so it is:

size

To see the number of rows in a table, use size:

update_graph

To get the UpdateGraph that schedules a table's updates, for example to pass to the locking functions in deephaven.update_graph, use update_graph:

Table methods

The following methods return information about a table: its attributes, whether it has given columns, and a string preview of its first rows. Unlike the properties above, call them with parentheses.

attributes

Table attributes are metadata that the engine attaches to a table, such as AddOnly and AppendOnly, which mark a table whose rows are only ever added or appended. The attributes method returns all of a table's defined attributes as a dict.

The miami table has no attributes, so attributes returns an empty dict. An input table created from it has the InputTable attribute. Because it has no key columns, it is append-only, so it also has the AddOnly and AppendOnly attributes.

has_columns

Use has_columns to check whether a table contains one or more named columns. It returns True if the table contains every column you name, or False if it lacks any of them.

to_string

Use to_string to return the first n rows of a table as a pipe-delimited string.

  • num_rows is the number of rows from the beginning of the table to return. The default is 10.
  • cols is the column name(s) to include in the string. The default is None, which includes all columns.

table_diff

Use the table_diff function from the deephaven.table module to compare two tables and return the differences between them as a string. It returns an empty string if the tables are the same.

table_diff checks the tables in this order:

  • Table size.
  • Schema: missing or extra columns, column order, and column types.
  • Data, column by column, for every column with the same name and type in both tables. It records only the first difference in each column.

It stops once it has recorded max_diffs differences.

To illustrate, let's compare two tables whose B and C columns differ on alternating rows.

The two tables have the same schema, so the only differences are in the data. table_diff treats the first table as the actual result and the second as the expected one, so each line reads "encountered t1 value expected t2 value". It reports that column B first differs at row 1 and column C at row 0. Column A is identical in both tables.