Access table metadata

This guide shows how to use the methods of Deephaven tables to access the metadata for your table, and how to use the static string and diff methods from TableTools to preview a table and compare two tables.

A table's metadata describes the table itself: its table type and size, whether it is refreshing, and the data type of each column. You might use it to confirm that a column is an int or a double, or to check whether a column is a partitioning column. A partitioning column gets its values from how the data is laid out in storage, not from the data files. For example, the key=value directory names of a key-value partitioned Parquet directory become partitioning columns.

meta

The meta method creates a new table that contains the table's metadata. Specifically, this table contains one row for every column in the original table, with four columns: Name, DataType, ColumnType, and IsPartitioning. ColumnType is Normal for a normal column or Partitioning for a partitioning column, and IsPartitioning gives the same information as a true/false value.

Let's create a table of weather data for Miami, Florida.

We can access the metadata as follows:

The meta table has one row for each of the Month, Temp, and Rain columns of miami. Its DataType column shows that Temp is an int and Rain is a double. Metadata is most useful for a table you're unfamiliar with, but even here it confirms each column's type at a glance.

Metadata also shows how an operation changes column types. Let's create a table of weather data for Miami over three dates in January, then average the high and low temperatures by day.

The Temp column starts as an int column, but avgBy turns it into a double column. We can see this by hovering over the column header in the UI, and also by accessing the table's metadata.

Attributes

Table attributes are metadata that the engine attaches to a table, such as AddOnly and AppendOnly, which mark a table whose rows are only ever added or appended. The following methods read a table's attributes.

getAttribute

Use the getAttribute method to get the value of a specific table attribute.

The example below creates an append-only input table from miami. The input table has the InputTable, AddOnly, and AppendOnly attributes. The example gets the value of its AddOnly attribute.

getAttributes

The getAttributes method returns all the attributes in the source table's AttributeMap. To return only the attributes whose names match a predicate, use the overload that takes a Predicate<String>. The example uses the miamiInput input table from the getAttribute example.

getAttributeKeys

getAttributeKeys returns an immutable set of the names of all the source table's attributes that have values. The example uses the miamiInput input table from the getAttribute example.

hasAttribute

The hasAttribute method returns true if the source table has the specified attribute, and false otherwise. The example checks the miamiInput input table from the getAttribute example.

Schema

The following methods describe a table's columns: their definitions, names, and number.

getDefinition

Use the getDefinition method to get the source table's TableDefinition.

getColumnNames

For a list of a table's column names, call getColumnNames on its TableDefinition:

hasColumns

The hasColumns method returns true if the source table contains every column you name, or false if it lacks any of them.

numColumns

The numColumns method returns the number of columns in the source table.

Table status

The following methods report a table's current state: whether it is a blink, empty, failed, flat, or refreshing table, how many rows it has, and which update graph it belongs to.

The isBlink method returns true if the source table is a blink table, and false otherwise. The static miami table is not a blink table. A time table built with blinkTable(true) is, because it keeps only the rows added in the current update cycle:

isEmpty

The isEmpty method returns true if the source table has no rows, and false otherwise.

isFailed

The isFailed method returns true if the source table is in a failure state and no longer usable, and false otherwise.

isFlat

The isFlat method returns true if the source table is guaranteed to be flat, meaning its row keys run contiguously from 0 to the number of rows minus 1, and false otherwise.

isRefreshing

The isRefreshing method returns true if the source table is refreshing, meaning its rows can change as new data arrives, and false otherwise. The miami table never changes, so it is not refreshing. A time table adds a row as time passes, so it is:

size

The size method returns the number of rows in the source table.

getUpdateGraph

To get the UpdateGraph that schedules a table's updates, use getUpdateGraph:

string

Use the static string method from TableTools to return the first n rows of a table as a pipe-delimited string. Pass the number of rows as the second argument; without it, string returns the first 10 rows. Any column names you pass after that limit the string to those columns.

diff

Use the static diff method from TableTools, which Groovy sessions import by default, to compare two tables and return the differences between them as a string. It returns an empty string if the tables are the same.

diff checks the tables in this order:

  • Table size.
  • Schema: missing or extra columns, column order, and column types.
  • Data, column by column, for every column with the same name and type in both tables. It records only the first difference in each column.

It stops once it has recorded the number of differences you pass as its third argument.

To illustrate, let's compare two tables whose B and C columns differ on alternating rows.

The two tables have the same schema, so the only differences are in the data. diff treats the first table as the actual result and the second as the expected one, so each line reads "encountered t1 value expected t2 value". It reports that column B first differs at row 1 and column C at row 0. Column A is identical in both tables.