Configuring input tables

Enable Input Tables

For users in your enterprise to create Input Tables, you first need to import __DbConfig.Tables directly as a user schema. This can be done either in a Deephaven console, or on a Deephaven server using the filesystem and the command line.

Option 1: Create the schema in a Deephaven console

Open a Deephaven console and run the following commands (note that this example uses Groovy):

schemaString =
 '''<Table namespace="__DbConfig" name="Tables" storageType="NestedPartitionedOnDisk" namespaceSet="User">
  <Partitions keyFormula="__WRITABLE_PARTITIONS__[abs((__NS_TN__ + Sym).hashCode() + __DAY_OF_YEAR__) % __NUM_WRITABLE_PARTITIONS__]" />

  <Column name="Date" dataType="String" columnType="Partitioning" />
  <Column name="Namespace" dataType="String" columnType="Normal" />
  <Column name="TableName" dataType="String" columnType="Normal" />
  <Column name="TableInputHandlerBlob" dataType="byte[]" columnType="Normal" />
  <Column name="_WorkerName" dataType="String" columnType="Normal" />
  <Column name="_ServerHostName" dataType="String" columnType="Normal" />
  <Column name="_ClientHostName" dataType="String" columnType="Normal" />
  <Column name="_ChangeTime" dataType="DateTime" columnType="Normal" />
  <Column name="_Deleted" dataType="Boolean" columnType="Normal" />
</Table>'''

schema = com.illumon.iris.db.schema.xml.SchemaXmlFactory.getXmlSchema(schemaString)
db.getSchemaService().createNamespace(com.illumon.iris.db.schema.NamespaceSet.USER, "__DbConfig")
db.getSchemaService().addSchema(schema)

Option 2: Create the schema from the command line

Create a file named __DbConfig.Tables.schema with the following contents:

<Table namespace="__DbConfig" name="Tables" storageType="NestedPartitionedOnDisk" namespaceSet="User">
  <Partitions keyFormula="__WRITABLE_PARTITIONS__[abs((__NS_TN__ + Sym).hashCode() + __DAY_OF_YEAR__) % __NUM_WRITABLE_PARTITIONS__]" />

  <Column name="Date" dataType="String" columnType="Partitioning" />
  <Column name="Namespace" dataType="String" columnType="Normal" />
  <Column name="TableName" dataType="String" columnType="Normal" />
  <Column name="TableInputHandlerBlob" dataType="byte[]" columnType="Normal" />
  <Column name="_WorkerName" dataType="String" columnType="Normal" />
  <Column name="_ServerHostName" dataType="String" columnType="Normal" />
  <Column name="_ClientHostName" dataType="String" columnType="Normal" />
  <Column name="_ChangeTime" dataType="DateTime" columnType="Normal" />
  <Column name="_Deleted" dataType="Boolean" columnType="Normal" />
</Table>

Import the schema:

dhconfig schema import -f <file> --key /etc/sysconfig/deephaven/auth/priv-iris.base64.txt

Once either option above is accomplished, it is possible to add and edit Input Tables in the Deephaven console.

Users must still be granted write access to Input tables via the ACL Editor before they will be able to create or edit Input Tables.

See Input Tables for more information.

Input Table Snapshotter Utility

The InputTable Shapshotter utility is used to reduce fetch-time for Input Tables with a large number of partitions. In effect, the utility will do a .lastBy() against the table, and store the latest “version” in today's partition. This ensures that we do not have to read back through all partitions when running a query against an Input Table.

In order to snapshot an InputTable, the table must be listed in:

/etc/sysconfig/illumon.d/resources/input_table_snapshotter_tasks.txt

The Namespace/TableName for each table to be snapshotted is listed; for example,

__DbConfig/Tables
SomeNameSpace/AnInputTable

The process may be run simply as:

sudo -u irisadmin /usr/illumon/latest/bin/iris input_table_snapshotter

Each table listed within the input_table_snapshotter_tasks.txt configuration will identify if the table has been successfully snapshotted or skipped (e.g., no data since last snapshot, does not exist, etc.).

[2021-03-10T15:01:43.382515-0500] - EMAIL - --------------------
[2021-03-10T15:01:43.382520-0500] - EMAIL - Tables Snapshotted:
[2021-03-10T15:01:43.382525-0500] - EMAIL - --------------------
[2021-03-10T15:01:43.382554-0500] - EMAIL - InputTableSnapshotTask-__DbConfig/Tables
[2021-03-10T15:01:43.382604-0500] - EMAIL -
[2021-03-10T15:01:43.382608-0500] - EMAIL - --------------------
[2021-03-10T15:01:43.382612-0500] - EMAIL - Tables Skipped:
[2021-03-10T15:01:43.382617-0500] - EMAIL - --------------------
[2021-03-10T15:01:43.382653-0500] - EMAIL - InputTableSnapshotTask-SomeNameSpace/AnInputTable: Table does not exist