User tables

User tables in Deephaven are tables that exist within the Deephaven database that are part of a user namespace. Unlike tables in system namespaces, user tables can be created, modified, and deleted by users without administrative privileges. Once a user table has been added to the database, it can be retrieved using the historical or live table API. Additionally, user tables do not require you to deploy a schema manually. The schema is inferred from the table itself.

This guide covers what user tables are as well as how to add, modify, and delete them from Deephaven's database.

Types of user tables

There are several different types of user tables in Deephaven.

Partitioned

A partitioned user table is split into partitions by one or more partitioning columns. In practice, they are most often partitioned on a single column. The most common partitioning column used is a Date column, for which each unique date has its own dataset. Partitioned user tables can have both direct and live partitions. Users must first add a schema before creating or adding data to a partitioned user table.

Unpartitioned

An unpartitioned user table is not split into partitions. Thus, the data for an unpartitioned user table is written to a single location in memory. All unpartitioned tables are direct.

Direct

A direct user table is managed directly on the filesystem by the query worker. Direct user tables cannot add, remove, or modify rows to pre-existing partitions, and do not update after retrieval. Direct user tables are considered historical data, and are therefore accessed via the historical table API. Direct user tables can be either partitioned or unpartitioned.

Live

Live user tables are managed centrally by the Deephaven system in Deephaven format. They are ingested and served similarly to system intraday data. These tables tick after retrieval. They are accessed using the live table API. All live user tables are partitioned.

Input

Input tables are a special type of user table where data can not only be added, modified, and removed programmatically, but also via the UI similar to a spreadsheet. For more information on input tables, see the Input table user guide.

Add, modify, and delete user tables

The Deephaven API allows user tables to be added, modified, and removed from the user namespace. The following sections provide examples of how to add, modify, and delete user tables.

Add user tables

Each of the following subsections deals with adding a user table of one of the previously mentioned types.

Unpartitioned

The following example adds an unpartitioned user table to the user namespace UserNamespace.

tableToWrite = newTable(
    doubleCol("Doubles", 3.1, 5.45, -1.0),
    stringCol("Strings", "Creating", "New", "Tables")
)

// Add the unpartitioned table assuming table UserNamespace.MyTable does not exist
db.addUnpartitionedTable("UserNamespace", "MyTable", tableToWrite)

// Get the table from the namespace
fromDisk = db.historicalTable("UserNamespace", "MyTable")

from deephaven import new_table
from deephaven.column import string_col, double_col

table_to_write = new_table(
    [
        double_col("Doubles", [3.1, 5.45, -1.0]),
        string_col("Strings", ["Creating", "New", "Tables"]),
    ]
)

# Add the unpartitioned table assuming table UserNamespace.MyTable does not exist
db.add_unpartitioned_table("UserNamespace", "MyTable", table_to_write)

# Get the table from the namespace
from_disk = db.historical_table("UserNamespace", "MyTable")

Partitioned

The following example writes the same table as the previous example as a partition to a partitioned user table. It first adds the schema for the partitioned user table before adding the partition.

import io.deephaven.engine.util.TableTools

// Create a table to write
tableToWrite = TableTools.newTable(
        TableTools.doubleCol("Doubles", 3.1, 5.45, -1.0),
        TableTools.stringCol("Strings", "Creating", "New", "Tables")
)

// Add the partitioned table schema
db.addPartitionedTableSchema("UserNamespace", "TableName", "Date", tableToWrite.getDefinition())

// Add a partition to the table
db.addTablePartition("UserNamespace", "TableName", "2023-01-01", tableToWrite)

// Read all partitions of the table back
fromDisk = db.historicalTable("UserNamespace", "TableName")

from deephaven import new_table
from deephaven.column import double_col, string_col

# Create a table to write
table_to_write = new_table(
    [
        double_col("Doubles", [3.1, 5.45, -1.0]),
        string_col("Strings", ["Creating", "New", "Tables"]),
    ]
)

# Add the partitioned table schema
db.add_partitioned_table_schema("UserNamespace", "TableName", "Date", table_to_write)

# Add a partition to the table
db.add_table_partition("UserNamespace", "TableName", "2023-01-01", table_to_write)

# Read all partitions of the table back
from_disk = db.historical_table("UserNamespace", "TableName")

Modify user tables

Only partitioned user tables can be modified. Unpartitioned user tables must be deleted and re-added to "change" their data.

Update schema

There are two restrictions to updating the schema of a partitioned user table:

• You cannot change the type of an existing column
• You cannot change the partitioning column

The following example updates the schema of the partitioned user table TableName in the user namespace UserNamespace. Updating the schema returns a boolean value indicating whether the update occurred or if there was already an identical definition.

wasUpdated = db.updatePartitionedTableSchema("UserNamespace", "TableName", source.getDefinition())

was_updated = db.update_partitioned_table_schema("UserNamespace", "TableName", source)

Add rows to a live partition

You can modify live partitions by adding data to them. The following example adds the rows from the source table to the 2025-01-01 partition of user table TableName found in the user namespace UserNamespace:

ref = db.appendLiveTable(
    "UserNamespace", "TableName", "2025-01-01", source
)

ref = db.append_live_table("UserNamespace", "TableName", "2025-01-01", source)

Additionally, the following example adds rows every time source ticks. Doing so returns a reference that must be held onto to keep the operation active. When finished with the append, you can close the reference to stop appending and clean up any related resources.

ref = db.appendLiveTableIncremental(
    "UserNamespace", "TableName", "2025-01-01", source
)
// Some time later when the append is finished
ref.close()

ref = db.append_live_table_incremental(
    "UserNamespace", "TableName", "2025-01-01", source
)
# Some time later when the append is finished
ref.close()

Delete user tables

Unpartitioned

The following example deletes the unpartitioned user table TableName from the user namespace UserNamespace. Deleting an unpartitioned table returns a boolean value indicating whether or not the operation was successful.

wasDeleted = db.deleteUnpartitionedTable("UserNamespace", "TableName")

was_deleted = db.delete_unpartitioned_table("UserNamespace", "TableName")

Partitioned

The following example deletes the partitioned user table PartitionedTableName from the user namespace UserNamespace. Deleting a partitioned table returns a boolean value indicating whether or not the operation was successful.

wasDeleted = db.deletePartitionedTable("UserNamespace", "PartitionedTableName")

was_deleted = db.delete_partitioned_table("UserNamespace", "PartitionedTableName")

Individual partitions of a partitioned user table can be deleted, leaving all other partitions untouched. The following example deletes both the direct and live partition 2025-01-01 from the partitioned user table PartitionedTableName in the user namespace UserNamespace. Deleting a partition returns a boolean value indicating whether or not the operation was successful.

wasDeletedDirect = db.deleteTablePartition("UserNamespace", "PartitionedTableName", "2025-01-01")

wasDeletedLive = db.deleteLiveTablePartition("UserNamespace", "PartitionedTableName", "2025-01-01")

was_deleted_direct = db.delete_table_partition(
    "UserNamespace", "PartitionedTableName", "2025-01-01"
)

was_deleted_live = db.delete_live_table_partition(
    "UserNamespace", "PartitionedTableName", "2025-01-01"
)

Related documentation

• How do I manually delete user tables?
• Input tables
• Database Pydoc
• WritableDatabase Javadoc