Read and write tables in Deephaven format

This guide demonstrates how to persist Deephaven tables using Deephaven Core+'s partitioned, columnar, random-access persistence format. This format enables real-time persistence at low latency and massive scale, supporting both high ingestion rates and large-scale fanout for delivery to workers.

Note

If you want to work with tables in Deephaven using system tables with schemas, see the Data guide. This page covers reading and writing individual table files using EnterpriseTableTools methods.

About the format

Deephaven tables are live data structures that support real-time updates. Deephaven Core+ provides a persistence format that allows these tables to be written to and read from disk.

When you write a table, it's stored as a directory containing multiple files (metadata, column data, etc.) - for example, a table at /path/to/users might contain files like table.tbl, Name.sym.bytes, Name.sym, Name.dat, and Age.dat. You only need to reference the directory path - the internal file structure is managed automatically.

When to use this format

The Deephaven persistence format is designed for specific use cases:

Use this format when:

Real-time persistence is required: The format enables the Deephaven Data Import Server to support real-time persistence at low latency and massive scale, both in terms of ingestion rate and fanout for delivery to workers.
High-performance I/O is critical: The format maximizes parallelism for fast reading and writing.
Efficient appends are needed: The format is optimized for efficient live append operations.
Working within the Deephaven ecosystem: Tables will only be accessed by Deephaven applications.

Use other formats when:

Historical data storage: For historical data, Parquet or Iceberg are recommended. While the Deephaven format can be used for historical storage, Parquet and Iceberg offer better lifecycle management, snapshotted data delivery, and schema evolution capabilities.
Interoperability is required: The format is proprietary and not accessible by external tools. Use Parquet or Iceberg for compatibility with other systems.
Compression is needed: The format does not support compression. Use Parquet or Iceberg for compressed storage.
Schema evolution is important: Formats like Iceberg provide better schema evolution support.

Reading tables

Python

import deephaven_enterprise.table_tools as table_tools

# Read a table from disk
table = table_tools.read_table("/path/to/your/table/")

Groovy

import io.deephaven.enterprise.table.EnterpriseTableTools

// Read a table from disk
table = EnterpriseTableTools.readTable("/path/to/your/table/")

The readTable method loads a table that was previously persisted in Deephaven format. The path should point to the directory containing the table data.

Writing tables

Persisting a table to disk in Deephaven format is straightforward - the only arguments needed are a table and a path.

import io.deephaven.enterprise.table.EnterpriseTableTools

// Write a table to disk
EnterpriseTableTools.writeTable(myTable, "/path/to/output/table/")

import deephaven_enterprise.table_tools as table_tools

# Write a table to disk
table_tools.write_table(my_table, "/path/to/output/table/")

The writeTable method persists a table to disk in Deephaven format. The specified path will be created as a directory containing the table's data files.

Complete example

This example demonstrates creating a table, writing it to disk, and reading it back.

Python

from deephaven import new_table
from deephaven.column import int_col, double_col, string_col
import deephaven_enterprise.table_tools as table_tools

# Create a sample table
grades = new_table(
    [
        string_col("Name", ["Ashley", "Jeff", "Rita", "Zach"]),
        int_col("Test1", [92, 78, 87, 74]),
        int_col("Test2", [94, 88, 81, 70]),
        int_col("Average", [93, 83, 84, 72]),
        double_col("GPA", [3.9, 2.9, 3.0, 1.8]),
    ]
)

# Write the table to disk
table_tools.write_table(grades, "/tmp/grades/")

# Read it back
grades_read = table_tools.read_table("/tmp/grades/")

Groovy

import io.deephaven.enterprise.table.EnterpriseTableTools

// Create a sample table
grades = newTable(
    stringCol("Name", "Ashley", "Jeff", "Rita", "Zach"),
    intCol("Test1", 92, 78, 87, 74),
    intCol("Test2", 94, 88, 81, 70),
    intCol("Average", 93, 83, 84, 72),
    doubleCol("GPA", 3.9, 2.9, 3.0, 1.8)
)

// Write the table to disk
EnterpriseTableTools.writeTable(grades, "/tmp/grades/")

// Read it back
grades_read = EnterpriseTableTools.readTable("/tmp/grades/")

Read and write tables in Deephaven format

About the format

When to use this format

Reading tables

Python

Groovy

Writing tables

Complete example

Python

Groovy

Related documentation