Read and write tables in Deephaven format

This guide demonstrates how to persist Deephaven tables using Deephaven Core+'s partitioned, columnar, random-access persistence format. This format enables real-time persistence at low latency and massive scale, supporting both high ingestion rates and large-scale fanout for delivery to workers.

Note

If you want to work with tables in Deephaven using system tables with schemas, see the Data guide. This page covers reading and writing individual table files using EnterpriseTableTools methods.

About the format

Deephaven tables are live data structures that support real-time updates. Deephaven Core+ provides a persistence format that allows these tables to be written to and read from disk.

When you write a table, it's stored as a directory containing multiple files (metadata, column data, etc.) - for example, a table at /path/to/users might contain files like table.tbl, Name.sym.bytes, Name.sym, Name.dat, and Age.dat. You only need to reference the directory path - the internal file structure is managed automatically.

When to use this format

The Deephaven persistence format is designed for specific use cases:

Use this format when:

  • Real-time persistence is required: The format enables the Deephaven Data Import Server to support real-time persistence at low latency and massive scale, both in terms of ingestion rate and fanout for delivery to workers.
  • High-performance I/O is critical: The format maximizes parallelism for fast reading and writing.
  • Efficient appends are needed: The format is optimized for efficient live append operations.
  • Working within the Deephaven ecosystem: Tables will only be accessed by Deephaven applications.

Use other formats when:

  • Historical data storage: For historical data, Parquet or Iceberg are recommended. While the Deephaven format can be used for historical storage, Parquet and Iceberg offer better lifecycle management, snapshotted data delivery, and schema evolution capabilities.
  • Interoperability is required: The format is proprietary and not accessible by external tools. Use Parquet or Iceberg for compatibility with other systems.
  • Compression is needed: The format does not support compression. Use Parquet or Iceberg for compressed storage.
  • Schema evolution is important: Formats like Iceberg provide better schema evolution support.

Reading tables

Python

Groovy

The readTable method loads a table that was previously persisted in Deephaven format. The path should point to the directory containing the table data.

Reading a table with no definition file

Tables persisted in the Deephaven format include a definition file - typically named table.tbl or <table name>.tbl - which specifies the column names and their data types. This file is required for reading the table, as the underlying column data files do not contain schema information. In cases where older merged partitions were written without a definition file, reading the table will fail with the error: Could not determine table definition for <path>: No definition present.

To read such a directory, supply the definition yourself. read_table_definition gets one from another directory of the same table that does have the file:

Any TableDefinitionLike works, so an existing table's definition can be used directly:

Two things to know about the supplied definition:

  • Partitioning columns are dropped. read_table reads one directory, and a partitioning column's value is recorded as a directory name rather than in a data file, so it cannot be reconstructed. Add it back yourself if you need it:

  • It is not validated against the data. A definition that does not match the files on disk fails when the mismatched column is read, not when read_table is called.

Note

For a system or user table with a registered schema, prefer db.historical_table(namespace, table_name). It takes the definition from the schema service, so it does not need a definition file on disk at all, and it returns all partitions rather than a single directory. read_table is for data with no registered schema or for directories outside the database root.

Writing tables

Persisting a table to disk in Deephaven format is straightforward - the only arguments needed are a table and a path.

The writeTable method persists a table to disk in Deephaven format. The specified path will be created as a directory containing the table's data files.

Complete example

This example demonstrates how to create a table, write it to disk, and read it back.

Python

Groovy