Iceberg

Apache Iceberg is an open table format for large analytic datasets. Deephaven Enterprise supports reading Iceberg tables through the iceberg extended storage type, which maps Iceberg tables to Deephaven schemas. This enables querying Iceberg data alongside other Deephaven tables, with full support for Iceberg schema evolution and nested structures.

Overview

To use an Iceberg table in Deephaven:

  1. Configure catalog access — Specify how to connect to your Iceberg catalog (Glue, REST, Hive, etc.).
  2. Identify the table — Provide the Iceberg namespace and table name.
  3. Deploy the schema — Register the table with Deephaven's schema service.

Deephaven uses the iceberg extended storage type in schemas, which references the catalog configuration and table identifier. You can create these schemas programmatically (recommended) or manually via XML.

Configuration

Deephaven works with any Iceberg catalog that implements the standard Catalog interface. Use the type property to specify built-in catalog types (e.g., glue, rest, hive, hadoop, nessie, jdbc), or use catalog-impl to specify a custom implementation class.

The easiest way to configure Iceberg tables for Deephaven is to use the built-in inference provided by LoadTableOptions. Advanced users can customize the mapping with a Resolver, or with custom inference options.

Required properties

Every Iceberg catalog configuration requires:

  • type or catalog-impl — The catalog type or fully-qualified implementation class.

Most catalog types also require:

  • warehouse — The URI where table data is stored (e.g., s3://bucket/warehouse).

Additional properties depend on your catalog type. See Iceberg Catalog properties for the full reference.

If you have a working Spark configuration, that can typically be translated into the necessary Catalog properties by removing the Spark prefix.

For example, the following Spark properties:

Translate into the given BuildCatalogOptions:

See BuildCatalogOptions and LoadTableOptions for more details on these structures.

Deployment

An iris-schemamanagers user is required to deploy the Schema.

Serialization format

Caution

The creation and deployment of a Deephaven Iceberg schema is typically performed programmatically, as shown in the previous sections. Exercise caution when manually creating or editing a schema.

An Iceberg table is referenced in a Deephaven table's schema using an ExtendedStorage element with the attribute type set to iceberg.

Catalog element

The Catalog element is a serialization of the core BuildCatalogOptions. It is composed of a Name, Properties, and optional HadoopConfig element. The Properties element is a map of string keys to string values. The optional HadoopConfig element is optional and is an additional map for Hadoop catalogs. For example:

The injection attribute on the Properties element controls whether Deephaven may automatically add properties that work around known upstream issues and/or supply defaults needed for Deephaven's Iceberg usage. The valid values are enabled and disabled. It is recommended to set this to enabled.

Table element

The Table element is a serialization of the core LoadTableOptions. It is composed of a TableIdentifier, Resolver, and NameMapping element.

The Resolver element contains a ColumnInstructions, Schema, and optional PartitionSpec element. The ColumnInstructions element contains the mapping from Deephaven column names to Iceberg fieldId, Iceberg partitionFieldId, or type unmapped. The Schema element contains the Iceberg Schema JSON. The optional PartitionSpec element contains the Iceberg Partition Spec JSON.

The NameMapping element provides fallback field ids to be used when a data file does not contain field id information. It has three different types, specified via the type attribute.

The table type means to read the Name Mapping from the Iceberg Table property schema.name-mapping.default (see https://iceberg.apache.org/spec/#column-projection).

The empty type means to not use name mapping.

The json type uses Iceberg Name Mapping JSON.

Full example

This example uses a Glue catalog, but the same pattern applies to any Iceberg catalog — substitute your catalog's type and properties as needed.

Assume an existing Iceberg table mycatalog.cities with the following schema:

To create a new Deephaven Schema with namespace DhExample and table name Cities that references this Iceberg Table, we would execute the following once:

This would result in the following Deephaven Schema: