Iceberg

Iceberg tables can be read as Deephaven historical tables using the Extended Storage feature. To configure Iceberg, you first use an Iceberg Endpoint, then discover and deploy a schema.

Configuration

The first step to linking Iceberg tables into Deephaven is configuring an IcebergEndpoint, which we refer to as simply an endpoint. The endpoint contains the parameters required to locate and connect to the Iceberg catalog and the data warehouse and the storage-specific parameters required to read the data. An endpoint is configured using the example below.

import io.deephaven.enterprise.iceberg.IcebergTools
import io.deephaven.extensions.s3.S3Instructions

// Create a new endpoint
endpoint = IcebergTools.newEndpoint()
        .catalogType("rest")                                 // The catalog is a REST catalog
        .catalogUri("http://mydata.com:8181")                // located at this URI.
        .warehouseUri("s3://warehouse/")                     // The data warehouse is an S3 warehouse at this URI
        .putProperties("s3.access-key-id", "my_access_key",  // These are the properties required by the Iceberg API.
                       "client.region"   , "us-east-1")      // See https://iceberg.apache.org/docs/nightly/configuration/#configuration
        .putSecrets("s3.secret-access-key", "s3.key")        // Include any named secrets
        .dataInstructions(S3Instructions.builder()           // Configure the S3 data parameters
                .regionName("us-east-1")
                .build())
        .build("my_company_iceberg");                        // Explicitly name the endpoint.

from deephaven_enterprise import iceberg
from deephaven.experimental import s3

# Create a new endpoint
endpoint = iceberg.make_endpoint("rest", \                  # The catalog is a REST catalog
              "http://mydata.com:8181", \                   # Located at this URI
              "s3://warehouse/", \                          # The data warehouse is an S3 warehouse at this URI
              s3.S3Instructions(region_name="us-east-1"), \ # Create the data instructions for reading data.
              endpoint_name="my_company_iceberg", \         # Explicitly name this endpoint
              properties={"s3.access-key-id" : "my_access_key", "client.region" : "us-east-1"}, # Set Iceberg configuration properties. See https://iceberg.apache.org/docs/nightly/configuration/#configuration
              secrets={"s3.secret-access-key" : "s3.key"})  # Include any named secrets

Properties

Endpoint properties are a key-value map of Iceberg configuration parameters to their values. Valid property keys can be found in the Iceberg documentation.

Secrets

Endpoint secrets are a key-value map where keys represent Iceberg configuration properties and values represent named references to secrets stored within Deephaven. When needed, the secrets are retrieved from Deephaven and merged into the properties before being used to access Iceberg.

Secrets may be stored either as a property in the Deephaven configuration file or as a JSON map in a protected file on disk. Secrets providers are visited in ascending priority order until one supplies a value or none can be found. More sophisticated secret stores are possible. Contact support for more information.

From Properties

Secrets may be stored in Deephaven configuration files as simple properties, such as s3.access_key=1234secret4321. The default priority of the Properties secrets provider is 100, which can be adjusted using the property PropertiesSecretsProvider.priority.

Warning

Property-based secrets are visible to anyone with access to the server, including access to persistent queries. It is not recommended to store sensitive information such as passwords this way.

From Files

Secrets may be stored in files on disk containing a simple JSON map. This format is more secure and better supports more complex secret values. You may configure multiple secrets files and their priorities using these properties:

Property	Description
`FileSecretsProvider.name.path`	The path to the secrets file for the provider `name`.
`FileSecretsProvider.name.priority`	The priority of the secrets provider `name`.

You may provide as many of these as you need, ensuring that each name is unique.

An example file:

{
  "s3.key": "some_secret_key",
  "secret_url": "https://verysecret.com:9001",
  "complicated": "<Secret type=\"important\"><Internal>Secrecy</Internal></Secret>"
}

Warning

If you supply secrets in files, they are visible to any process with access to that file. For example, if they must be visible to query workers, any user running query workers under the same operating system user can access them. This should be considered before storing secrets such as passwords in files.

See per-user workers for details on configuring different operating system users for different Deephaven users.

Deployment

The endpoint can be deployed to the Deephaven configuration as long as a name has been provided. Once deployed, you may reference the endpoint by name in the schema for Iceberg tables to avoid duplication.

// Deploy the endpoint to Deephaven configuration, failing if it already exists.
endpoint.deploy(false)

# Deploy the endpoint to Deephaven configuration, Overwriting if it already exists.
# The overwrite_existing parameter defaults to False. The deployment will fail if the endpoint already exists.
endpoint.deploy(overwrite_existing=True)

Discovery

Once an endpoint has been configured, you can discover an Iceberg table to create and deploy a Deephaven schema. If you have previously deployed an endpoint, you can retrieve it by name as well.

import io.deephaven.enterprise.iceberg.IcebergTools

// Load an endpoint that was already configured
endpoint = IcebergTools.getEndpointByName("my_company_iceberg")

// Discover a table derive the schema and deploy it, deriving the namespace and table name
// from the table identifier, referencing the endpoint by name.
discovery = IcebergTools.discover(DiscoveryConfig.builder()
                .tableIdentifier("market.trades")
                .endpoint(endpoint)
                .build())

discovery.deployWithEndpointReference()

from deephaven_enterprise import iceberg

# Load an endpoint that was already configured
endpoint = iceberg.get_named_endpoint("my_company_iceberg")

# Discover a table derive the schema and deploy it, deriving the namespace and table name
# from the table identifier, referencing the endpoint by name.
result = iceberg.discover("market.trades", endpoint)
result.deploy_named()

In the examples above, the Deephaven namespace and table name are derived directly from the Iceberg table identifier. You may specify your own by setting the namespace and tableName properties during discovery.

discovery = IcebergTools.discover(DiscoveryConfig.builder()
                .tableIdentifier("market.trades")
                .namespace("MarketUS")
                .tableName("EqTrades")
                .endpoint(endpoint)
                .build())

result = iceberg.discover(
    "market.trades", namespace="MarketUS", table_name="EqTrades", endpoint
)

Example

Below is a complete example that creates an endpoint, discovers a table, deploys a schema, and then fetches the table.

import io.deephaven.enterprise.iceberg.IcebergTools
import io.deephaven.enterprise.iceberg.discovery.DiscoveryConfig
import io.deephaven.extensions.s3.S3Instructions

// Create a new endpoint
endpoint = IcebergTools.newEndpoint()
        .catalogType("rest")
        .catalogUri("http://mydata.com:8181")
        .warehouseUri("s3://warehouse/")
        .putProperties("s3.access-key-id", "access_key",
                       "client.region" , "us-east-1")
        .putSecrets("s3.secret-access-key", "s3.key")
        .dataInstructions(S3Instructions.builder()
                .regionName("us-east-1")
                .build())
        .build("my_company_iceberg");

endpoint.deploy(true)

discovery = IcebergTools.discover(DiscoveryConfig.builder()
                .tableIdentifier("market.trades")
                .endpoint(endpoint)
                .build())

data = db.historicalTable("market", "trades")

from deephaven_enterprise import iceberg
from deephaven.experimental import s3

endpoint = iceberg.make_endpoint(
    "rest",
    "http://mydata.com:8181",
    "s3://warehouse/",
    s3.S3Instructions(region_name="us-east-1"),
    endpoint_name="my_company_iceberg",
    properties={"s3.access-key-id": "access_key", "client.region": "us-east-1"},
    secrets={"s3.secret-access-key": "s3.key"},
)
endpoint.deploy(True)

result = iceberg.discover("market.trades", endpoint)
result.deploy_named()

data = db.historical_table("market", "trades")