IcebergTableWriter

The IcebergTableWriter class is responsible for writing Deephaven tables to Iceberg tables. Each instance is associated with a single IcebergTableAdapter and is used to write multiple Deephaven tables to a single Iceberg table.

Constructors

The IcebergTableWriter class is constructed with the IcebergTableAdapter.tableWriter method:

import io.deephaven.iceberg.util.*

icebergTableWriter = icebergTableAdapter.tableWriter(IcebergWriteInstructions writeInstructions)

Parameters

When constructing an IcebergTableWriter, the IcebergWriteInstructions class specifies instructions to use when writing to the Iceberg table.

Methods

  • append: Append the Deephaven table(s) to the Iceberg table using the specified write instructions.
  • writeDataFiles: Write data from the Deephaven table(s) to an Iceberg table without creating a new snapshot.

Examples

The following example creates an IcebergTableWriter from an IcebergTableAdapter and appends two tables with identical definitions to an Iceberg table. It uses the Docker deployment defined in the Deephaven and Iceberg guide.

import io.deephaven.iceberg.util.*
import io.deephaven.extensions.s3.*
import org.apache.iceberg.catalog.*

restAdapter = IcebergTools.createAdapter(
    "minio-iceberg",
    [
        "type": "rest",
        "uri": "http://rest:8181",
        "client.region": "us-east-1",
        "s3.access-key-id": "admin",
        "s3.secret-access-key": "password",
        "s3.endpoint": "http://minio:9000",
        "io-impl": "org.apache.iceberg.aws.s3.S3FileIO"
    ]
)

icebergTaxis = restAdapter.loadTable("nyc.taxis")

// Create sample data tables
sourceData2024 = emptyTable(100).update("Year = 2024", "X = i", "Y = 2 * X", "Z = randomDouble(-1, 1)")
sourceData2025 = emptyTable(50).update("Year = 2025", "X = 100 + i", "Y = 3 * X", "Z = randomDouble(-100, 100)")

// Get table definition and create Iceberg table
sourceDef = sourceData2024.getDefinition()
sourceAdapter = restAdapter.createTable("nyc.source", sourceDef)

// Configure writer options
writerOptions = TableParquetWriterOptions.builder()
    .tableDefinition(sourceDef)
    .build()

// Create writer and append data
sourceWriter = sourceAdapter.tableWriter(writerOptions)
sourceWriter.append(IcebergWriteInstructions.builder()
    .addTables(sourceData2024, sourceData2025).build())