IcebergTableWriter

The IcebergTableWriter class writes Deephaven tables to an Iceberg table. Each instance is associated with a single IcebergTableAdapter. A single instance of an IcebergTableWriter can write as many Deephaven tables as needed to the associated Iceberg table if the Deephaven tables all have the same definition. It is far more efficient to create a single IcebergTableWriter to write multiple Deephaven tables to the same Iceberg table than to create a new IcebergTableWriter for each Deephaven table.

Constructors

An IcebergTableWriter is constructed using the table_writer method of an IcebergTableAdapter.

Methods

  • append: Append the provided Deephaven table(s) as new partitions to the existing Iceberg table in a single snapshot.

Examples

The following example writes a Deephaven table to an Iceberg table using an IcebergTableWriter:

from deephaven.experimental import iceberg
from deephaven import empty_table

source = empty_table(10).update(["X = i", "Y = 0.1 * X", "Z = Y ** 2"])

local_adapter = iceberg.adapter_s3_rest(
    name="minio-iceberg",
    catalog_uri="http://rest:8181",
    warehouse_location="s3a://warehouse/wh",
    region_name="us-east-1",
    access_key_id="admin",
    secret_access_key="password",
    end_point_override="http://minio:9000",
)

writer_options = iceberg.TableParquetWriterOptions(table_definition=source_def)

source_writer = source_adapter.table_writer(writer_options=writer_options)

source_writer.append(iceberg.IcebergWriteInstructions(source))