IcebergTableWriter
The IcebergTableWriter class writes Deephaven tables to an Iceberg table. Each instance is associated with a single IcebergTableAdapter. A single instance of an IcebergTableWriter can write as many Deephaven tables as needed to the associated Iceberg table if the Deephaven tables all have the same definition. It is far more efficient to create a single IcebergTableWriter to write multiple Deephaven tables to the same Iceberg table than to create a new IcebergTableWriter for each Deephaven table.
Constructors
An IcebergTableWriter is constructed using the table_writer method of an IcebergTableAdapter.
Methods
append: Append the provided Deephaven table(s) as new partitions to the existing Iceberg table in a single snapshot.
Examples
The following example writes a Deephaven table to an Iceberg table using an IcebergTableWriter:
from deephaven.experimental import iceberg
from deephaven.experimental import s3
from deephaven import empty_table
source = empty_table(10).update(["X = i", "Y = 0.1 * X", "Z = Y ** 2"])
local_adapter = iceberg.adapter_s3_rest(
name="minio-iceberg",
catalog_uri="http://rest:8181",
warehouse_location="s3a://warehouse/wh",
region_name="us-east-1",
access_key_id="admin",
secret_access_key="password",
end_point_override="http://minio:9000",
)
s3_instructions = s3.S3Instructions(
region_name="us-east-1",
endpoint_override="http://minio:9000",
credentials=s3.Credentials.basic("admin", "password"),
)
writer_options = iceberg.TableParquetWriterOptions(
table_definition=source_def, data_instructions=s3_instructions
)
source_writer = source_adapter.table_writer(writer_options=writer_options)
source_writer.append(iceberg.IcebergWriteInstructions(source))