writeKeyValuePartitionedTable

The writeKeyValuePartitionedTable method writes a table to disk in Parquet format with partitioning columns written as key=value directories. For example, for a partitioning column Date, this creates a directory structure like Date=2021-01-01/<base_name>.parquet, Date=2021-01-02/<base_name>.parquet, etc.

Syntax

Parameters

ParameterTypeDescription
sourceTableTable

A table with partitioning columns defined in its table definition. The method calls partitionBy internally using these columns to create the nested directory structure. This is an advanced overload — most users should use the PartitionedTable overload by calling partitionBy first.

partitionedTablePartitionedTable

A partitioned table to write. The partitioned table's key columns are used to create the nested directory structure.

destinationDirString

The path or URI to the destination root directory to store partitioned data in nested format. Non-existing directories are created.

writeInstructionsParquetInstructions

Instructions for customizations while writing. Build using ParquetInstructions.builder() with options including:

  • setBaseNameForPartitionedParquetData(String) — Sets the base name for individual partitioned files. Supports tokens:
    • {uuid} — Replaced with a random UUID
    • {partitions} — Replaced with underscore-delimited partition values
    • {i} — Replaced with an auto-incremented integer
  • setGenerateMetadataFiles(boolean) — Whether to generate _metadata and _common_metadata files
  • addIndexColumns(String...) — Add index columns to write as sidecar tables
  • setTableDefinition(TableDefinition) — Set a custom table definition
  • Compression options: setCompressionCodecName("SNAPPY"), "GZIP", "ZSTD", "LZ4_RAW", "BROTLI", "UNCOMPRESSED"

Use ParquetInstructions.EMPTY for default settings.

Returns

None. Writes partitioned Parquet files to the specified directory.

Examples

Note

All examples in this document write data to the /data directory in Deephaven. For more information on this directory and how it relates to your local file system, see Docker data volumes.

Write a partitioned table

In this example, writeKeyValuePartitionedTable writes a partitioned table with the X column as the partitioning key:

This creates:

Write with custom base name

In this example, writeKeyValuePartitionedTable uses a custom base name:

This creates files like X=A/data-0.parquet, X=B/data-0.parquet, etc.

Write with metadata files

Generate metadata files to speed up reading:

This creates the partitioned files plus _metadata and _common_metadata files in the root directory.