Export Deephaven tables to Parquet files

The Deephaven Parquet module provides tools to integrate Deephaven with the Parquet file format. This document covers writing Deephaven tables to single Parquet files, key-value partitioned Parquet directories, and flat partitioned Parquet directories. You can write each layout to local storage, S3, or Google Cloud Storage.

By default, Deephaven writes Parquet files with SNAPPY compression, and the codec applies to the whole file. Write to a single Parquet file shows how to choose a different codec, and the Parquet instructions document lists the available options.

First, create some tables to use in the examples in this guide.

Write to a single Parquet file

To local storage

Write a Deephaven table to a single Parquet file with ParquetTools.writeTable. Pass the table as the sourceTable argument and the destination file path as the destination argument. The destination must end with the .parquet file extension.

To set compression and other options, pass a ParquetInstructions object as the optional writeInstructions argument. ParquetTools provides ready-made instructions for common codecs. For example, ParquetTools.GZIP is equivalent to ParquetInstructions.builder().setCompressionCodecName("GZIP").build().

Write _metadata and _common_metadata files by calling ParquetInstructions.Builder.setGenerateMetadataFiles(true). These files hold the schema and other metadata for the Parquet files that the write produces, and Deephaven places them in the destination directory. Readers use them to find the data files and the full schema without listing the directory tree, so they matter most for the partitioned Parquet directories described later in this guide. See Metadata-partitioned directories for what each file contains.

To S3

Use ParquetTools.writeTable to write Deephaven tables to Parquet files on S3. The destination should be the URI of the destination file in S3. Supply an instance of the S3Instructions class to the setSpecialInstructions method of ParquetInstructions.Builder to specify the details of the connection to the S3 instance. A gs:// URI writes to Google Cloud Storage with the same instructions. The example defines s3Instructions once. The later S3 examples in this guide reuse it.

Partitioned Parquet directories

Deephaven can also write tables to a directory of Parquet files instead of a single file. It supports two directory layouts:

• A key-value partitioned directory nests its Parquet files in subdirectories named key=value, one level per partitioning column. A partitioning column's values name the subdirectories, such as Class=Math, instead of being stored inside each file.
• A flat partitioned directory holds its Parquet files side by side in a single directory. It has no partitioning columns.

When a query filters on a partitioning column, Deephaven can skip the subdirectories the filter excludes.

Write to a key-value partitioned Parquet directory

A key-value partitioned directory stores each partition in a key=value subdirectory, as described in Partitioned Parquet directories. Writing the grades table with Class as the partitioning column produces the subdirectories Class=Math, Class=Science, and Class=History.

You can write a key-value partitioned directory from a regular Deephaven table or from a partitioned table. When you write a partitioned table, its key columns become the partitioning columns.

Define a table definition

A regular table needs a table definition, which is the table's schema with the partitioning columns marked. A regular table's own definition usually has no partitioning columns, so the write fails unless you pass a definition that has them. Pass it to ParquetInstructions.Builder.setTableDefinition.

Build a table definition with TableDefinition.of, passing one ColumnDefinition per column. Factory methods such as ColumnDefinition.ofString take the column name. To mark a column as a partitioning column, call its withPartitioning method.

Create a table definition for the grades table defined above.

To local storage

ParquetTools.writeKeyValuePartitionedTable takes three arguments:

• sourceTable or partitionedTable: The table to write, either a regular table or a partitioned table.
• destinationDir: The root directory for the partitioned Parquet data. Deephaven creates any missing directories in the path.
• writeInstructions: A ParquetInstructions object.

Call setGenerateMetadataFiles(true) on the builder to write _metadata and _common_metadata files at the root of the directory, as described in Write to a single Parquet file.

To S3

Use ParquetTools.writeKeyValuePartitionedTable to write key-value partitioned Parquet directories to S3. The destinationDir should be the URI of the destination directory in S3. Pass the s3Instructions defined in Write to a single Parquet file to the setSpecialInstructions method of ParquetInstructions.Builder. A gs:// URI writes to Google Cloud Storage with the same instructions.

Write to a flat partitioned Parquet directory

A flat partitioned directory holds one Parquet file per table, as described in Partitioned Parquet directories.

To local storage

Use ParquetTools.writeTable or ParquetTools.writeTables to write Deephaven tables to Parquet files in flat partitioned directories.

Call ParquetTools.writeTable once per table, giving each file its own path in the same directory, as in Write to a single Parquet file.

Use ParquetTools.writeTables to accomplish the same thing by passing multiple tables to the sources argument and multiple destination paths to the destinations argument. If the tables have different definitions, also call setTableDefinition on the ParquetInstructions.Builder.

To write a Deephaven partitioned table to a flat partitioned Parquet directory, look up each constituent table by key with constituentFor, so that each file name matches its class, and pass the tables to ParquetTools.writeTables.

To S3

Use ParquetTools.writeTables to write an array of Deephaven tables to a flat partitioned Parquet directory in S3. The destinations should be the URIs of the destination files in S3. Pass the s3Instructions defined in Write to a single Parquet file to the setSpecialInstructions method of ParquetInstructions.Builder. A gs:// URI writes to Google Cloud Storage with the same instructions.

Related documentation

• Import Parquet files
• Parquet formats
• Parquet instructions