Supported Parquet formats

Apache Parquet is an open-source, column-oriented data file format for efficient data storage and retrieval. Parquet is designed for complex, large-scale data and supports several compression codecs.

Parquet files are composed of row groups, a header, and a footer. Each row group contains data from every column, stored in a columnar format. This column-oriented structure optimizes performance and minimizes I/O.

Parquet is a standard interchange format for batch and interactive workloads. Deephaven reads most standard Parquet files without extra plugins or configuration. This page describes the file layouts Deephaven can read, the Parquet logical types whose Deephaven column types may be unexpected, and the column types Deephaven can't read.

File layouts

Deephaven reads four Parquet file layouts, one for each value of ParquetFileLayout. When you call read without a file_layout argument, Deephaven infers the layout from the path:

• A path ending in .parquet is read as a single file.
• A path whose last component is _metadata or _common_metadata is read as metadata-partitioned. The _metadata file must exist in the same directory.
• Any other path is treated as a directory. Deephaven reads it as metadata-partitioned if it contains a _metadata file, and as key-value partitioned otherwise. Deephaven also reads flat directories this way. A single Parquet file must have a name ending in .parquet, even when you set the single-file layout explicitly.

To read a layout explicitly, pass its value as the file_layout argument to read:

Single Parquet file

Deephaven supports single Parquet files. Using a single large Parquet file may be more storage efficient than many smaller files with accompanying metadata files. It can be faster to read and process because there is less overhead in opening and closing files.

Deephaven doesn't support refreshing reads of a single file, so is_refreshing=True fails.

Flat partitioned directories

A directory can contain multiple Parquet files, and Deephaven can load them as sections of a single table. In a flat partitioned directory, the Parquet files sit directly in one directory. A flat directory has no key=value subdirectories, so the table has no partitioning columns.

When you write data, a flat layout may be useful if:

• You want to control the size of each file.
• You write one file at a time, for example, once per hour.
• Several systems write files to the same directory.

Key-value partitioned directories

In a key-value partitioned directory, Parquet files are hierarchically partitioned into nested directories named key=value. All of the Parquet files sit at the same nesting level.

Deephaven takes each partitioning column's name from the directory keys and infers its type from the directory values. Write data in this layout when it is partitioned hierarchically and you don't write metadata files that record the partitioning column types.

If Deephaven cannot infer a more specific type for a partition value, the column becomes a String column. To choose the type yourself, pass a table definition with the table_definition argument to read.

See Read a key-value partitioned Parquet directory for examples.

Metadata-partitioned directories

A metadata-partitioned directory is a Parquet dataset with a _metadata file, and optionally a _common_metadata file, at its root:

• _metadata describes every row group in every Parquet file in the dataset. Deephaven reads it to find all of the data files at once, without listing the directory tree.
• _common_metadata supplies the full schema, including the partitioning columns, which _metadata does not describe. If a dataset has _metadata but no _common_metadata, Deephaven reads it without partitioning columns unless you pass a table definition with the table_definition argument to read.

Write these files by passing generate_metadata_files=True to write or write_partitioned, as shown in Export Parquet files.

Reading a metadata-partitioned directory with is_refreshing=True fails.

Unexpected logical type mappings

Deephaven maps Parquet logical types to Deephaven column types on read. Some of these mappings are ones you might not expect:

• ENUM: Read as String. Some Parquet writers use ENUM to mark columns that hold a finite set of string values. Physically, an ENUM column is stored the same way as a STRING column, as UTF-8 bytes in a BINARY column.
• UINT_8 and UINT_16: Read as char, Java's unsigned 16-bit type.
• UINT_32: Read as long.
• UINT_64: Read as java.math.BigInteger by default, because no Java primitive holds the full UINT_64 range. To read it as a long instead, see Reading UINT_64 as a long.

Reading UINT_64 as a long

BigInteger represents every UINT_64 value exactly but allocates an object per value. To read the column as a primitive long instead, pass read a ColumnInstruction for the column.

When reading, every ColumnInstruction must set both parquet_column_name and column_name. Omitting either raises an error.

Set unsigned_long_target to one of two UnsignedLongTarget options:

• UnsignedLongTarget.LONG reads values that fit in a long and raises an error when Deephaven reads a page that contains a value greater than 2⁶³ - 1. The call to read itself succeeds; the error comes up when that data is accessed. Use this option when you know the data stays within the long range and an unnoticed overflow is unacceptable.
• UnsignedLongTarget.SIGNED_LONG reinterprets the bit pattern as signed, so values greater than 2⁶³ - 1 read as negative numbers. This option never fails. The value 2⁶³ reads as NULL_LONG. Deephaven reserves that long value for null, so a stored 2⁶³ is indistinguishable from a null.

The following example reads a UINT_64 column as a long, rejecting any value that does not fit:

Deephaven never writes UINT_64, so these options apply only to reads, and only when Deephaven infers the table definition. A table definition you supply governs the column type instead, and Deephaven rejects one that disagrees with the target. For how Arrow Flight handles unsigned 64-bit integers, see the Arrow type support matrix.

Unsupported column types

Deephaven can't infer a column type for the following Parquet columns:

• JSON, BSON, UUID, INTERVAL, FLOAT16, GEOMETRY, GEOGRAPHY, VARIANT, and MAP columns
• Group columns, also called structs, that have more than one field
• Nested repeated columns, such as lists of lists

Reading a file that contains such a column fails with an error unless you pass read a table_definition that leaves that column out.

Related documentation

• Import Parquet into Deephaven video
• Import Parquet files
• Export Parquet files
• ColumnInstruction