Read Parquet files into Deephaven tables

Use ParquetTools.readTable from the ParquetTools class to load Parquet files into tables. This document covers reading single Parquet files, key-value partitioned directories, and flat partitioned directories, each from local storage or from S3.

Read a single Parquet file

Read a single Parquet file when one file holds all of the table's data.

From local storage

Read single Parquet files into Deephaven tables with ParquetTools.readTable. The method takes a single required argument, source, which gives the full file path of the Parquet file.

From S3

The io.deephaven.extensions.s3 package supports reading from S3. This package contains the S3Instructions class, which holds the settings Deephaven uses to connect to the S3 instance. Learn more about this class in the Parquet instructions document.

Use ParquetTools.readTable to read a single Parquet file from S3, where the source argument is the S3 URI of the Parquet file (for example, s3://bucket/key.parquet). Pass connection details in the optional second argument, readInstructions, which takes a ParquetInstructions object. To build it, call ParquetInstructions.builder, pass an S3Instructions instance to the builder's setSpecialInstructions method, and call build. Optional arguments lists the other builder methods that apply when reading.

Note

When reading from S3, run the Deephaven instance in the same AWS region as the S3 bucket for the best performance. To improve performance further, store the data in a directory bucket in a single AWS Availability Zone, and run the Deephaven instance in that same Availability Zone. For more information, see the AWS article on S3 Express One Zone directory buckets.

Read partitioned Parquet directories

A partitioned Parquet directory spreads the data for one table across many Parquet files. Deephaven reads two kinds of partitioned directory:

  • A key-value partitioned directory names each subdirectory after a partition column and its value, such as Year=2024/.
  • A flat partitioned directory keeps all of its Parquet files in a single directory, with no subdirectories and no partition columns.

Either kind can also contain the Parquet metadata files _metadata and _common_metadata, which describe the whole dataset. A directory that contains a _metadata file is read as a third layout, metadata-partitioned, which avoids opening each Parquet file to read its metadata. Deephaven chooses this layout automatically when it infers the layout, or you can pass ParquetFileLayout.METADATA_PARTITIONED to setFileLayout. Either way, _metadata is required. _common_metadata is optional, and it is what lets Deephaven infer the partitioning columns of a key-value dataset.

When Deephaven reads a partitioned Parquet directory, it returns a single table that contains the data from every file. For a key-value partitioned directory, each partition column appears as a column in the result. To work with each partition as a separate table, call partitionBy on the result. See the guide on partitioned tables for more information.

Read a key-value partitioned Parquet directory

A key-value partitioned directory stores each partition in a column=value subdirectory, such as Year=2024/. Compared with a flat partitioned directory, it lets filters on partition columns skip reading the files in non-matching subdirectories, at the cost of a deeper directory tree to maintain.

From local storage

Use ParquetTools.readTable to read a key-value partitioned Parquet directory into a Deephaven table. ParquetTools.readTable can infer the directory layout. Alternatively, pass ParquetFileLayout.KV_PARTITIONED to the setFileLayout builder method. Then pass the resulting ParquetInstructions as the readInstructions argument. Providing the layout skips only the check for a _metadata file. Deephaven still infers the partition columns and schema from the files unless you also pass a schema to setTableDefinition.

To read through the metadata files, pass ParquetFileLayout.METADATA_PARTITIONED to the setFileLayout method.

From S3

Use ParquetTools.readTable to read a key-value partitioned Parquet directory from S3. Pass an instance of the S3Instructions class to the setSpecialInstructions method. To skip the check for a _metadata file, pass ParquetFileLayout.KV_PARTITIONED to the setFileLayout method. Pass the built ParquetInstructions as the readInstructions argument. For performance, run Deephaven in the same AWS region as the bucket, as described in Read a single Parquet file from S3. Reading from S3 differs from reading locally only in the S3 path and the S3Instructions passed through setSpecialInstructions; the layout arguments work the same way.

To read an S3-hosted dataset through its metadata files, pass ParquetFileLayout.METADATA_PARTITIONED to the setFileLayout method.

Read a flat partitioned Parquet directory

A flat partitioned Parquet directory is a single directory of Parquet files with no partition subdirectories. Deephaven reads the .parquet files in the directory into one table. For a local directory, it skips hidden files whose names start with .. Because a flat directory has no partition columns, filters can't skip its files by partition value the way they can with a key-value partitioned directory.

From local storage

Read local flat partitioned Parquet directories into Deephaven tables with ParquetTools.readTable, which can infer the directory layout. To skip the check for a _metadata file, pass ParquetFileLayout.FLAT_PARTITIONED to the setFileLayout method and pass the resulting ParquetInstructions as the readInstructions argument.

To read through the metadata files, pass ParquetFileLayout.METADATA_PARTITIONED to the setFileLayout method.

From S3

Use ParquetTools.readTable to read a flat partitioned Parquet directory from S3. Pass an instance of the S3Instructions class to the setSpecialInstructions method. To skip the check for a _metadata file, pass ParquetFileLayout.FLAT_PARTITIONED to the setFileLayout method. Pass the built ParquetInstructions as the readInstructions argument. For performance, run Deephaven in the same AWS region as the bucket, as described in Read a single Parquet file from S3.

Reading a flat directory from S3 differs from reading it locally only in the S3 path and the S3Instructions passed through setSpecialInstructions. To read a flat S3 dataset through its metadata files, pass ParquetFileLayout.METADATA_PARTITIONED to setFileLayout and follow the key-value S3 metadata example above.

Optional arguments

The required first argument to ParquetTools.readTable, source, is the Parquet file, metadata file (_metadata or _common_metadata, with the _metadata file in the same directory), or directory to read. This is typically a string containing a local path, an S3 URI, or a Google Cloud Storage (gs://) URI.

The method also takes an optional second argument, readInstructions, which is a ParquetInstructions object built with ParquetInstructions.builder. The ParquetInstructions.Builder methods that apply when reading include:

  • setFileLayout: The Parquet file or directory layout, provided as a ParquetFileLayout: SINGLE_FILE, FLAT_PARTITIONED, KV_PARTITIONED, or METADATA_PARTITIONED. See File layouts. If you don't call this method, Deephaven infers the layout.
  • setTableDefinition: The table definition or schema. If you don't call this method, Deephaven infers the definition from the Parquet file(s).
  • setSpecialInstructions: Special instructions for reading Parquet files from S3, an S3-compatible store, or Google Cloud Storage (gs:// URIs), provided as an instance of S3Instructions.
  • setIsRefreshing: Whether the Parquet data represents a refreshing source. When true, Deephaven checks a partitioned directory for new Parquet files and adds their rows to the table, so the result is a refreshing table. Refreshing reads aren't supported for a single Parquet file or for a directory read through its metadata files. The default is false.
  • setIsLegacyParquet: Whether the Parquet data is in legacy Parquet format. When true, Deephaven reads binary columns that have no logical type annotation and no recorded codec as strings rather than as byte arrays. Some older Parquet writers store strings this way. The default is false.
  • addColumnNameMapping: Maps a column name in the Parquet data to a column name in the Deephaven table.
  • addColumnCodec: Sets the ObjectCodec class that converts a column's values to and from bytes, for types with no language-agnostic Parquet representation. This is not a compression codec.
  • setUnsignedLongTarget: Sets the Deephaven type to read an unsigned 64-bit integer (UINT_64) column as, provided as an UnsignedLongTarget. If you don't call this method, Deephaven reads such columns as BigInteger. It applies only when Deephaven infers the table definition.

See the Parquet instructions guide for the ParquetInstructions.Builder and S3Instructions methods most users need. The Javadoc is the complete API reference.