Read Parquet files into Deephaven tables

Use parquet.read from the Parquet Python module to load Parquet files into tables. This document covers reading single Parquet files, key-value partitioned directories, and flat partitioned directories, each from local storage or from S3.

Read a single Parquet file

Read a single Parquet file when one file holds all of the table's data.

From local storage

Read single Parquet files into Deephaven tables with parquet.read. The function takes a single required argument path, which gives the full file path of the Parquet file.

From S3

The deephaven.experimental.s3 Python module supports reading from S3. The module is experimental, so its API is subject to change. It contains the S3Instructions class, which holds the settings Deephaven uses to connect to the S3 instance. Learn more about this class in the special instructions section of this document.

Use parquet.read to read a single Parquet file from S3, where the path argument is the S3 URI of the Parquet file (for example, s3://bucket/key.parquet). Supply an instance of the S3Instructions class to the special_instructions argument to specify the details of the connection to the S3 instance.

Note

When reading from S3, run the Deephaven instance in the same AWS region as the S3 bucket for the best performance. To improve performance further, store the data in a directory bucket in a single AWS Availability Zone, and run the Deephaven instance in that same Availability Zone. For more information, see the AWS article on S3 Express One Zone directory buckets.

Read partitioned Parquet directories

A partitioned Parquet directory spreads the data for one table across many Parquet files. Deephaven reads two kinds of partitioned directory:

  • A key-value partitioned directory names each subdirectory after a partition column and its value, such as Year=2024/.
  • A flat partitioned directory keeps all of its Parquet files in a single directory, with no subdirectories and no partition columns.

Either kind can also contain the Parquet metadata files _metadata and _common_metadata, which describe the whole dataset. A directory that contains a _metadata file is read as a third layout, metadata-partitioned, which avoids opening each Parquet file to read its metadata. Deephaven chooses this layout automatically when it infers the layout, or you can set file_layout to METADATA_PARTITIONED. Either way, _metadata is required. _common_metadata is optional, and it is what lets Deephaven infer the partitioning columns of a key-value dataset.

When Deephaven reads a partitioned Parquet directory, it returns a single table that contains the data from every file. For a key-value partitioned directory, each partition column appears as a column in the result. To work with each partition as a separate table, call partition_by on the result. See the guide on partitioned tables for more information.

Read a key-value partitioned Parquet directory

A key-value partitioned directory stores each partition in a column=value subdirectory, such as Year=2024/. Compared with a flat partitioned directory, it lets filters on partition columns skip reading the files in non-matching subdirectories, at the cost of a deeper directory tree to maintain.

From local storage

Use parquet.read to read a key-value partitioned Parquet directory into a Deephaven table. parquet.read can infer the directory layout. Alternatively, set the file_layout argument to parquet.ParquetFileLayout.KV_PARTITIONED. Providing the layout skips only the check for a _metadata file. Deephaven still infers the partition columns and schema from the files unless you also pass a schema in the table_definition argument.

To read through the metadata files, set the file_layout argument to parquet.ParquetFileLayout.METADATA_PARTITIONED.

From S3

Use parquet.read to read a key-value partitioned Parquet directory from S3. Supply the special_instructions argument with an instance of the S3Instructions class. To skip the check for a _metadata file, set the file_layout argument to parquet.ParquetFileLayout.KV_PARTITIONED. For performance, run Deephaven in the same AWS region as the bucket, as described in Read a single Parquet file from S3. Reading from S3 differs from reading locally only in the S3 path and the special_instructions argument; the layout arguments work the same way.

To read an S3-hosted dataset through its metadata files, set the file_layout argument to parquet.ParquetFileLayout.METADATA_PARTITIONED.

Read a flat partitioned Parquet directory

A flat partitioned Parquet directory is a single directory of Parquet files with no partition subdirectories. Deephaven reads the .parquet files in the directory into one table. For a local directory, it skips hidden files whose names start with .. Because a flat directory has no partition columns, filters can't skip its files by partition value the way they can with a key-value partitioned directory.

From local storage

Read local flat partitioned Parquet directories into Deephaven tables with parquet.read, which can infer the directory layout. To skip the check for a _metadata file, set the file_layout argument to parquet.ParquetFileLayout.FLAT_PARTITIONED.

To read through the metadata files, set the file_layout argument to parquet.ParquetFileLayout.METADATA_PARTITIONED.

From S3

Use parquet.read to read a flat partitioned Parquet directory from S3. Supply the special_instructions argument with an S3Instructions instance. To skip the check for a _metadata file, set the file_layout argument to parquet.ParquetFileLayout.FLAT_PARTITIONED. For performance, run Deephaven in the same AWS region as the bucket, as described in Read a single Parquet file from S3.

Reading a flat directory from S3 differs from reading it locally only in the S3 path and the special_instructions argument. To read a flat S3 dataset through its metadata files, set file_layout to parquet.ParquetFileLayout.METADATA_PARTITIONED and follow the key-value S3 metadata example above.

Arguments

parquet.read accepts the following arguments. The examples above use only some of them.

  • path (required): The Parquet file, metadata file (_metadata or _common_metadata, with the _metadata file in the same directory), or directory to read. This is typically a string containing a local path, an S3 URI, or a Google Cloud Storage (gs://) URI.
  • col_instructions: Per-column read settings, such as the Parquet column name to read into each Deephaven column, provided as a list of ColumnInstruction objects. The default is None, which means no specialization for any column.
  • is_legacy_parquet: True or False indicating if the Parquet data is in legacy Parquet format. When True, Deephaven reads binary columns that have no logical type annotation and no recorded codec as strings rather than as byte arrays. Some older Parquet writers store strings this way. The default is False.
  • is_refreshing: True or False indicating if the Parquet data represents a refreshing source. When True, Deephaven checks a partitioned directory for new Parquet files and adds their rows to the table, so the result is a refreshing table. Refreshing reads aren't supported for a single Parquet file or for a directory read through its metadata files. The default is False.
  • file_layout: The Parquet file or directory layout, provided as a ParquetFileLayout: SINGLE_FILE, FLAT_PARTITIONED, KV_PARTITIONED, or METADATA_PARTITIONED. See File layouts. The default is None, which means Deephaven infers the layout.
  • table_definition: The table definition or schema, provided as a TableDefinition, a dictionary of string-DType pairs, or a list of ColumnDefinition instances. When not provided, Deephaven infers the definition from the Parquet file(s).
  • special_instructions: Special instructions for reading Parquet files from S3, an S3-compatible store, or Google Cloud Storage (gs:// URIs), provided as an instance of S3Instructions. The default is None.

Column instructions

The col_instructions argument to parquet.read takes a list of ColumnInstruction objects. Each ColumnInstruction maps a column in the Parquet data to a column in the Deephaven table.

ColumnInstruction has the following arguments:

  • column_name: The Deephaven column name. Required when reading and when writing.
  • parquet_column_name: The name of the corresponding column in the Parquet dataset. Required when reading.
  • codec_name: The fully qualified name of an ObjectCodec class that serializes the column's values to and from bytes, such as io.deephaven.util.codec.LocalDateCodec. Use a codec for types that have no language-agnostic Parquet representation. This is not a compression codec.
  • codec_args: An implementation-specific argument string passed to the codec named by codec_name.
  • unsigned_long_target: The Deephaven type to read an unsigned 64-bit integer (UINT_64) column as, provided as a parquet.UnsignedLongTarget. The default is None, which reads such columns as BigInteger. It applies only when Deephaven infers the table definition.

The use_dictionary argument applies only when writing. See Parquet export.

Special instructions (S3 and GCS)

The special_instructions argument to parquet.read is relevant when reading from S3, an S3-compatible store, or Google Cloud Storage (gs:// URIs), and takes an instance of the S3Instructions class. This class specifies details for connecting to the store.

Deephaven reads an S3 object in fragments, which are byte ranges of the file that it fetches with separate requests. Several of the arguments below tune how Deephaven fetches fragments.

S3Instructions has the following arguments for reading:

  • region_name: The AWS region of the S3 bucket that holds the Parquet data. When this is not set, the AWS SDK looks for a region in the following places, in order:

    • The aws.region system property
    • The AWS_REGION environment variable
    • The {user.home}/.aws/credentials and {user.home}/.aws/config files
    • The EC2 metadata service, when Deephaven runs in EC2

    If none of these gives a region, or the region is wrong for the bucket, Deephaven finds the correct region itself. This costs one extra request.

  • credentials: A Credentials object for authenticating to the S3 instance. The default is None, which uses Credentials.resolving.

  • endpoint_override: The endpoint to connect to. Set it when connecting to a non-AWS, S3-compatible API. Connections to AWS typically don't need it. The default is None.

  • read_ahead_count: The number of fragments that Deephaven reads asynchronously ahead of the fragment it is currently reading. The default is 32.

  • fragment_size: The maximum size of each fragment to read in bytes. The default is 65536 bytes (64 KiB).

  • read_timeout: The time to wait for a fragment read to complete before timing out. The default is 2 seconds.

  • max_concurrent_requests: The maximum number of concurrent requests to make to S3. The default is 256.

  • connection_timeout: The time to wait for a successful S3 connection before timing out. The default is 2 seconds.

  • profile_name: The AWS profile name used to configure the default region, credentials, and so on.

  • config_file_path: The path to the AWS configuration file.

  • credentials_file_path: The path to the AWS credentials file.

The write_timeout, write_part_size, and num_concurrent_write_parts arguments apply only when writing to S3. See the S3Instructions pydoc for details.

The access_key_id, secret_access_key, and anonymous_access arguments are deprecated. Use Credentials.basic(access_key_id, secret_access_key) or Credentials.anonymous for the credentials argument instead.