Parquet cheat sheet

This cheat sheet shows common ways to read and write Parquet files. The examples use these functions from the deephaven.parquet module:

The examples read and write files under /data, the data directory in Deephaven's Docker images. To map it to a directory on your computer, see Docker bind mounts. In other deployments, use any path the server can read and write.

Create a table

The examples on this page write and read this sample table.

Write Parquet files

This block writes source to a single file twice, once with the default codec and once with GZIP, and then writes it as a key-value partitioned directory.

Compression codecs

Pass one of these values as the compression_codec_name argument to write or write_partitioned. They all work in a default installation. BROTLI also exists, but it needs a Brotli codec added to the server classpath. See Write Parquet files.

  • SNAPPY (default): A codec based on Google's Snappy compression format that aims for high speed and reasonable compression.
  • UNCOMPRESSED: No compression.
  • LZ4_RAW: A codec based on the LZ4 block format.
  • LZ4: A deprecated codec. Use LZ4_RAW instead.
  • LZO: A codec based on or interoperable with the LZO compression library.
  • GZIP: A codec based on the GZIP format defined by RFC 1952. It differs from the related zlib and deflate formats.
  • ZSTD: A codec based on the Zstandard format defined by RFC 8478 that aims for a high compression ratio.

Read Parquet files

The pems example reads the PEMS traffic dataset at /data/examples/Pems/parquet/pems from Deephaven's example data. To download it, see Find example data.

read detects the codec from the file, so reading /data/output_GZIP.parquet needs no codec argument.

How read handles a directory

When you pass a directory to read, it finds the data files, the schema, and the partitioning columns in one of two ways, depending on whether the directory has a _metadata file.

With a _metadata file:

  • read takes the data files and the schema from _metadata.
  • The partitioning columns are the columns that the _common_metadata file in the same directory lists and _metadata does not. Without _common_metadata, the key=value directories do not become columns unless you pass a table_definition.
  • The PEMS directory has both files, but its _common_metadata lists no columns beyond those in _metadata, so its station=... directories do not become a column.

Without a _metadata file:

  • read loads every .parquet file under the directory and skips names that start with ..
  • It takes the schema from one of those files, so all the files should share a schema.
  • Each key=value subdirectory becomes a partitioning column, and every subdirectory that holds Parquet files must use a key=value name. Reading /data/partitioned_output adds the partitioning column X from the X=A/, X=B/, and X=C/ directories that write_partitioned created.

Read a directory that receives new files

Pass is_refreshing=True to read to get a refreshing table that picks up new Parquet files added to a directory and adds their rows. Add data as new files rather than changing files that have already been read. Refreshing reads don't work on a single file or with the metadata-file layout, which read uses when the directory has a _metadata file and no file_layout is set. To read such a directory as a refreshing table, pass file_layout set to ParquetFileLayout.KV_PARTITIONED or ParquetFileLayout.FLAT_PARTITIONED. See Refreshing tables for an example that adds files.

Read legacy string columns

Some Parquet writers store string columns as byte-array columns (BINARY or FIXED_LEN_BYTE_ARRAY) with no logical type annotation. To read those columns as strings instead of byte arrays, pass is_legacy_parquet=True to read, as in parquet.read(path, is_legacy_parquet=True).

More options

The how-to guides cover more options: reading from S3 and writing to S3, setting the file_layout and table_definition arguments, and generating metadata files.