Parquet cheat sheet

This cheat sheet shows common ways to read and write Parquet files. The examples use these methods:

The examples read and write files under /data, the data directory in Deephaven's Docker images. To map it to a directory on your computer, see Docker data volumes. In other deployments, use any path the server can read and write.

Create a table

The examples on this page write and read this sample table.

Write Parquet files

This block writes source to a single file twice, once with the default codec and once with GZIP, and then writes it as a key-value partitioned directory.

Compression codecs

To set the codec, pass a ParquetTools constant as the ParquetInstructions argument to writeTable. The example above passes ParquetTools.GZIP. The constants that work in a default installation are listed below. ParquetTools.BROTLI also exists, but it needs a Brotli codec added to the server classpath. See Parquet instructions.

  • SNAPPY (default): A codec based on Google's Snappy compression format that aims for high speed and reasonable compression.
  • UNCOMPRESSED: No compression.
  • LZ4_RAW: A codec based on the LZ4 block format.
  • LZ4: A deprecated codec. Use LZ4_RAW instead.
  • LZO: A codec based on or interoperable with the LZO compression library.
  • GZIP: A codec based on the GZIP format defined by RFC 1952. It differs from the related zlib and deflate formats.
  • ZSTD: A codec based on the Zstandard format defined by RFC 8478 that aims for a high compression ratio.

Read Parquet files

The pems example reads the PEMS traffic dataset at /data/examples/Pems/parquet/pems from Deephaven's example data. To download it, see Find example data.

readTable detects the codec from the file, so reading /data/output_GZIP.parquet needs no codec setting.

How readTable handles a directory

When you pass a directory to readTable, it finds the data files, the schema, and the partitioning columns in one of two ways, depending on whether the directory has a _metadata file.

With a _metadata file:

  • readTable takes the data files and the schema from _metadata.
  • The partitioning columns are the columns that the _common_metadata file in the same directory lists and _metadata does not. Without _common_metadata, the key=value directories do not become columns unless the ParquetInstructions set a table definition with setTableDefinition.
  • The PEMS directory has both files, but its _common_metadata lists no columns beyond those in _metadata, so its station=... directories do not become a column.

Without a _metadata file:

  • readTable loads every .parquet file under the directory and skips names that start with ..
  • It takes the schema from one of those files, so all the files should share a schema.
  • Each key=value subdirectory becomes a partitioning column, and every subdirectory that holds Parquet files must use a key=value name. Reading /data/partitioned_output adds the partitioning column X from the X=A/, X=B/, and X=C/ directories that writeKeyValuePartitionedTable created.

Read a directory that receives new files

Pass ParquetInstructions built with setIsRefreshing(true) to readTable to get a refreshing table that picks up new Parquet files added to a directory and adds their rows. Add data as new files rather than changing files that have already been read. Refreshing reads don't work on a single file or with the metadata-file layout, which readTable uses when the directory has a _metadata file and no file layout is set. To read such a directory as a refreshing table, also call setFileLayout on the builder with ParquetInstructions.ParquetFileLayout.KV_PARTITIONED or FLAT_PARTITIONED. See Parquet instructions for the other reading options.

Read legacy string columns

Some Parquet writers store string columns as byte-array columns (BINARY or FIXED_LEN_BYTE_ARRAY) with no logical type annotation. To read those columns as strings instead of byte arrays, pass ParquetTools.LEGACY to readTable, as in readTable(path, ParquetTools.LEGACY).

More options

The how-to guides cover more options: reading from S3 and writing to S3, setting the file layout and table definition, and generating metadata files.