Parquet cheat sheet
This cheat sheet shows common ways to read and write Parquet files. The examples use these functions from the deephaven.parquet module:
The examples read and write files under /data, the data directory in Deephaven's Docker images. To map it to a directory on your computer, see Docker bind mounts. In other deployments, use any path the server can read and write.
Create a table
The examples on this page write and read this sample table.
Write Parquet files
This block writes source to a single file twice, once with the default codec and once with GZIP, and then writes it as a key-value partitioned directory.
Compression codecs
Pass one of these values as the compression_codec_name argument to write or write_partitioned. They all work in a default installation. BROTLI also exists, but it needs a Brotli codec added to the server classpath. See Write Parquet files.
SNAPPY(default): A codec based on Google's Snappy compression format that aims for high speed and reasonable compression.UNCOMPRESSED: No compression.LZ4_RAW: A codec based on the LZ4 block format.LZ4: A deprecated codec. UseLZ4_RAWinstead.LZO: A codec based on or interoperable with the LZO compression library.GZIP: A codec based on the GZIP format defined by RFC 1952. It differs from the related zlib and deflate formats.ZSTD: A codec based on the Zstandard format defined by RFC 8478 that aims for a high compression ratio.
Read Parquet files
The pems example reads the PEMS traffic dataset at /data/examples/Pems/parquet/pems from Deephaven's example data. To download it, see Find example data.
read detects the codec from the file, so reading /data/output_GZIP.parquet needs no codec argument.
How read handles a directory
When you pass a directory to read, it finds the data files, the schema, and the partitioning columns in one of two ways, depending on whether the directory has a _metadata file.
With a _metadata file:
readtakes the data files and the schema from_metadata.- The partitioning columns are the columns that the
_common_metadatafile in the same directory lists and_metadatadoes not. Without_common_metadata, thekey=valuedirectories do not become columns unless you pass atable_definition. - The PEMS directory has both files, but its
_common_metadatalists no columns beyond those in_metadata, so itsstation=...directories do not become a column.
Without a _metadata file:
readloads every.parquetfile under the directory and skips names that start with..- It takes the schema from one of those files, so all the files should share a schema.
- Each
key=valuesubdirectory becomes a partitioning column, and every subdirectory that holds Parquet files must use akey=valuename. Reading/data/partitioned_outputadds the partitioning columnXfrom theX=A/,X=B/, andX=C/directories thatwrite_partitionedcreated.
Read a directory that receives new files
Pass is_refreshing=True to read to get a refreshing table that picks up new Parquet files added to a directory and adds their rows. Add data as new files rather than changing files that have already been read. Refreshing reads don't work on a single file or with the metadata-file layout, which read uses when the directory has a _metadata file and no file_layout is set. To read such a directory as a refreshing table, pass file_layout set to ParquetFileLayout.KV_PARTITIONED or ParquetFileLayout.FLAT_PARTITIONED. See Refreshing tables for an example that adds files.
Read legacy string columns
Some Parquet writers store string columns as byte-array columns (BINARY or FIXED_LEN_BYTE_ARRAY) with no logical type annotation. To read those columns as strings instead of byte arrays, pass is_legacy_parquet=True to read, as in parquet.read(path, is_legacy_parquet=True).
More options
The how-to guides cover more options: reading from S3 and writing to S3, setting the file_layout and table_definition arguments, and generating metadata files.