Parquet cheat sheet
This cheat sheet shows common ways to read and write Parquet files. The examples use these methods:
The examples read and write files under /data, the data directory in Deephaven's Docker images. To map it to a directory on your computer, see Docker data volumes. In other deployments, use any path the server can read and write.
Create a table
The examples on this page write and read this sample table.
Write Parquet files
This block writes source to a single file twice, once with the default codec and once with GZIP, and then writes it as a key-value partitioned directory.
Compression codecs
To set the codec, pass a ParquetTools constant as the ParquetInstructions argument to writeTable. The example above passes ParquetTools.GZIP. The constants that work in a default installation are listed below. ParquetTools.BROTLI also exists, but it needs a Brotli codec added to the server classpath. See Parquet instructions.
SNAPPY(default): A codec based on Google's Snappy compression format that aims for high speed and reasonable compression.UNCOMPRESSED: No compression.LZ4_RAW: A codec based on the LZ4 block format.LZ4: A deprecated codec. UseLZ4_RAWinstead.LZO: A codec based on or interoperable with the LZO compression library.GZIP: A codec based on the GZIP format defined by RFC 1952. It differs from the related zlib and deflate formats.ZSTD: A codec based on the Zstandard format defined by RFC 8478 that aims for a high compression ratio.
Read Parquet files
The pems example reads the PEMS traffic dataset at /data/examples/Pems/parquet/pems from Deephaven's example data. To download it, see Find example data.
readTable detects the codec from the file, so reading /data/output_GZIP.parquet needs no codec setting.
How readTable handles a directory
When you pass a directory to readTable, it finds the data files, the schema, and the partitioning columns in one of two ways, depending on whether the directory has a _metadata file.
With a _metadata file:
readTabletakes the data files and the schema from_metadata.- The partitioning columns are the columns that the
_common_metadatafile in the same directory lists and_metadatadoes not. Without_common_metadata, thekey=valuedirectories do not become columns unless theParquetInstructionsset a table definition withsetTableDefinition. - The PEMS directory has both files, but its
_common_metadatalists no columns beyond those in_metadata, so itsstation=...directories do not become a column.
Without a _metadata file:
readTableloads every.parquetfile under the directory and skips names that start with..- It takes the schema from one of those files, so all the files should share a schema.
- Each
key=valuesubdirectory becomes a partitioning column, and every subdirectory that holds Parquet files must use akey=valuename. Reading/data/partitioned_outputadds the partitioning columnXfrom theX=A/,X=B/, andX=C/directories thatwriteKeyValuePartitionedTablecreated.
Read a directory that receives new files
Pass ParquetInstructions built with setIsRefreshing(true) to readTable to get a refreshing table that picks up new Parquet files added to a directory and adds their rows. Add data as new files rather than changing files that have already been read. Refreshing reads don't work on a single file or with the metadata-file layout, which readTable uses when the directory has a _metadata file and no file layout is set. To read such a directory as a refreshing table, also call setFileLayout on the builder with ParquetInstructions.ParquetFileLayout.KV_PARTITIONED or FLAT_PARTITIONED. See Parquet instructions for the other reading options.
Read legacy string columns
Some Parquet writers store string columns as byte-array columns (BINARY or FIXED_LEN_BYTE_ARRAY) with no logical type annotation. To read those columns as strings instead of byte arrays, pass ParquetTools.LEGACY to readTable, as in readTable(path, ParquetTools.LEGACY).
More options
The how-to guides cover more options: reading from S3 and writing to S3, setting the file layout and table definition, and generating metadata files.