ColumnInstruction
A ColumnInstruction specifies the instructions for reading or writing a Parquet column.
Syntax
Parameters
| Parameter | Type | Description |
|---|
| column_name optional | str | The name of the column to apply these instructions to. |
|---|
| parquet_column_name optional | str | The name of the column in the resulting Parquet file. |
|---|
| codec_name optional | str | The compression codec to use. Options are:
SNAPPY: (default) Aims for high speed and a reasonable amount of compression. Based on Google's Snappy compression format.UNCOMPRESSED: The output will not be compressed.LZ4_RAW: A codec based on the LZ4 block format. Should always be used instead of LZ4.LZ4: Deprecated Compression codec loosely based on the LZ4 compression algorithm, but with an additional undocumented framing scheme. The framing is part of the original Hadoop compression library and was historically copied first in parquet-mr, then emulated with mixed results by parquet-cpp. Note that LZ4 is deprecated; use LZ4_RAW instead.LZO: Compression codec based on or interoperable with the LZO compression library.GZIP: Compression codec based on the GZIP format (not the closely-related "zlib" or "deflate" formats) defined by RFC 1952.ZSTD: Compression codec with the highest compression ratio based on the Zstandard format defined by RFC 8478. |
|---|
| codec_args optional | str | An implementation-specific string used to map types to/from bytes. Typically used in cases where there is no obvious language-agnostic representation in Parquet. Default is None. |
|---|
| use_dictionary optional | bool | Whether or not to use dictionary-based encoding for string columns. |
|---|
| unsigned_long_target optional | UnsignedLongTarget | The Deephaven type to read an unsigned 64-bit integer (UINT_64) column as. This parameter applies to reads only, and only to columns that carry the UINT_64 logical type. It is ignored when writing, because Deephaven never writes UINT_64. Default is None, which is equivalent to UnsignedLongTarget.BIG_INTEGER. Options are:
UnsignedLongTarget.BIG_INTEGER: (default) Read the column as java.math.BigInteger, which represents every UINT_64 value exactly.UnsignedLongTarget.LONG: Read the column as long. Values greater than 263 - 1 have no long representation, so reading a page that contains one raises an error.UnsignedLongTarget.SIGNED_LONG: Read the column as long, reinterpreting the bit pattern as signed. Values greater than 263 - 1 read as negative numbers, and 263 reads as NULL_LONG, which is indistinguishable from a null. For an explanation of how Deephaven maps UINT_64 and other logical types, see Parquet formats. |
|---|
Returns
A ColumnInstruction object that will give Deephaven instructions for handling a particular column.
Examples
In this example, we create a ColumnInstruction that can be passed into read or write.
In this example, unsigned_long_target reads an unsigned 64-bit integer column as a long instead of the default java.math.BigInteger. The example requires a Parquet file with a UINT_64 column, which Deephaven does not write.