Package io.deephaven.parquet.table

Class ParquetTools

java.lang.Object

io.deephaven.parquet.table.ParquetTools

public class ParquetTools
extends Object

Tools for managing and manipulating tables on disk in parquet format.

Field Summary

Fields

Modifier and Type	Field	Description
static final ParquetInstructions	BROTLI
static final ParquetInstructions	GZIP
static final ParquetInstructions	LEGACY
static final ParquetInstructions	LZ4	Deprecated. Use LZ4_RAW instead, as explained here
static final ParquetInstructions	LZ4_RAW
static final ParquetInstructions	LZO
static final ParquetInstructions	SNAPPY
static final ParquetInstructions	UNCOMPRESSED
static final ParquetInstructions	ZSTD

Method Summary

Modifier and Type	Method	Description
static Pair<List<ColumnDefinition<?>>,ParquetInstructions>	convertSchema(@NotNull org.apache.parquet.schema.MessageType schema, @NotNull Map<String,String> keyValueMetadata, @NotNull ParquetInstructions readInstructionsIn)	Convert schema information from a ParquetMetadata into ColumnDefinitions.
static void	deleteTable(File path)	Deletes a table on disk.
static ParquetFileReader	getParquetFileReader(@NotNull File parquetFile, @NotNull ParquetInstructions readInstructions)	Make a ParquetFileReader for the supplied File.
static ParquetFileReader	getParquetFileReader(@NotNull URI parquetFileURI, @NotNull ParquetInstructions readInstructions)	Make a ParquetFileReader for the supplied URI.
static ParquetFileReader	getParquetFileReaderChecked(@NotNull File parquetFile, @NotNull ParquetInstructions readInstructions)	Make a ParquetFileReader for the supplied File.
static ParquetFileReader	getParquetFileReaderChecked(@NotNull URI parquetFileURI, @NotNull ParquetInstructions readInstructions)	Make a ParquetFileReader for the supplied URI.
static String	getRelativeIndexFilePath(@NotNull File tableDest, @NotNull String columnName)	Generates the index file path relative to the table destination file path.
static String	legacyGroupingFileName(@NotNull File tableDest, @NotNull String columnName)	Legacy method for generating a grouping file name.
static Table	readFlatPartitionedTable(@NotNull File directory, @NotNull ParquetInstructions readInstructions)	Creates a partitioned table via the flat parquet files from the root directory, inferring the table definition from those files.
static Table	readFlatPartitionedTable(@NotNull File directory, @NotNull ParquetInstructions readInstructions, @NotNull TableDefinition tableDefinition)	Creates a partitioned table via the flat parquet files from the root directory using the provided tableDefinition.
static Table	readKeyValuePartitionedTable(@NotNull File directory, @NotNull ParquetInstructions readInstructions)	Creates a partitioned table via the key-value partitioned parquet files from the root directory, inferring the table definition from those files.
static Table	readKeyValuePartitionedTable(@NotNull File directory, @NotNull ParquetInstructions readInstructions, @NotNull TableDefinition tableDefinition)	Creates a partitioned table via the key-value partitioned parquet files from the root directory using the provided tableDefinition.
static Table	readParquetSchemaAndTable(@NotNull File source, @NotNull ParquetInstructions readInstructionsIn, org.apache.commons.lang3.mutable.MutableObject<ParquetInstructions> instructionsOut)
static Table	readPartitionedTable(@NotNull TableLocationKeyFinder<ParquetTableLocationKey> locationKeyFinder, @NotNull ParquetInstructions readInstructions)	Reads in a table from files discovered with locationKeyFinder using a definition built from the highest (by location key order) location found, which must have non-null partition values for all partition keys.
static Table	readPartitionedTable(@NotNull TableLocationKeyFinder<ParquetTableLocationKey> locationKeyFinder, @NotNull ParquetInstructions readInstructions, @NotNull TableDefinition tableDefinition)	Reads in a table from files discovered with locationKeyFinder using the provided table definition.
static Table	readPartitionedTableInferSchema(@NotNull TableLocationKeyFinder<ParquetTableLocationKey> locationKeyFinder, @NotNull ParquetInstructions readInstructions)	Deprecated. use readPartitionedTable(TableLocationKeyFinder, ParquetInstructions)
static Table	readPartitionedTableWithMetadata(@NotNull File directory, @NotNull ParquetInstructions readInstructions)	Reads in a table using metadata files found in the supplied directory.
static Table	readSingleFileTable(@NotNull ParquetTableLocationKey tableLocationKey, @NotNull ParquetInstructions readInstructions, @NotNull TableDefinition tableDefinition)	Reads in a table from a single parquet file using the provided table definition.
static Table	readSingleFileTable(@NotNull File file, @NotNull ParquetInstructions readInstructions)	Creates a single table via the parquet file using the table definition derived from that file.
static Table	readSingleFileTable(@NotNull File file, @NotNull ParquetInstructions readInstructions, @NotNull TableDefinition tableDefinition)	Creates a single table via the parquet file using the provided tableDefinition.
static Table	readSingleFileTable(@NotNull String source, @NotNull ParquetInstructions readInstructions)	Creates a single table via the parquet source using the provided tableDefinition.
static Table	readSingleFileTable(@NotNull String source, @NotNull ParquetInstructions readInstructions, @NotNull TableDefinition tableDefinition)	Creates a single table via the parquet source using the provided tableDefinition.
static Table	readTable(@NotNull File sourceFile)	Reads in a table from a single parquet file, metadata file, or directory with recognized layout.
static Table	readTable(@NotNull File sourceFile, @NotNull ParquetInstructions readInstructions)	Reads in a table from a single parquet file, metadata file, or directory with recognized layout.
static Table	readTable(@NotNull String source)	Reads in a table from a single parquet file, metadata file, or directory with recognized layout.
static Table	readTable(@NotNull String source, @NotNull ParquetInstructions readInstructions)	Reads in a table from a single parquet file, metadata file, or directory with recognized layout.
static void	setDefaultCompressionCodecName(String compressionCodecName)	Deprecated. Do not use this method, instead pass the above codecs as arguments to writeTable(Table, File, ParquetInstructions) method
static void	writeParquetTables(@NotNull Table[] sources, @NotNull TableDefinition definition, @NotNull ParquetInstructions writeInstructions, @NotNull File[] destinations, @NotNull String[] groupingColumns)	Writes tables to disk in parquet format to a supplied set of destinations.
static void	writeTable(@NotNull Table sourceTable, @NotNull File destFile)	Write a table to a file.
static void	writeTable(@NotNull Table sourceTable, @NotNull File destFile, @NotNull TableDefinition definition)	Write a table to a file.
static void	writeTable(@NotNull Table sourceTable, @NotNull File destFile, @NotNull TableDefinition definition, @NotNull ParquetInstructions writeInstructions)	Write a table to a file.
static void	writeTable(@NotNull Table sourceTable, @NotNull File destFile, @NotNull ParquetInstructions writeInstructions)	Write a table to a file.
static void	writeTable(@NotNull Table sourceTable, @NotNull String destPath)	Write a table to a file.
static void	writeTable(@NotNull Table sourceTable, @NotNull String destPath, @NotNull TableDefinition definition, @NotNull ParquetInstructions writeInstructions)	Write a table to a file.
static void	writeTables(@NotNull Table[] sources, @NotNull TableDefinition definition, @NotNull File[] destinations)	Write out tables to disk.
static void	writeTables(@NotNull Table[] sources, @NotNull TableDefinition definition, @NotNull File[] destinations, @NotNull ParquetInstructions writeInstructions)	Write out tables to disk.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

UNCOMPRESSED

public static final ParquetInstructions UNCOMPRESSED

LZ4

@Deprecated
public static final ParquetInstructions LZ4

Deprecated.

Use LZ4_RAW instead, as explained here

LZ4_RAW

public static final ParquetInstructions LZ4_RAW

LZO

public static final ParquetInstructions LZO

GZIP

public static final ParquetInstructions GZIP

ZSTD

public static final ParquetInstructions ZSTD

SNAPPY

public static final ParquetInstructions SNAPPY

BROTLI

public static final ParquetInstructions BROTLI

LEGACY

public static final ParquetInstructions LEGACY

Method Details

readTable

public static Table readTable(@NotNull
 @NotNull String source)

Reads in a table from a single parquet file, metadata file, or directory with recognized layout. The source provided can be a local file path or a URI to be resolved via the provided SeekableChannelsProviderPlugin.

This method attempts to "do the right thing." It examines the source to determine if it's a single parquet file, a metadata file, or a directory. If it's a directory, it additionally tries to guess the layout to use. Unless a metadata file is supplied or discovered in the directory, the highest (by location key order) location found will be used to infer schema.

Delegates to one of readSingleFileTable(String, ParquetInstructions), readSingleFileTable(File, ParquetInstructions), readPartitionedTableWithMetadata(File, ParquetInstructions), readFlatPartitionedTable(File, ParquetInstructions), or readKeyValuePartitionedTable(File, ParquetInstructions).

Parameters:

source - The path or URI of file or directory to examine

Returns:

table

See Also:

• ParquetSingleFileLayout
• ParquetMetadataFileLayout
• ParquetKeyValuePartitionedLayout
• ParquetFlatPartitionedLayout

readTable

public static Table readTable(@NotNull
 @NotNull String source,
 @NotNull
 @NotNull ParquetInstructions readInstructions)

Reads in a table from a single parquet file, metadata file, or directory with recognized layout. The source provided can be a local file path or a URI to be resolved via the provided SeekableChannelsProviderPlugin.

This method attempts to "do the right thing." It examines the source to determine if it's a single parquet file, a metadata file, or a directory. If it's a directory, it additionally tries to guess the layout to use. Unless a metadata file is supplied or discovered in the directory, the highest (by location key order) location found will be used to infer schema.

Delegates to one of readSingleFileTable(File, ParquetInstructions), readPartitionedTableWithMetadata(File, ParquetInstructions), readFlatPartitionedTable(File, ParquetInstructions), or readKeyValuePartitionedTable(File, ParquetInstructions).

Parameters:

source - The path or URI of file or directory to examine

readInstructions - Instructions for customizations while reading

Returns:

table

See Also:

• ParquetSingleFileLayout
• ParquetMetadataFileLayout
• ParquetKeyValuePartitionedLayout
• ParquetFlatPartitionedLayout

readTable

public static Table readTable(@NotNull
 @NotNull File sourceFile)

Reads in a table from a single parquet file, metadata file, or directory with recognized layout.

This method attempts to "do the right thing." It examines the source to determine if it's a single parquet file, a metadata file, or a directory. If it's a directory, it additionally tries to guess the layout to use. Unless a metadata file is supplied or discovered in the directory, the highest (by location key order) location found will be used to infer schema.

Delegates to one of readSingleFileTable(File, ParquetInstructions), readPartitionedTableWithMetadata(File, ParquetInstructions), readFlatPartitionedTable(File, ParquetInstructions), or readKeyValuePartitionedTable(File, ParquetInstructions).

Parameters:

sourceFile - The file or directory to examine

Returns:

table

See Also:

• ParquetSingleFileLayout
• ParquetMetadataFileLayout
• ParquetKeyValuePartitionedLayout
• ParquetFlatPartitionedLayout

readTable

public static Table readTable(@NotNull
 @NotNull File sourceFile,
 @NotNull
 @NotNull ParquetInstructions readInstructions)

Reads in a table from a single parquet file, metadata file, or directory with recognized layout.

This method attempts to "do the right thing." It examines the source to determine if it's a single parquet file, a metadata file, or a directory. If it's a directory, it additionally tries to guess the layout to use. Unless a metadata file is supplied or discovered in the directory, the highest (by location key order) location found will be used to infer schema.

Delegates to one of readSingleFileTable(File, ParquetInstructions), readPartitionedTableWithMetadata(File, ParquetInstructions), readFlatPartitionedTable(File, ParquetInstructions), or readKeyValuePartitionedTable(File, ParquetInstructions).

Parameters:

sourceFile - The file or directory to examine

readInstructions - Instructions for customizations while reading

Returns:

table

See Also:

• ParquetSingleFileLayout
• ParquetMetadataFileLayout
• ParquetKeyValuePartitionedLayout
• ParquetFlatPartitionedLayout

writeTable

public static void writeTable(@NotNull
 @NotNull Table sourceTable,
 @NotNull
 @NotNull String destPath)

Write a table to a file.

Parameters:

sourceTable - source table

destPath - destination file path; the file name should end in ".parquet" extension If the path includes non-existing directories they are created If there is an error any intermediate directories previously created are removed; note this makes this method unsafe for concurrent use

writeTable

public static void writeTable(@NotNull
 @NotNull Table sourceTable,
 @NotNull
 @NotNull File destFile)

Write a table to a file.

Parameters:

sourceTable - source table

destFile - destination file; the file name should end in ".parquet" extension If the path includes non-existing directories they are created

writeTable

public static void writeTable(@NotNull
 @NotNull Table sourceTable,
 @NotNull
 @NotNull File destFile,
 @NotNull
 @NotNull TableDefinition definition)

Write a table to a file.

Parameters:

sourceTable - source table

destFile - destination file; its path must end in ".parquet". Any non existing directories in the path are created If there is an error any intermediate directories previously created are removed; note this makes this method unsafe for concurrent use

definition - table definition to use (instead of the one implied by the table itself)

writeTable

public static void writeTable(@NotNull
 @NotNull Table sourceTable,
 @NotNull
 @NotNull File destFile,
 @NotNull
 @NotNull ParquetInstructions writeInstructions)

Write a table to a file.

Parameters:

sourceTable - source table

destFile - destination file; its path must end in ".parquet". Any non existing directories in the path are created If there is an error any intermediate directories previously created are removed; note this makes this method unsafe for concurrent use

writeInstructions - instructions for customizations while writing

writeTable

public static void writeTable(@NotNull
 @NotNull Table sourceTable,
 @NotNull
 @NotNull String destPath,
 @NotNull
 @NotNull TableDefinition definition,
 @NotNull
 @NotNull ParquetInstructions writeInstructions)

Write a table to a file.

Parameters:

sourceTable - source table

destPath - destination path; it must end in ".parquet". Any non existing directories in the path are created If there is an error any intermediate directories previously created are removed; note this makes this method unsafe for concurrent use

definition - table definition to use (instead of the one implied by the table itself)

writeInstructions - instructions for customizations while writing

writeTable

public static void writeTable(@NotNull
 @NotNull Table sourceTable,
 @NotNull
 @NotNull File destFile,
 @NotNull
 @NotNull TableDefinition definition,
 @NotNull
 @NotNull ParquetInstructions writeInstructions)

Write a table to a file.

Parameters:

sourceTable - source table

destFile - destination file; its path must end in ".parquet". Any non-existing directories in the path are created If there is an error any intermediate directories previously created are removed; note this makes this method unsafe for concurrent use

definition - table definition to use (instead of the one implied by the table itself)

writeInstructions - instructions for customizations while writing

getRelativeIndexFilePath

public static String getRelativeIndexFilePath(@NotNull
 @NotNull File tableDest,
 @NotNull
 @NotNull String columnName)

Generates the index file path relative to the table destination file path.

Parameters:

tableDest - Destination path for the main table containing these indexing columns

columnName - Name of the indexing column

Returns:

The relative index file path. For example, for table with destination "table.parquet" and indexing column "IndexingColName", the method will return ".dh_metadata/indexes/IndexingColName/index_IndexingColName_table.parquet"

legacyGroupingFileName

public static String legacyGroupingFileName(@NotNull
 @NotNull File tableDest,
 @NotNull
 @NotNull String columnName)

Legacy method for generating a grouping file name. We used to place grouping files right next to the original table destination.

Parameters:

tableDest - Destination path for the main table containing these grouping columns

columnName - Name of the grouping column

Returns:

The relative grouping file path. For example, for table with destination "table.parquet" and grouping column "GroupingColName", the method will return "table_GroupingColName_grouping.parquet"

writeParquetTables

public static void writeParquetTables(@NotNull
 @NotNull Table[] sources,
 @NotNull
 @NotNull TableDefinition definition,
 @NotNull
 @NotNull ParquetInstructions writeInstructions,
 @NotNull
 @NotNull File[] destinations,
 @NotNull
 @NotNull String[] groupingColumns)

Writes tables to disk in parquet format to a supplied set of destinations. If you specify grouping columns, there must already be grouping information for those columns in the sources. This can be accomplished with .groupBy(<grouping columns>).ungroup() or .sort(<grouping column>).

Parameters:

sources - The tables to write

definition - The common schema for all the tables to write

writeInstructions - Write instructions for customizations while writing

destinations - The destinations paths. Any non-existing directories in the paths provided are created. If there is an error any intermediate directories previously created are removed; note this makes this method unsafe for concurrent use

groupingColumns - List of columns the tables are grouped by (the write operation will store the grouping info)

writeTables

public static void writeTables(@NotNull
 @NotNull Table[] sources,
 @NotNull
 @NotNull TableDefinition definition,
 @NotNull
 @NotNull File[] destinations)

Write out tables to disk.

Parameters:

sources - source tables

definition - table definition

destinations - destinations

writeTables

public static void writeTables(@NotNull
 @NotNull Table[] sources,
 @NotNull
 @NotNull TableDefinition definition,
 @NotNull
 @NotNull File[] destinations,
 @NotNull
 @NotNull ParquetInstructions writeInstructions)

Write out tables to disk.

Parameters:

sources - source tables

definition - table definition

destinations - destinations

writeInstructions - instructions for customizations while writing

deleteTable

@VisibleForTesting
public static void deleteTable(File path)

Deletes a table on disk.

Parameters:

path - path to delete

readSingleFileTable

public static Table readSingleFileTable(@NotNull
 @NotNull ParquetTableLocationKey tableLocationKey,
 @NotNull
 @NotNull ParquetInstructions readInstructions,
 @NotNull
 @NotNull TableDefinition tableDefinition)

Reads in a table from a single parquet file using the provided table definition.

Callers may prefer the simpler methods readSingleFileTable(File, ParquetInstructions) or readSingleFileTable(File, ParquetInstructions, TableDefinition).

Parameters:

tableLocationKey - The location keys to include

readInstructions - Instructions for customizations while reading

tableDefinition - The table's definition

Returns:

The table

readPartitionedTable

public static Table readPartitionedTable(@NotNull
 @NotNull TableLocationKeyFinder<ParquetTableLocationKey> locationKeyFinder,
 @NotNull
 @NotNull ParquetInstructions readInstructions)

Reads in a table from files discovered with locationKeyFinder using a definition built from the highest (by location key order) location found, which must have non-null partition values for all partition keys.

Parameters:

locationKeyFinder - The source of location keys to include

readInstructions - Instructions for customizations while reading

Returns:

The table

readPartitionedTable

public static Table readPartitionedTable(@NotNull
 @NotNull TableLocationKeyFinder<ParquetTableLocationKey> locationKeyFinder,
 @NotNull
 @NotNull ParquetInstructions readInstructions,
 @NotNull
 @NotNull TableDefinition tableDefinition)

Reads in a table from files discovered with locationKeyFinder using the provided table definition.

Parameters:

locationKeyFinder - The source of location keys to include

readInstructions - Instructions for customizations while reading

tableDefinition - The table's definition

Returns:

The table

readPartitionedTableInferSchema

@Deprecated
public static Table readPartitionedTableInferSchema(@NotNull
 @NotNull TableLocationKeyFinder<ParquetTableLocationKey> locationKeyFinder,
 @NotNull
 @NotNull ParquetInstructions readInstructions)

Deprecated.

use readPartitionedTable(TableLocationKeyFinder, ParquetInstructions)

Reads in a table from files discovered with locationKeyFinder using a definition built from the highest (by location key order) location found, which must have non-null partition values for all partition keys.

Parameters:

locationKeyFinder - The source of location keys to include

readInstructions - Instructions for customizations while reading

Returns:

The table

readPartitionedTableWithMetadata

public static Table readPartitionedTableWithMetadata(@NotNull
 @NotNull File directory,
 @NotNull
 @NotNull ParquetInstructions readInstructions)

Reads in a table using metadata files found in the supplied directory.

Parameters:

directory - The source of location keys to include

readInstructions - Instructions for customizations while reading

Returns:

The table

readKeyValuePartitionedTable

public static Table readKeyValuePartitionedTable(@NotNull
 @NotNull File directory,
 @NotNull
 @NotNull ParquetInstructions readInstructions)

Creates a partitioned table via the key-value partitioned parquet files from the root directory, inferring the table definition from those files.

Callers wishing to be more explicit and skip the inference step may prefer to call readKeyValuePartitionedTable(File, ParquetInstructions, TableDefinition).

Parameters:

directory - the source of location keys to include

readInstructions - the instructions for customizations while reading

Returns:

the table

See Also:

• ParquetKeyValuePartitionedLayout(File, int, ParquetInstructions)
• readPartitionedTable(TableLocationKeyFinder, ParquetInstructions)

readKeyValuePartitionedTable

public static Table readKeyValuePartitionedTable(@NotNull
 @NotNull File directory,
 @NotNull
 @NotNull ParquetInstructions readInstructions,
 @NotNull
 @NotNull TableDefinition tableDefinition)

Creates a partitioned table via the key-value partitioned parquet files from the root directory using the provided tableDefinition.

Parameters:

directory - the source of location keys to include

readInstructions - the instructions for customizations while reading

tableDefinition - the table definition

Returns:

the table

See Also:

• ParquetKeyValuePartitionedLayout(File, TableDefinition, ParquetInstructions)
• readPartitionedTable(TableLocationKeyFinder, ParquetInstructions, TableDefinition)

readFlatPartitionedTable

public static Table readFlatPartitionedTable(@NotNull
 @NotNull File directory,
 @NotNull
 @NotNull ParquetInstructions readInstructions)

Creates a partitioned table via the flat parquet files from the root directory, inferring the table definition from those files.

Callers wishing to be more explicit and skip the inference step may prefer to call readFlatPartitionedTable(File, ParquetInstructions, TableDefinition).

Parameters:

directory - the source of location keys to include

readInstructions - the instructions for customizations while reading

Returns:

the table

See Also:

• readPartitionedTable(TableLocationKeyFinder, ParquetInstructions)
• ParquetFlatPartitionedLayout(File, ParquetInstructions)

readFlatPartitionedTable

public static Table readFlatPartitionedTable(@NotNull
 @NotNull File directory,
 @NotNull
 @NotNull ParquetInstructions readInstructions,
 @NotNull
 @NotNull TableDefinition tableDefinition)

Creates a partitioned table via the flat parquet files from the root directory using the provided tableDefinition.

Parameters:

directory - the source of location keys to include

readInstructions - the instructions for customizations while reading

tableDefinition - the table definition

Returns:

the table

See Also:

• readPartitionedTable(TableLocationKeyFinder, ParquetInstructions, TableDefinition)
• ParquetFlatPartitionedLayout(File, ParquetInstructions)

readSingleFileTable

public static Table readSingleFileTable(@NotNull
 @NotNull File file,
 @NotNull
 @NotNull ParquetInstructions readInstructions)

Creates a single table via the parquet file using the table definition derived from that file.

Callers wishing to be more explicit (for example, to skip some columns) may prefer to call readSingleFileTable(File, ParquetInstructions, TableDefinition).

Parameters:

file - the parquet file

readInstructions - the instructions for customizations while reading

Returns:

the table

See Also:

• ParquetTableLocationKey(URI, int, Map, ParquetInstructions)
• readSingleFileTable(ParquetTableLocationKey, ParquetInstructions, TableDefinition)

readSingleFileTable

public static Table readSingleFileTable(@NotNull
 @NotNull String source,
 @NotNull
 @NotNull ParquetInstructions readInstructions)

Creates a single table via the parquet source using the provided tableDefinition. The source provided can be a local file path or a URI to be resolved via the provided SeekableChannelsProviderPlugin

Callers wishing to be more explicit (for example, to skip some columns) may prefer to call readSingleFileTable(String, ParquetInstructions, TableDefinition).

Parameters:

source - the path or URI for the parquet file

readInstructions - the instructions for customizations while reading

Returns:

the table

See Also:

• ParquetTableLocationKey(URI, int, Map, ParquetInstructions)
• readSingleFileTable(ParquetTableLocationKey, ParquetInstructions, TableDefinition)

readSingleFileTable

public static Table readSingleFileTable(@NotNull
 @NotNull File file,
 @NotNull
 @NotNull ParquetInstructions readInstructions,
 @NotNull
 @NotNull TableDefinition tableDefinition)

Creates a single table via the parquet file using the provided tableDefinition.

Parameters:

file - the parquet file

readInstructions - the instructions for customizations while reading

tableDefinition - the table definition

Returns:

the table

See Also:

• ParquetTableLocationKey(URI, int, Map, ParquetInstructions)
• readSingleFileTable(ParquetTableLocationKey, ParquetInstructions, TableDefinition)

readSingleFileTable

public static Table readSingleFileTable(@NotNull
 @NotNull String source,
 @NotNull
 @NotNull ParquetInstructions readInstructions,
 @NotNull
 @NotNull TableDefinition tableDefinition)

Creates a single table via the parquet source using the provided tableDefinition. The source provided can be a local file path or a URI to be resolved via the provided SeekableChannelsProviderPlugin

Parameters:

source - the path or URI for the parquet file

readInstructions - the instructions for customizations while reading

tableDefinition - the table definition

Returns:

the table

See Also:

• ParquetTableLocationKey(URI, int, Map, ParquetInstructions)
• readSingleFileTable(ParquetTableLocationKey, ParquetInstructions, TableDefinition)

getParquetFileReader

public static ParquetFileReader getParquetFileReader(@NotNull
 @NotNull File parquetFile,
 @NotNull
 @NotNull ParquetInstructions readInstructions)

Make a ParquetFileReader for the supplied File. Wraps IOException as TableDataException.

Parameters:

parquetFile - The File to read

readInstructions - the instructions for customizations while reading

Returns:

The new ParquetFileReader

getParquetFileReader

public static ParquetFileReader getParquetFileReader(@NotNull
 @NotNull URI parquetFileURI,
 @NotNull
 @NotNull ParquetInstructions readInstructions)

Make a ParquetFileReader for the supplied URI. Wraps IOException as TableDataException.

Parameters:

parquetFileURI - The URI to read

readInstructions - the instructions for customizations while reading

Returns:

The new ParquetFileReader

getParquetFileReaderChecked

public static ParquetFileReader getParquetFileReaderChecked(@NotNull
 @NotNull File parquetFile,
 @NotNull
 @NotNull ParquetInstructions readInstructions)
                                                     throws IOException

Make a ParquetFileReader for the supplied File.

Parameters:

parquetFile - The File to read

Returns:

The new ParquetFileReader

Throws:

IOException - if an IO exception occurs

getParquetFileReaderChecked

public static ParquetFileReader getParquetFileReaderChecked(@NotNull
 @NotNull URI parquetFileURI,
 @NotNull
 @NotNull ParquetInstructions readInstructions)
                                                     throws IOException

Make a ParquetFileReader for the supplied URI.

Parameters:

parquetFileURI - The URI to read

Returns:

The new ParquetFileReader

Throws:

IOException - if an IO exception occurs

readParquetSchemaAndTable

@VisibleForTesting
public static Table readParquetSchemaAndTable(@NotNull
 @NotNull File source,
 @NotNull
 @NotNull ParquetInstructions readInstructionsIn,
 org.apache.commons.lang3.mutable.MutableObject<ParquetInstructions> instructionsOut)

convertSchema

public static Pair<List<ColumnDefinition<?>>,ParquetInstructions> convertSchema(@NotNull
 @NotNull org.apache.parquet.schema.MessageType schema,
 @NotNull
 @NotNull Map<String,String> keyValueMetadata,
 @NotNull
 @NotNull ParquetInstructions readInstructionsIn)

Convert schema information from a ParquetMetadata into ColumnDefinitions.

Parameters:

schema - Parquet schema. DO NOT RELY ON ParquetMetadataConverter FOR THIS! USE ParquetFileReader!

keyValueMetadata - Parquet key-value metadata map

readInstructionsIn - Input conversion ParquetInstructions

Returns:

A Pair with ColumnDefinitions and adjusted ParquetInstructions

setDefaultCompressionCodecName

@Deprecated
public static void setDefaultCompressionCodecName(String compressionCodecName)

Deprecated.

Do not use this method, instead pass the above codecs as arguments to writeTable(Table, File, ParquetInstructions) method