dhconfig checkpoint

Checkpoint files are special files written by Deephaven to periodically record the state of tables on disk. These files allow the system to efficiently recover or reconstruct table data after a restart or failure, by resuming from the last checkpoint rather than replaying all historical data. Checkpoints are particularly important for large or persistent tables, as they significantly reduce recovery time and resource usage. Administrators may need to examine checkpoint files to verify table persistence, troubleshoot data issues, or audit system state.

The checkpoint configuration data type of the dhconfig tool is used to examine and repair table checkpoint files on disk. Authentication options do not apply to this command; the executing user must have read (and, for repair, write) access to the files to be operated on. Checkpoint files are named table.size by default. File arguments may be the checkpoint file itself, or the directory containing table.size.

checkpoint has two actions: list and repair. The --help argument provides detailed information on each action's available options and arguments.

list

list prints the contents of checkpoint files to stdout. Include --help for a description of all the subcommand's options and some usage examples:

list requires at least one checkpoint file, directory, or list file. Trailing arguments are treated as checkpoint files, directories (searched recursively), or list files.

OptionDescription
-f,--file <arg>A checkpoint file (for example, table.size) to read. A directory is searched recursively, as with --directory. Can be given multiple times.
-d,--directory <arg>A directory to recursively search for checkpoint files. Can be given multiple times.
-l,--list <arg>A file that lists checkpoint files to read, one per line. Can be given multiple times.
-e,--continue-on-errorProcess all arguments even if some of them cause errors. Without this option, list stops at the first error.
-csv,--csvPrint one summary line per checkpoint file in CSV format, instead of the full checkpoint record.
-head,--headerPrint a header line before the CSV output. Only valid with --csv.

list examples

Display a specific checkpoint file:

Display all checkpoint files starting at a directory (recursively searching):

Display a summary of all checkpoint files starting at a directory in CSV format:

Display all checkpoint files under a directory, continuing past any files that cannot be read:

Display a summary of all checkpoint files in a list file in CSV format:

Examples with output:

Output:

Output:

The header lists columnCount, but each row has only seven values: filename, version, tableSize, lastModificationTime, lastModificationTimeStr, sourceFile, and sourceBytes. A missing source file prints as null. Read the values after lastModificationTimeStr by position, not by header name.

repair

repair reports on (and, with --force, repairs) checkpoint records for the selected tables. Empty or non-directory locations are deleted; locations with a missing or out-of-date checkpoint record are regenerated or upgraded. Tables are selected either by name with --table-name or by namespace set with --namespaceset (System or User); the two options are mutually exclusive.

Include --help for a description of all the subcommand's options and some usage examples:

A description of all the subcommand's options and some usage examples follows.

Without --force, issues are only reported and nothing is changed. When --force repairs a system historical location, repair advises running dhctl metadata update to rebuild the corresponding metadata index.

For a system namespace, repair inspects both intraday and historical locations. Historical locations are reported by their Partitions path, the same path the metadata index and the query engine use. --force writes to the Partitions path when it is writable. When it is not, such as when Partitions is mounted read-only, --force writes through the corresponding WritablePartitions entry instead. If the namespace has a separate WritablePartitions directory with no entry for the location's internal partition, the location is still reported, but --force cannot repair it and marks it failed; repair it from a server that can write that partition. Extended layouts have no checkpoint records and are never inspected.

repair examples

Report checkpoint record issues for a single table without changing anything:

Repair checkpoint records for every table in Namespace1 and the table Namespace2.MyTable:

Repair checkpoint records for every user table:

Repair checkpoint records for every table in all namespaces. The * should be quoted to avoid shell file globbing:

repair from scripts

The repair action is a thin wrapper over the io.deephaven.configuration.CheckpointControl class, and the same functionality is available from query scripts: in Groovy on any worker, and in Python on a Core+ worker. The worker must run with filesystem permission to read (and, when repairing, write) the affected table locations.

Nothing is modified unless the repair is forced; by default problems are only reported.

Individually selected tables, whole namespaces, and whole namespace sets are unioned, and at least one selection must be made.

Check the results

The result reports how much was inspected and describes every location that had a problem. Locations without problems are not reported.

  • summary_status (Python) or getSummaryStatus() (Groovy) is the worst status reported: OK, FLAGGED (a problem was found but not acted on, because the repair was not forced), REPAIRED, or FAILED.
  • location_results / getLocationResults() lists the affected locations. Each entry carries the namespace, table name, path, the issue found, the resulting status, and a message. The path is None (null in Groovy) when the table's definition could not be read (DEFINITION_UNAVAILABLE), because none of that table's locations were inspected.
  • metadata_index_affected / isMetadataIndexAffected() is true when any reported location belongs to a System historical table, whose locations may be tracked by a metadata index. Rebuild the index for those tables with dhctl metadata update once they are repaired.