dhconfig checkpoint

Checkpoint files are special files written by Deephaven to periodically record the state of tables on disk. These files allow the system to efficiently recover or reconstruct table data after a restart or failure, by resuming from the last checkpoint rather than replaying all historical data. Checkpoints are particularly important for large or persistent tables, as they significantly reduce recovery time and resource usage. Administrators may need to examine checkpoint files to verify table persistence, troubleshoot data issues, or audit system state.

The checkpoint configuration data type of the dhconfig tool is used to examine and repair table checkpoint files on disk. Authentication options do not apply to this command; the executing user must have read (and, for repair, write) access to the files to be operated on. Checkpoint files are named table.size by default. File arguments may be the checkpoint file itself, or the directory containing table.size.

checkpoint has two actions: list and repair. The --help argument provides detailed information on each action's available options and arguments.

list prints checkpoint configurations to a file or to stdout:

repair reports on (and, with --force, repairs) checkpoint records for the selected tables. Empty or non-directory locations are deleted; locations with a missing or out-of-date checkpoint record are regenerated or upgraded. Tables are selected either by name with --table-name or in bulk by namespace set with --namespaceset (System or User); the two options are mutually exclusive:

Without --force, issues are only reported and nothing is changed. When any system table location is repaired or flagged, repair advises running dhctl metadata update to rebuild the corresponding metadata index.

...followed by a description of all of the subcommand's options and some usage examples.

list examples

Display a specific checkpoint file:

Display all checkpoint files starting at a directory (recursively searching):

Display a summary of all checkpoint files starting at a directory in CSV format:

Display a summary of all checkpoint files in a list file in CSV format:

Examples with output:

Output:

Output:

repair examples

Report checkpoint record issues for a single table without changing anything:

Repair checkpoint records for every table in Namespace1 and the table Namespace2.MyTable:

Repair checkpoint records for every user table:

Repair checkpoint records for every table in all namespaces. The * should be quoted to avoid shell file globbing:

repair from scripts

The repair action is a thin wrapper over the io.deephaven.configuration.CheckpointControl class, and the same functionality is available from query scripts: in Groovy on any worker, and in Python on a Core+ worker. The worker must run with filesystem permission to read (and, when repairing, write) the affected table locations.

Nothing is modified unless the repair is forced; by default problems are only reported.

Individually selected tables, whole namespaces, and whole namespace sets are unioned, and at least one selection must be made.

Check the results

The result reports how much was inspected and describes every location that had a problem. Locations without problems are not reported.

  • summary_status (Python) or getSummaryStatus() (Groovy) is the worst status reported: OK, FLAGGED (a problem was found but not acted on, because the repair was not forced), REPAIRED, or FAILED.
  • location_results / getLocationResults() lists the affected locations. Each entry carries the namespace, table name, path, the issue found, the resulting status, and a message.
  • system_tables_affected / isSystemTablesAffected() is true when any reported location belongs to a System table. Rebuild the metadata index for those tables with dhctl metadata update once they are repaired.