Use the data control tool from a Legacy Groovy worker

Warning

Legacy documentation: This documentation applies to Legacy Deephaven Enterprise only and does not apply to Core+.

Tip

For Core+ workers (Groovy or Python), see Data control from scripts.

The data control tool (dhctl) is a command-line tool that allows you to manage data in Deephaven. This functionality is also available from any Legacy worker with sufficient permissions.

Truncate and delete intraday partitions

Truncate and delete are configured by an Options value built via Options.builder(). Build an Options and pass it to IntradayControl.truncateIntradayPartition(options) or IntradayControl.deleteIntradayPartition(options).

import io.deephaven.configuration.IntradayControl

// truncate the partitions for this table and column partition value (like --partitions)
opts = IntradayControl.Options.builder().key(namespace, tableName, columnPartitionValue).build()
IntradayControl.truncateIntradayPartition(opts)

// truncate the single partition for this table and partition values (like --singlePartition)
opts = IntradayControl.Options.builder()
        .key(namespace, tableName, internalPartitionValue, columnPartitionValue).build()
IntradayControl.truncateIntradayPartition(opts)

// truncate the location(s) specified by an existing FullTableLocationKey
opts = IntradayControl.Options.builder().key(existingKey).build()
IntradayControl.truncateIntradayPartition(opts)

// delete uses the same shape; the partition must be truncated first
opts = IntradayControl.Options.builder().key(namespace, tableName, columnPartitionValue).build()
IntradayControl.deleteIntradayPartition(opts)

Tip

You will want to check the results of these commands.

result = IntradayControl.truncateIntradayPartition(opts)
println result

Options Builder

The options builder can be useful when programmatically constructing complex truncate or delete commands.

import io.deephaven.configuration.IntradayControl.Options
import io.deephaven.configuration.IntradayControl
builder = Options.builder()

Modify the builder with the desired options, much like the dhctl command line options.

Set the partition to be deleted:

builder.key("namespace", "tableName", "columnPartition")
builder.key("namespace", "tableName", "internalPartition", "columnPartition")
builder.key(existingKey)

For example:

key = new FullTableLocationKey.AggregateTableLocationKey("DbInternal", "ProcessEventLog",  SYSTEM_INTRADAY, lastBusinessDateNy())
builder.key(key)

// or, equivalently
builder.key("DbInternal", "ProcessEventLog", lastBusinessDateNy())

Dry run options

Change the dry run option:

builder.dryRun(true)  // perform a dry run, and do not perform the truncate or delete action
builder.dryRun(false) // perform the actual operation (the default)

Add a DIS

Add a DIS to the include or exclude list:

// one DIS at a time
builder.addDisInclude("dis_name")
builder.addDisExclude("dis_name")

// or add several at once from any Iterable<String>
builder.addAllDisIncludes(["dis_a", "dis_b"])
builder.addAllDisExcludes(["dis_c", "dis_d"])

Build options

Build the options as configured in the builder:

opts = builder.build()

Pass the options to the Intraday Control tool and check the results:

result = IntradayControl.truncateIntradayPartition(opts)
println result
result = IntradayControl.deleteIntradayPartition(opts)
println result

You can display the contents of the options:

println opts

Check the results

Truncate and delete commands return DisCommandResult objects containing detailed information about the operation. The intraday operations can be complex, so the result object is also necessarily complex. The DisCommandResult object has an overall result that indicates success or failure of the operation as a whole. It also contains a map of results for each DIS that was involved in the operation. There is an overall result code for each DIS, and a collection of results for the individual locations that were processed. The code examples below illustrate how to check the results at various levels of detail. These are intended as examples; adjust the code to suit your needs.

This example executes a truncate dry run, and then executes the truncate only if none of the locations to be truncated are still in use:

import io.deephaven.configuration.IntradayControl
import io.deephaven.configuration.DisCommandResult

namespace="DbInternal"
tableName="ProcessEventLog"
date=currentDateNy()

def dryRunOpts = IntradayControl.Options.builder()
        .key(namespace, tableName, date)
        .dryRun(true)
        .build()

dryRunResult = IntradayControl.truncateIntradayPartition(dryRunOpts)

// verify the overall result
okToTruncate = dryRunResult.getSummaryResult() == DisCommandResult.Status.SUCCESS

// verify all the DIS results are success
okToTruncate = okToTruncate &&
               dryRunResult.getDisResultsMap().values().every { disResult -> disResult.getStatus() == DisCommandResult.Status.SUCCESS }

// check for locations being actively tailed
hasActiveProcessor = dryRunResult.getDisResultsMap().values().any { disResult -> disResult.getLocationResults().any { locationResult -> locationResult.hasActiveProcessor() } }
okToTruncate = okToTruncate && !hasActiveProcessor
println okToTruncate

if (okToTruncate) {
    def realOpts = IntradayControl.Options.builder()
            .key(namespace, tableName, date)
            .build()
    commandResult = IntradayControl.truncateIntradayPartition(realOpts)
    println commandResult
}

This example executes a truncate command and then, if the truncate was successful, delete:

import io.deephaven.configuration.IntradayControl
import io.deephaven.configuration.DisCommandResult

namespace="DbInternal"
tableName="ProcessEventLog"
date="2025-03-19"
opts = IntradayControl.Options.builder().key(namespace, tableName, date).build()

truncateResult = IntradayControl.truncateIntradayPartition(opts)
printf "Truncate %s.%s partition %s: \n", namespace, tableName, date
println truncateResult

// verify the overall result
if (truncateResult.getSummaryResult() == DisCommandResult.Status.SUCCESS &&
    truncateResult.getDisResultsMap().size() >= 1 &&
    truncateResult.getDisResultsMap().values().every { disResult -> disResult.getStatus() == DisCommandResult.Status.SUCCESS }) {
    deleteResult = IntradayControl.deleteIntradayPartition(opts)
    printf "Delete %s.%s partition %s: \n", namespace, tableName, date
    println deleteResult
}

Rescan tables

This command instructs the DIS handling table DbInternal.ProcessEventLog to look for new data:

import io.deephaven.configuration.IntradayControl

result = IntradayControl.rescan(IntradayControl.RescanOptions.builder().table("DbInternal", "ProcessEventLog").build())
println result
...
Result: SUCCESS
    DIS: db_dis(all)
        Result: SUCCESS

Caution

Always specify a --table-name when running rescan. Rescans should target specific tables for specific purposes. Omitting this option instructs DISes to rescan all intraday tables, which is disruptive and rarely the intended action. This is especially important in scheduled or scripted scenarios — a blanket rescan can cause unnecessary load and delays.

The following example shows rescanning all tables on all DISs. This is not recommended.

import io.deephaven.configuration.IntradayControl

result = IntradayControl.rescan(IntradayControl.RescanOptions.builder().build())
println result
...
Result: SUCCESS
    DIS: db_dis(all)
        Result: SUCCESS
    DIS: Ingester1(all)
        Result: SUCCESS

Caveats

This method makes a best-effort attempt to delete everything on all appropriate Data Import Servers. This cannot be atomic, so the operation might have only partial success. Make sure you check all the results.
The truncated partitions are marked as permanently truncated, and further ingestion of data will be disallowed. This is to prevent confusion if loggers produce new data for the partition, or if tailers have not finished all existing data files.
Before logging new data for the truncated partitions, remove any existing data files (bin files), and then delete the partitions with dhctl intraday delete ....
It is possible to delete data on one Data Import Server and leave it on another (e.g., a backup). Be extremely careful with this, as it can create confusion.