Use the data control tool from a Legacy Groovy worker

Warning

Legacy documentation: This documentation applies to Legacy Deephaven Enterprise only and does not apply to Core+.

The data control tool (dhctl) is a command-line tool that allows you to manage data in Deephaven. This functionality is also available from any Legacy worker with sufficient permissions.

Truncate and delete intraday partitions

A builder, described below, provides full control over parameters. For simplicity, some simple options can be invoked via helper methods. These methods do not allow for dry runs or the selection of which Data Import Servers are included.

import io.deephaven.configuration.IntradayControlImpl

// truncate the location(s) specified by the key
IntradayControlImpl.truncateIntradayPartition(FullTableLocationKey)

// truncate the partitions for this table and column partition value (like --partitions)
IntradayControlImpl.truncateIntradayPartition(namespace, tableName, columnPartitionValue)

// truncate the single partition for this table and partition values (like --singlePartition)
IntradayControlImpl.truncateIntradayPartition(namespace, tableName, internalPartitionValue, columnPartitionValue)

// truncate partition(s) as indicated by the options argument (see Options Builder below)
IntradayControlImpl.truncateIntradayPartition(options)

// delete the location(s) specified by the key
IntradayControlImpl.deleteIntradayPartition(FullTableLocationKey)

// delete the partitions for this table and column partition value (like --partitions)
IntradayControlImpl.deleteIntradayPartition(namespace, tableName, columnPartitionValue)

// delete the single partition for this table and partition values (like --singlePartition)
IntradayControlImpl.deleteIntradayPartition(namespace, tableName, internalPartitionValue, columnPartitionValue)

// delete partition(s) as indicated by the options argument (see Options Builder below)
IntradayControlImpl.deleteIntradayPartition(options)

Tip

You will want to check the results of these commands.

result = IntradayControlImpl.truncateIntradayPartition(...)
println result

println IntradayControlImpl.truncateIntradayPartition(...)

Options Builder

The options builder can be useful when programmatically constructing complex truncate or delete commands.

import io.deephaven.configuration.IntradayControlImpl.Options
import io.deephaven.configuration.IntradayControlImpl
builder = Options.builder()

Modify the builder with the desired options, much like the dhctl command line options.

Set the partition to be deleted:

builder.key("namespace", "tableName", "columnPartition")
builder.key("namespace", "tableName", "internalPartition", "columnPartition")
builder.key(key)

For example:

key = new FullTableLocationKey.AggregateTableLocationKey("DbInternal", "ProcessEventLog",  SYSTEM_INTRADAY, lastBusinessDateNy())
builder.key(key)

builder.key("DbInternal", "ProcessEventLog", lastBusinessDateNy())

Note

Only one key is permitted at this time. You can call one of the key() methods again, but you must call builder.clearKey() in between.

Dry run options

Change the dry run option:

builder.dryRun()     // delegates to builder.dryRun(true)
builder.dryRun(true) // perform a dry run, and do not perform the truncate or delete action

Authentication

Change the authentication (by default, the command will be run using the default authentication of the worker):

builder.authenticate("username", "password")
builder.authenticate("username", "password", "operateAsUser")
builder.authenticate("path_to_keyfile") // keyfile must be readable by the worker
builder.authenticate() // do default authentication, according to process environment and properties

Add a DIS

Add a DIS to the include or exclude list:

builder.include("dis_name")
builder.exclude("dis_name")

Build options

Build the options as configured in the builder. You may call build() multiple times. This allows you to use a builder to check a dry run and then perform the delete, or to change the key in a loop.

opts = builder.build()

You can invoke the truncate or delete methods directly from the builder or options:

result = builder.doTruncate()
result = opts.doTruncate()
println result
result = builder.doDelete()
result = opts.doDelete()
println result

You can pass the options to the Intraday Control tool and check the results:

result = IntradayControlImpl.truncateIntradayPartition(opts)
println result

You can display the contents of the builder or options:

println builder
println opts

Check the results

Truncate and delete commands return result objects containing detailed information about the operation. The intraday operations can be complex, so the result object is also necessarily complex. The DISCommandUtil.ActionResult object has an overall result that indicates success or failure of the operation as a whole. It also contains a map of results for each DIS that was involved in the operation. There is an overall result code for each DIS, and a collection of results for the individual locations that were processed. The code examples below illustrate how to check the results at various levels of detail. These are intended as examples; adjust the code to suit your needs.

This example executes a truncate dry run, and then executes the truncate only if none of the locations to be truncated are still in use:

import io.deephaven.configuration.IntradayControlImpl
import com.illumon.iris.db.tables.dataimport.logtailer.DISCommandUtil

namespace="DbInternal"
tableName="ProcessEventLog"
date=currentDateNy()

def builder = IntradayControlImpl.Options.builder()
        .key(namespace, tableName, date)
        .dryRun(true)

dryRunResult = builder.doTruncate()

// verify the overall result
okToTruncate = dryRunResult.getSummaryResult() == DISCommandUtil.ActionResult.Result.SUCCESS

// verify all the if any DIS results are success
okToTruncate = okToTruncate &&
               dryRunResult.getDisResultsMap().values().every { disResult -> disResult.getResult() == DISCommandUtil.ActionResult.Result.SUCCESS }

// check for locations being actively tailed
hasActiveProcessor = dryRunResult.getDisResultsMap().values().any { disResult -> disResult.getLocationResults().any { locationResult -> locationResult.hasActiveProcessor() } }
okToTruncate = okToTruncate && !hasActiveProcessor
println okToTruncate

if (okToTruncate) {
    builder.dryRun(false)
    commandResult = builder.doTruncate()
    println commandResult
}

This example executes a truncate command and then, if the truncate was successful, delete:

import io.deephaven.configuration.IntradayControlImpl
import com.illumon.iris.db.tables.dataimport.logtailer.DISCommandUtil

namespace="DbInternal"
tableName="ProcessEventLog"
date="2025-03-19"

truncateResult = IntradayControlImpl.truncateIntradayPartition(namespace, tableName, date)
printf "Truncate %s.%s partition %s: \n", namespace, tableName, date
println truncateResult

// verify the overall result
if (truncateResult.getSummaryResult() == DISCommandUtil.ActionResult.Result.SUCCESS &&
    truncateResult.getDisResultsMap().size() >= 1 &&
    truncateResult.getDisResultsMap().values().every { disResult -> disResult.getResult() == DISCommandUtil.ActionResult.Result.SUCCESS }) {
    deleteResult = IntradayControlImpl.truncateIntradayPartition(namespace, tableName, date)
    printf "Delete %s.%s partition %s: \n", namespace, tableName, date
    println deleteResult
}

Rescan tables

This command instructs the DIS handling table DbInternal.ProcessEventLog to look for new data:

import io.deephaven.configuration.IntradayControlImpl

result = IntradayControlImpl.rescanTable("DbInternal", "ProcessEventLog")
println result
...
Re-scan Result: SUCCESS
     DIS: db_dis(all)
        Result: SUCCESS

This command instructs all DISs to look for new data for all tables:

import io.deephaven.configuration.IntradayControlImpl

result = IntradayControlImpl.rescanAll()
println result
...
Re-scan Result: SUCCESS
     DIS: db_dis(all)
        Result: SUCCESS
     DIS: Ingester1(all)
        Result: SUCCESS

Caveats

This method makes a best-effort attempt to delete everything on all appropriate Data Import Servers. This cannot be atomic, so the operation might have only partial success. Make sure you check all the results.
The truncated partitions are marked as permanently truncated, and further ingestion of data will be disallowed. This is to prevent confusion if loggers produce new data for the partition, or if tailers have not finished all existing data files.
Before logging new data for the truncated partitions, remove any existing data files (bin files), and then delete the partitions with dhctl intraday delete ....
It is possible to delete data on one Data Import Server and leave it on another (e.g., a backup). Be extremely careful with this, as it can create confusion.