Intraday control tool

`dhctl`

The command line tool dhctl allows you to truncate or delete Intraday partitions via the DIS. On any server, /usr/illumon/latest/bin/dhctl intraday subcommands can be used to truncate or delete intraday partitions.

Like the configuration tool, dhconfig, there is some built-in guidance. All fields and arguments can be abbreviated, as long as they remain unambiguous.

For example, to get help:

dhctl help
Usage: dhctl [intraday] [truncate|delete|help] [arguments]

dhctl intraday truncate --help
usage: dhctl intraday truncate [-d] [-e <arg>] [-h] [-i <arg>] [-k <arg> | --user <arg>] [-p <arg>] [-part <arg>] [-pf
       <arg>] [-s <arg>]
 -d,--dry-run                 print what actions would be performed without actually doing it
 -e,--exclude-dis <arg>       do not send requests to data import servers specified in an exclude parameter
 -h,--help                    print help for a intraday command
 -i,--include-dis <arg>       send requests only to data import servers specified in an include parameter
 -k,--key <arg>               specify a private key file to use for authentication
 -p,--password <arg>          specify a password for the given user
 -part,--partitions <arg>     partition to act on (all internal partitions), as namespace.tableName.columnPartition
 -pf,--pwfile <arg>           specify a file containing the base64 encoded password for the given user
 -s,--singlePartition <arg>   single partition to act on, as namespace.tableName.internalPartition.columnPartition
    --user <arg>              specify a user for authentication

truncate removes data at the specified partition(s) and disables further data ingestion.
Once truncated, partitions may be deleted with the delete action, and then new data can be ingested.

Deleting intraday data happens in two steps:

The first step is to truncate a location. This removes the data and marks the partition as permanently truncated. Any tailers for that partition will be disconnected, and any future attempt to tail data for that partition will be rejected.
After a partition has been truncated, you may prepare to accept new data into the same partition with the delete command. The delete command is permitted only when all the target partitions have been truncated.

If you want to append new data to the same location (with the same internal partition value), you will need to first truncate the partition, then remove any intraday log (.bin) files the tailer would send for that partition, and then delete the partition. At that point, the system is prepared to accept new data for the location.

The intraday control tool uses the configured data routing to locate the Data Import Server or Servers responsible for the data specified on the command line. All applicable servers will be instructed to truncate or delete the data location(s) unless they are excluded. Any DIS listed in an --exclude-dis argument will be skipped. If any DIS is included with a --include-dis argument, then any DIS not explicitly included will be skipped.

Authentication is required to remove data. dhctl will attempt to use the iris user default private key for authentication if it is readable. In most installations, if you invoke dhctl with sudo -u irisadmin dhctl the tool will automatically authenticate as the iris user. Otherwise, you need to provide a username and password, or the location of an authorized private key file.

One of --singlePartition or --partitions must be specified:

singlePartition removes a single data directory.
partitions removes the data directories for all internal partition values that match the key.

Examples

Any invocation may be a “dry run”, in which case the result will be a list of what the command would have done.

`truncate`

This command is a dry run truncating all internal partitions for 2021-07-07, excluding a backup dis.

$ dhctl intraday truncate --user iris --password iris --exclude-dis db_dis_backup --partitions DbInternal.ProcessEventLog.2021-07-07 --dry-run
Authenticating connection using user and password…
...

DIS: db_dis_backup is excluded by command line parameter
DIS: db_dis_backup
    Result: SKIPPED
    Location results: 0
DIS: db_dis
    Result: SUCCESS
    Location results: 3
        {key=DbInternal.ProcessEventLog.I.db_query_server_servername-bhs-vm_int_illumon_com.2021-07-07, dir=/db/Intraday/DbInternal/ProcessEventLog/db_query_server_servername-bhs-vm_int_illumon_com/2021-07-07/ProcessEventLog, message=Dry run, result=DRY_RUN, hasActiveProcessor=true}
        {key=DbInternal.ProcessEventLog.I.servername-bhs-vm_int_illumon_com.2021-07-07, dir=/db/Intraday/DbInternal/ProcessEventLog/servername-bhs-vm_int_illumon_com/2021-07-07/ProcessEventLog, message=Dry run, result=DRY_RUN, hasActiveProcessor=true}
        {key=DbInternal.ProcessEventLog.I.db_merge_server_servername-bhs-vm_int_illumon_com.2021-07-07, dir=/db/Intraday/DbInternal/ProcessEventLog/db_merge_server_servername-bhs-vm_int_illumon_com/2021-07-07/ProcessEventLog, message=Dry run, result=DRY_RUN, hasActiveProcessor=true}

This command is similar to the previous, but truncates only a single location, having internal partition value "db_query_server_servername-bhs-vm_int_illumon_com".

$ dhctl intraday truncate --user iris --password iris --exclude-dis db_dis_backup --singlePartition DbInternal.ProcessEventLog.db_query_server_servername-bhs-vm_int_illumon_com.2021-07-07 --dry-run
Authenticating connection using user and password…
...

DIS: db_dis_backup is excluded by command line parameter
DIS: db_dis_backup
    Result: SKIPPED
    Location results: 0
DIS: db_dis
    Result: SUCCESS
    Location results: 1
        {key=DbInternal.ProcessEventLog.I.db_query_server_servername-bhs-vm_int_illumon_com.2021-07-07, dir=/db/Intraday/DbInternal/ProcessEventLog/db_query_server_servername-bhs-vm_int_illumon_com/2021-07-07/ProcessEventLog, message=Dry run, result=DRY_RUN, hasActiveProcessor=true}

Note that in these examples, the results include hasActiveProcessor=true. A value of true here indicates that the partition has an active tailer, so it’s likely this partition is not ready to be truncated yet.

`delete`

The delete command is similar. The delete will not proceed unless all partitions on all Import Servers will succeed, and the dry run output indicates this.

$dhctl intraday delete --user iris --password iris --exclude-dis db_dis_backup --partitions DbInternal.ProcessEventLog.2021-07-07 --dry-run
...
DIS: db_dis2
    Result: SKIPPED
    Location results: 0
DIS: db_dis
    Result: FAILURE
    Location results: 3
        {key=DbInternal.ProcessEventLog.I.db_query_server_servername-bhs-vm_int_illumon_com.2021-07-07, dir=/db/Intraday/DbInternal/ProcessEventLog/db_query_server_servername-bhs-vm_int_illumon_com/2021-07-07/ProcessEventLog, message=Location has not been truncated, result=FAILED, hasActiveProcessor=true}
        {key=DbInternal.ProcessEventLog.I.servername-bhs-vm_int_illumon_com.2021-07-07, dir=/db/Intraday/DbInternal/ProcessEventLog/servername-bhs-vm_int_illumon_com/2021-07-07/ProcessEventLog, message=Location has not been truncated, result=FAILED, hasActiveProcessor=true}
        {key=DbInternal.ProcessEventLog.I.db_merge_server_servername-bhs-vm_int_illumon_com.2021-07-07, dir=/db/Intraday/DbInternal/ProcessEventLog/db_merge_server_servername-bhs-vm_int_illumon_com/2021-07-07/ProcessEventLog, message=Location has not been truncated, result=FAILED, hasActiveProcessor=true}

Scripting API

This functionality is available from any worker with sufficient permissions.

Full control over parameters is provided through a builder, described below. For simplicity, some simple options can be invoked via helper methods. These methods do not allow for dry runs, or selection of which Data Import Servers are included.

import io.deephaven.configuration.IntradayControlImpl

// truncate the location(s) specified by the key
IntradayControlImpl.truncateIntradayPartition(FullTableLocationKey)

// truncate the partitions for this table and column partition value (like --partitions)
IntradayControlImpl.truncateIntradayPartition(namespace, tableName, columnPartitionValue)

// truncate the single partition for this table and partition values (like --singlePartition)
IntradayControlImpl.truncateIntradayPartition(namespace, tableName, internalPartitionValue, columnPartitionValue)

// truncate partition(s) as indicated by the options argument (see Options Builder below)
IntradayControlImpl.truncateIntradayPartition(options)

// delete the location(s) specified by the key
IntradayControlImpl.deleteIntradayPartition(FullTableLocationKey)

// delete the partitions for this table and column partition value (like --partitions)
IntradayControlImpl.deleteIntradayPartition(namespace, tableName, columnPartitionValue)

// delete the single partition for this table and partition values (like --singlePartition)
IntradayControlImpl.deleteIntradayPartition(namespace, tableName, internalPartitionValue, columnPartitionValue)

// delete partition(s) as indicated by the options argument (see Options Builder below)
IntradayControlImpl.deleteIntradayPartition(options)

Note

You will want to save or print the results of these commands:

result = IntradayControlImpl.truncateIntradayPartition(...)
println result

println IntradayControlImpl.truncateIntradayPartition(...)

Options Builder

import io.deephaven.configuration.IntradayControlImpl.Options
import io.deephaven.configuration.IntradayControlImpl
builder = Options.builder()

Modify the builder with the desired options, much like the dhctl command line options.

Set the partition to be deleted:

builder.key("namespace", "tableName", "columnPartition")
builder.key("namespace", "tableName", "internalPartition", "columnPartition")
builder.key(key)

For example:

key = new FullTableLocationKey.AggregateTableLocationKey(“DbInternal”, “ProcessEventLog”,  SYSTEM_INTRADAY, lastBusinessDateNy())
builder.key(key)

builder.key(“DbInternal”, “ProcessEventLog”, lastBusinessDateNy())

Note

Only one key is permitted at this time. You can call one of the key() methods again, but you must call builder.clearKey() in between.

Dry run options

Change the dry run option:

builder.dryRun()     // delegates to builder.dryRun(true)
builder.dryRun(true) // perform a dry run, and do not perform the truncate or delete action

Authentication

Change the authentication (by default, the command will be run using the default authentication of the worker):

builder.authenticate("username", "password")
builder.authenticate("username", "password", "operateAsUser")
builder.authenticate("path_to_keyfile") // keyfile must be readable by the worker
builder.authenticate() // do default authentication, according to process environment and properties

Add a DIS

Add a DIS to the include or exclude list:

builder.include("dis_name")
builder.exclude("dis_name")

Build options

Build the options as configured in the builder. You may call build() multiple times. This allows you to use a builder to check a dry run and then perform the delete, or to change the key in a loop.

opts = builder.build()

You can invoke the truncate or delete methods directly from the builder or options:

result = builder.doTruncate()
result = opts.doTruncate()
println result
result = builder.doDelete()
result = opts.doDelete()
println result

You can pass the options to the Intraday Control tool and check the results:

result = IntradayControlImpl.truncateIntradayPartition(opts)
println result

You can display the contents of the builder or options:

println builder
println opts

Caveats

This method makes a best effort to delete everything on all appropriate Data Import Servers. This cannot be atomic, so the operation might have only partial success. Make sure you check all the results.
The truncated partitions are marked as permanently truncated, and further ingestion of data will be disallowed. This is to prevent confusion if loggers produce new data for the partition, or if tailers have not finished all existing data files.
Before logging new data for the truncated partitions, remove any existing data files (bin files), and then delete the partitions with dhctl intraday delete ....
It is possible to delete data on one Data Import Server, and leave it on another (e.g., a backup). Be extremely careful with this, because it creates an opportunity for confusion.