How to extract a Data Import Server from the data routing file

Additional Data Import Servers (DIS) are most easily managed as separate configuration items. This feature was introduced in Deephaven 1.20231218, and any Deephaven system created before then might have additional DIS configurations in the main data routing file. When a DIS configuration is mentioned elsewhere in the data routing file (such as in the tableDataServices section), removing the DIS from the routing file and adding it as an independent DIS in a single step is sometimes necessary.

This page has detailed instructions for extracting a DIS from the data routing configuration file so that it can be managed independently.

Back up the existing data routing configuration

The full data routing configuration now has two parts.

Before making any changes, back up the existing data routing configuration.

/usr/illumon/latest/bin/dhconfig routing export --file /tmp/routing.yml
/usr/illumon/latest/bin/dhconfig dis export --file /tmp/dises.yml

The second command does not include DISes defined in the main routing file, and it might print an error if there are no additional DIS configurations. This is normal, and it should still create an empty file named dises.yml.

Errors encountered exporting data import server configurations:
No data import server configurations were found to export

You can always restore this configuration with the following command:

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig dis import --clobber --ignore-errors --etcd --routing-file /tmp/routing.yml --file /tmp/dises.yml

Move the DIS from routing.yml to dises.yml

Make a copy of the backup files for safekeeping. In routing.yml, find the dataImportServers: section. Each key under this heading is a DIS configuration. Delete the entire configuration for the DIS you want to extract.

Export the DIS configuration you want to extract from the routing file. For example, to extract the default DIS, db_dis, use the following command:

/usr/illumon/latest/bin/dhconfig dis export --name db_dis --include-all-dises --file /tmp/exported_dis.yml

The default configuration for db_dis looks something like this:

---
db_dis:
  endpoint:
    serviceRegistry: none
    host: somehost.company.com
    tailerPort: 22021
    tableDataPort: 22015
  userIntradayDirectoryName: IntradayUser
  throttleKbps: -1
  webServerParameters:
    authenticationRequired: false
    port: 8086
    enabled: true
    sslRequired: false
  storage: default
  filters:
    - online: true
      namespaceSet: System
  properties:
    StringCacheHint.columnNameEquals_Exchange: ConcurrentUnboundedStringCache,CompressedString,50
    StringCacheHint.columnNameEquals_Parity: ConcurrentUnboundedStringCache,CompressedString,2
    StringCacheHint.columnNameEquals_SecurityType: ConcurrentUnboundedStringCache,CompressedString,9
    StringCacheHint.columnNameEquals_Sym: ConcurrentUnboundedStringCache,String,800000
    StringCacheHint.columnNameEquals_USym: ConcurrentUnboundedStringCache,String,20000
    StringCacheHint.default: ConcurrentBoundedStringCache,MappedCompressedString,1000000,2
    StringCacheHint.tableNameAndColumnNameEquals_QuoteArcaStock/Exchange: ConcurrentUnboundedStringCache,String,1

Note

Anchors (&anchor) and aliases (*alias) do not work across files. The command above replaces any aliases with the current defined values. You may also consider using a service registry endpoint instead of hardcoded values.

Import the updated configuration files as a unit

The following command will import the updated routing configuration and the extracted DIS configuration.

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig dis import --clobber --routing-file /tmp/routing.yml --file /tmp/dises.yml --file /tmp/exported_dis.yml

The whole configuration is evaluated as a single unit. If there are any errors, the import fails and the configuration is not changed.