Configuration overview

The majority of Deephaven server configuration is managed by the Configuration Server. The Configuration Server provides a single method of access to various configuration data for all Deephaven server services. For its underlying data store, the Configuration Server uses etcd. etcd is a distributed property store with built-in high availability and fault tolerance features.

File locations

Most Deephaven server configuration is in the following locations:

1. Configuration Server This is a service backed by etcd that provides up-to-date configuration information of the following types:

   • Properties files
   • Data routing
   • Schemas and table definitions

2. /etc/sysconfig/deephaven This is where the Deephaven installation-specific files reside. When a new installation RPM is applied, either for a patch update, or for a new release, configuration files are copied to a new, version-specific directory under this path. This allows preservation of previous versions of the configuration files. To maintain backwards-compatibility with earlier configuration paths, /etc/sysconfig/illumon.d and /etc/sysconfig/illumon.confs are links to the latest versions of paths under /etc/sysconfig/deephaven. Of particular interest:

   • /etc/sysconfig/illumon.confs.<version>/illumon.iris.hostconfig - This also may be found via the soft-link /etc/sysconfig/illumon. This file controls the environment for each Deephaven server-side process (run via /usr/illumon/latest/bin/iris <service name>). For example, see Adding JAR Directories to add customer JAR directories to the Deephaven installation with this file.
   • /etc/sysconfig/illumon.d/resources - Configuration files for each Deephaven server-side process (referenced by the hostconfig).
   • /etc/sysconfig/illumon.d/schema - Database schema files (XML). This used to be the recommended location for customer database schema files. Schemas are now stored in etcd via the configuration services.
   • /etc/sysconfig/illumon.d/java_lib - Custom user and third-party Java libraries. "Extension" libraries placed here will be found on the classpath.
   • /etc/sysconfig/illumon.d/override - This path contains any class files that should override those contained in java_lib and hotfix JARs.
   • /etc/sysconfig/illumon.d/hotfixes - This path contains any JAR files that should override those contained in the java_lib directories.

3. /usr/illumon This is where Deephaven system files reside. These are not meant to be modified except when a new release occurs. The current release is referenced by the soft-link /usr/illumon/latest. While it should not be necessary to modify these files, it may be useful to be aware of the following:

   • /usr/illumon/latest/etc - System-wide internal configuration and defaults.
   • /usr/illumon/latest/java_lib - Deephaven system Java libraries. Deephaven uses several formats of configuration data to control its operation. For core Deephaven services, there are three main forms of configuration, as detailed below.

File formats

1. Java properties files These are text files, usually named with a .prop extension, that store key-value pairs. These are the largest stores of Deephaven configuration data, and control most of its operations and behavior. Note that properties files are accessed via the configuration service and not from the filesystem, so any files on disk are just for reference and are not authoritative.

   There are four main properties files in a typical Deephaven installation:

   • iris-common.prop - In previous releases, this was the main configuration file for all installation-specific settings. In this release, most of the contents of iris-common.prop have been moved to iris-environment.prop, and iris-common.prop remains as a well-known property file for use by existing services. All settings for iris-common.prop come from inclusions of other property files.
   • iris-environment.prop - contains the majority of installation-specific settings.
   • iris-endpoints.prop - defines server names and addresses, service host points, and ports for Deephaven services in the installation.
   • iris-defaults.prop - defines common settings needed for all Deephaven installations. These properties may be changed by a new version of Deephaven and should not be changed by the customer. Any environment-specific alterations should be placed in iris-environment.prop or iris-endpoints.prop, where they will override the defaults.

2. XML files Some Deephaven processes and utilities use eXtensible Markup Language files. XML is a text format, but with specific requirements for delimitation of data elements, and with support for hierarchical representations of information.

3. YAML files All Deephaven installations have a YAML file for configuration of the Data Routing Service. This file centralizes the information governing the locations, servers, and services that determine how data is handled. The data routing configuration is accessed via the configuration service, and not from the filesystem.

   Systems running the Envoy proxy will usually also have a .yaml file to configure this service.

4. Other configuration files These are also text files, but their format is specified by the applications that use them. The main example for Deephaven of this type of file is the hostconfig file that configures process startup for services. In this case, the file is actually a shell script.

There are also other configuration settings and other uses of the principal configuration files in Deephaven. The details of these are described in the following sections.

Deephaven Configuration Server

The Configuration Server provides a single method of access to various configuration data for all Deephaven server services. For its underlying data store, the Configuration Server uses etcd. etcd is a distributed property store with built-in high availability and fault tolerance features.

Accessing configuration through this service centralizes the management and distribution of configuration information and enables more control over security and authorization.

The configuration server encapsulates:

• Schemas and table definitions
• Properties and properties files
• Data routing configuration

Connecting to the Configuration Server

The configuration server provides access to the properties for Deephaven processes, so these cannot be used to configure access to the configuration server itself. Deephaven processes will use environment variables, system properties set when launching, or files on disk to bootstrap this access (as clients of the configuration server).

Configuring the Configuration Server

The Configuration Server process will necessarily get its configuration information from another source. This will be the properties files stored in etcd by default.

The server will be initialized with the following properties:

• configuration.server.port - the service will bind to this port.
• configuration.server.ssl - the service will be secured by TLS unless this is set to false.

If ssl is not disabled:

• configuration.server.cacertfile - file containing the server certificate
• configuration.server.keyfile - file containing the server private key, or the PKCS12 keystore.
• configuration.server.key_alias - alias for the key; if it is stored in a keystore, the default is configuration_server.

The password for the key file is located similar to the authentication server.

• CONFIGURATION_SERVER_PASSPHRASE
• configuration.server.passphrase.file
• configuration.server.passphrase

If the password is not defined, it is assumed that the certificate and key files are PEM format and are not password protected.

If the password is defined, the key file is expected to be a PKCS12 keystore file protected by that password. The certificate and key named by the configured alias will be extracted from this file. The cert file is not used in this case.

Client Configuration

Clients find a configuration service using system options in the following order. We expect option 3 to be the active option in most cases on the Deephaven servers, and option 2 for most remote clients (e.g., the Swing console).

Note that the properties are System Properties (e.g., set using the -D java option when starting a process) and cannot be set in the configuration service or properties files.

1. If the system property dh.config.client.host exists:

   • Host is taken from system property dh.config.client.host.
   • Port is taken from system property dh.config.client.port.
   • If TLS is used, the trusted certificate file is taken from system property dh.config.client.cacert (if dh.config.client.cacert is not set, ssl will not be used).
   • If mutually authenticated TLS is used:
     • the client certificate file is taken from system property dh.config.client.cert.
     • the client private key file is taken from system property dh.config.client.key. This key must be in PEM format without password protection.
     • Note that this option does not allow multiple servers to be specified.

2. If the system property dh.config.client.bootstrap exists, the value must specify a directory containing one or more subdirectories, each containing files as follows:

   • host contains the address of the host. This should be an IP address or fully qualified domain name. The service’s certificate should contain this value in the SAN field.
   • port contains the network port for the service.
   • cacert is a PEM formatted file containing the certificate trust chain for the server certificate (often just the self-signed certificate). This file may contain one or many trusted certificates.
   • cert (optional) contains the client certificate for mutual authentication.
   • key (optional) contains the client private key to be used in mutual authentication. Must be PKCS8/PEM format with no password.

• The bootstrap directory may optionally contain:
  • authority contains a string to be used for hostname verification on all servers. This string must be in the common name or subject alternative name of all the server certificates. “configuration-server” is the default if nothing is specified.
  • If a custom authority is not configured, the host value for each server will be used.
  • This option supports configuration of multiple servers. These will be treated as equivalent, and automatically provide redundancy to clients.
  • All servers must be configured the same way (see “Equivalent” configuration below).

3. The default value for dh.config.client.bootstrap is /etc/sysconfig/illumon.d/dh-config/clients. If this location exists, it will be parsed as above.
4. The configuration of localhost:22023 with cacert path /etc/sysconfig/illumon.d/resources/truststore-iris.pem.

“Equivalent” configuration

• When multiple configuration servers are specified, they must match.
• All must be configured for TLS, or none.
• If an authority is defined (via the authority file, all the host certificates must contain that value in the SAN field; e.g., DNS:configuration-server.
• Mutual authentication is not supported when there are multiple servers.
• The cacert files specified may be independent and will effectively be concatenated by the client.

Viewing and changing configuration

Configuration information is not stored in files on disk. Tools are available to export configuration where it can be viewed and edited. Any changes need to be imported again before they are effective.

Properties files

To edit a properties file you will need to export the file, make edits, and import the changed file. The example below uses an arbitrary directory (/tmp/propfiles), where the executing user has write permission, and irisadmin has read permission.

For example:

/usr/illumon/latest/bin/dhconfig properties export iris-environment.prop --directory /tmp/propfiles

vi /tmp/propfiles/iris-environment.prop

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import /tmp/propfiles/iris-environment.prop

Data Routing

To edit the system’s data routing configuration, you will need to export the configuration to a file, edit the file, and then update the system.

/usr/illumon/latest/bin/dhconfig routing export --file /tmp/routing.yml

vi /tmp/routing.yml

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig routing import --file /tmp/routing.yml

Schemas

See Deploying Schemas to Deephaven for details.

Properties Files

Properties files, or "prop" files, are Java constructs that provide applications with easy access to a collection of settings. Most Deephaven processes use the unified properties file iris-common.prop. This makes it easier to find the specific setting you need, and to synchronize processes that need to share a configuration value (e.g., a port number). It is managed by the Configuration Server process and stored in etcd.

Changing a setting in a property file that is managed by the Configuration Server requires first exporting its contents to file, editing the file, and then reimporting it. The main tool for this is dhconfig.

dhconfig can export one or all properties files, and can also import individual properties files.

Examples

To export all properties files to a props directory under the current path. This props directory must already exist:

/usr/illumon/latest/bin/dhconfig properties export --directory ./props

To export iris-environment.prop to /tmp/props:

/usr/illumon/latest/bin/dhconfig properties export iris-environment.prop --directory /tmp/props

To import iris-environment.prop from the current directory:

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import iris-environment.prop

Once a properties file has been exported, edited, and reimported, the service(s) that will use the changed properties can be restarted or reloaded to pick up the changes.

Properties files syntax

Properties files contain key=value settings, comments (starting with a pound/hash sign #) and some Deephaven extensions. Deephaven's extensions of the standard Java properties file format allows for scoped stanzas of configuration that will only apply to certain processes. Deephaven's includefiles directive allows other files to be included in a properties file.

The following demonstrates a scoped section for the Access Control List write server, which needs to use a user account with higher permissions than the standard ACL access credentials:

[service.name=db_acl_write_server|iris_db_user_mod] {
  # Override the standard credentials and use a login with write permissions
  MysqlDbAclProvider.user=aclwriter
  MysqlDbAclProvider.passwordFile=acl_write_passphrase.txt
}

Deephaven's configuration file parser includes a set of language constructs, which we will explain here. Users may want to write their own properties files - for example, depending on the worker they are using - or create custom entries for certain processes. The existing file set up includes:

• Scope definitions
• Scope openers
• Scope closers
• The includefiles keyword
• The final keyword
• The finalize keyword

Scope definitions

A scope definition establishes the conditions that must exist for the configuration parser to read the subsequent lines until the end of the scoped region. A scope definition is of the form:

[property1.name=value1{|value2|...valueN}{, property2.name=valueX{|valueY…}}]

The minimum scope declaration is [property.name=value]. In this case, only if the system has a defined system configuration property with name property.name and a value equal to value will the subsequent lines be parsed.

• It is possible to include multiple values for a given property, or to include multiple properties. To include multiple values, use the vertical pipe character (|) to distinguish between the different values. Note that this is not escaped; currently it is not possible to use property values with a vertical pipe in them for scoping purposes. This is an or operation; as long as at least one of the specified values matches the value of the specified system configuration property, the scope is considered to match the context.

  `[property1.name=value1|value2] → "Property1.name equals value1 OR value2."`
  

• It is also possible to include multiple property names within one scope. This is an and operation; each property name within a scope declaration must match at least one value from the defined list of available values. Undefined properties are considered to never match any scope; a property with a defined value of empty-string is also not permitted.

  `[property1.name=value1|value2,property2.name=valueX] → "property1.name equals value1 or value2, and property2.name equals valueX"`
  

• Whitespace is ignored for property names, and is ignored on the left side of values, but trailing whitespace is not ignored; standard Java properties do accept trailing whitespace, so this definition does as well.

  [        property1.name    =  value1  ]  →  "property1.name" = "value1  "
  

• A scope opener may be placed immediately after a scope definition on the same line. Anything else immediately after a scope definition is an invalid format.

  `[property1.name=value1] {`
  

Scope openers

A scope opener is the open-curly-brace character: {. A scope opener defines the block of lines that will be read or ignored depending on whether or not the scope definition is matched. A scope opener may be placed on the same line as a scope definition, immediately after the scope definition (ignoring whitespace), or may be placed onto a line of its own.

A scope opener that does not follow a scope declaration is an invalid format. As { by itself is not a valid Java property declaration, this is consistent with the Java property specification, and no potential Java properties are thereby excluded.

A scope opener with no subsequent scope closer is an invalid format. Each scope section must be properly closed.

Scope closers

A scope closer is the end-curly-brace character: }. A scope closer defines the end of a block of lines that will be read or ignored depending on whether or not the scope definition is matched. A scope closer must be placed on a line by itself, except in the trivial case of a scope opener followed immediately by a scope closer with no declarations in between (in which case no action can happen and there is no point to making such a section).

A scope closer that does not follow a scope opener is an invalid format. As } by itself is not a valid Java property declaration, this is consistent with the Java property specification, and no potential Java properties are thereby excluded.

Anything following a scope closer will be parsed normally, including new property declarations or scope definitions.

The includefiles keyword

Deephaven property files may use the includefiles keyword to indicate that the contents of some other file should be included into the properties generated by the current file. iris-environment.prop includes a reference of this sort to the iris-defaults.prop file, which contains default values for most Deephaven properties. The values from the current file will be considered to override the values from the included file, if both files include a property with the same name. includefiles explicitly must be the first non-comment, non-whitespace line in the .prop file, and may not be included in a scope, as shown in the example below:

includefiles=iris-defaults.prop,my-settings.prop

...

########## Iris DB RemoteQueryDispatcher settings ##########

RemoteQueryUtils.hostnameSuffixesToTruncateForDisplay=

RemoteQueryDispatcherParameters.name=default
RemoteQueryDispatcherParameters.host=localhost
RemoteQueryDispatcherParameters.queryPort=22013
RemoteQueryDispatcher.workerPort=22012
RemoteQueryDispatcher.workerServerPorts=23000-23999
RemoteQueryDispatcher.webPort=8084

Note that everything before the first equals sign is the key name, and everything after it is the value. If a value is a list, it is up to the application to define how that list should be formatted. Some Deephaven lists (such as tailer configuration file names and includefiles names) are comma-delimited, while others (such as schema paths) are semi-colon delimited.

A property name without an assigned value (such as the RemoteQueryUtils.hostnameSuffixesToTruncateForDisplay property above) will be treated as defining the property to have an empty value (i.e., an empty string). This is not the same as if the property does not exist in the properties files.

Only one includefiles directive is allowed in a properties file, but it can list multiple files to include, and these can, in turn, include other files.

There is no uniqueness requirement when adding a setting into a properties file. Instead, the properties collection that applications use within Java provides a "last in, wins" view of the settings. If a value is set multiple times, in a single properties file, or in multiple properties files linked with includefiles, whichever setting is applied last is the one that is used. Properties files are evaluated from top to bottom. Properties file entries are always case-sensitive for the property name, and often case-sensitive for the value.

The iris-defaults.prop properties file is delivered with the Deephaven installation. It provides default values for settings that can be changed, but rarely need to be, and also provides initial values for properties that will most likely need to be updated in production environments. These latter properties might work in a single-machine test environment, but need to be set differently when Deephaven is scaled to multiple servers. As mentioned above, the recommendation is to have a custom properties file (or, potentially, multiple custom properties files for different services, servers, or users) that includes iris-defaults.prop at the top, and then has its own specific settings further down that add to or override the inherited settings.

Java also allows for properties to be set for a process by passing them as JVM arguments. These are of the form:

-DpropertyName=propertyValue

Settings using -D are applied after properties file(s) settings, so they have the highest priority and override settings taken from properties files.

The properties file that a Deephaven process will load is set with a -D property as well:

-DConfiguration.rootFile=myDeephavenSettings.prop

The final keyword

A new language element is the final keyword. This keyword establishes that the immediately subsequent property declaration on the same line may not be further modified, whether within the same file or by some file that includes the file in which the final keyword appeared. This can be important if there are properties which are important to keep consistent between processes, ensuring that a key property cannot be overridden in a scope section or subsequent file.

The final keyword is used in the following form:

final <declaration>

The <declaration> is a standard Java property declaration of the form property.name{=value}. If no value is specified, then the property is assigned the value of empty-string. Anything after the final keyword is assumed to be part of the Java property declaration, so no other commands may be included on the same line as a final keyword.

Using the final keyword without a subsequent declaration is an invalid format; some property must exist to be declared final.

Note that the final keyword means that no property may be declared with the exact name 'final', with or without a value.

The finalize keyword

A new language element is the finalize keyword. This keyword establishes that the immediately subsequent property name may not be further modified, as though that property had been declared final. This can be useful if a property may have different values in different scopes, but after the sections have been parsed, whatever state the property is in should not be further changed.

The finalize keyword is used in the following form:

finalize <property.name>{, <property2.name>}

The <property.name> is a standard Java property name. As Java property names do not have leading or trailing whitespace, any whitespace around the property name will be trimmed. If the property being finalized did not previously exist, it is not created, and is not able to be created by anything later. This is a key difference from the use of the final keyword.

Note that everything on the line after the finalize keyword is treated as property names; no other commands may follow a use of the finalize keyword. The finalize keyword means that no property may be declared with the exact name 'finalize', with or without a value.

Using the finalize keyword without a subsequent property name is an invalid format; some property must be specified to be finalized.

The Property Inspector utility

The Property Inspector class in the Deephaven Util JAR can inspect properties files or chains of properties files to provide visibility into what properties files are being included, where properties are set, and where properties are duplicated or overridden.

On a Deephaven server, this utility can be executed using iris_exec, as follows:

/usr/illumon/latest/bin/iris_exec property_inspector

Example

The command below searches for all properties whose names start with IrisLogDefaults. Note: the "--" is required. It separates arguments to iris_exec from arguments to property_inspector.

/usr/illumon/latest/bin/iris_exec property_inspector -- -pf iris-console.prop -pn IrisLogDefaults.*

When the command above is run, the following results are generated (edited for brevity/clarity):

Searching for: IrisLogDefaults.*, starting with file: "iris-console.prop"

Name: IrisLogDefaults.useMainClassNameForLogs Value: true --- Line: 562 --- File: iris-defaults.prop --- Scope: []
Name: IrisLogDefaults.logLevel Value: INFO --- Line: 563 --- File: iris-defaults.prop --- Scope: []
Name: IrisLogDefaults.logType Value: STANDARD_LOG_CREATOR --- Line: 564 --- File: iris-defaults.prop --- Scope: []
Name: IrisLogDefaults.writeDatabaseProcessLogs Value: false --- Line: 565 --- File: iris-defaults.prop --- Scope: []
Name: IrisLogDefaults.writeDatabaseAuditLogs Value: false --- Line: 566 --- File: iris-defaults.prop --- Scope: []
Name: IrisLogDefaults.captureLog4j Value: true --- Line: 567 --- File: iris-defaults.prop --- Scope: []
Name: IrisLogDefaults.captureSysout Value: false --- Line: 568 --- File: iris-defaults.prop --- Scope: []
Name: IrisLogDefaults.captureSyserr Value: false --- Line: 569 --- File: iris-defaults.prop --- Scope: []
Name: IrisLogDefaults.useLogAggregatorService Value: false --- Line: 570 --- File: iris-defaults.prop --- Scope: []
Name: IrisLogDefaults.useDynamicPartitions Value: false --- Line: 571 --- File: iris-defaults.prop --- Scope: []
Name: IrisLogDefaults.aliveMessageSeconds Value: 0 --- Line: 572 --- File: iris-defaults.prop --- Scope: []
Name: IrisLogDefaults.binaryLogTimeZone Value: --- Line: 573 --- File: iris-defaults.prop --- Scope: []

Configuration files

Configuration files are text files whose format varies depending on the requirements of the applications that use them. The main use of conf files in Deephaven is the hostconfig file, which is used in all configurations.

hostconfig

The hostconfig file is normally located under /etc/sysconfig/illumon.confs, and named illumon.iris.hostconfig. The hostconfig file is sourced (run) by another bash script, which means it supports bash scripting to make if/then and case statement type decisions. This file provides information needed to start and monitor Deephaven processes. It contains details that must be passed to the processes as they are started, such as how much memory should be allocated, what account they should run under, and what properties file they should use.

Whenever a Deephaven service is started by Monit, it uses the Deephaven launch script to start the service, which sources the hostconfig file. For more information on Monit Services, see the Monit Manual.

For example, the following is a section of a hostconfig file that configures the Deephaven Controller process:

iris_controller)
   CONFIGFILE=iris-common.prop
   EXTRA_ARGS="-j -Xms4G -j -Xmx4G -j -Xmn32m -j -Dservice.name=iris_controller"
   RUN_AS=irisadmin
   WORKSPACE=/db/TempFiles/$RUN_AS/$proc
   ;;

In this case, the process will be directed to use the iris-common.prop properties file, run under the irisadmin account, and be allocated 4GB of initial, and maximum, heap space, with 32MB of Eden space. In the EXTRA_ARGS line, the -j indicates to the Deephaven scripts that these are parameters to be passed through directly to the JVM.

If needed, additional processes to be managed by Monit can be added to the hostconfig file. If an existing process needs changes to allocated memory or other startup properties, those changes must be made in this file. See the Deephaven Operations Guide for additional documentation on Deephaven's use of Monit.

Other Configuration Files

Most other configuration files in Deephaven are text files that contain lists of values for various purposes:

• db_rqc_default_class_push_list.txt
• default_static_imports.txt
• default_class_imports.txt
• default_package_imports.txt
• authusers.txt
• superusers.txt
• dsakeys.txt
• db_acl_write_server_passphrase.txt
• db_acl_ro_passphrase.txt

db_rqc_default_class_push_list.txt

This is the default file for listing classes that exist in the classpath of the client system and should be pushed to the class cache of the query worker at startup. An alternate file for this purpose can be specified by setting the property RemoteQueryClient.defaultClassPushList.

In general, classes pushed from this file will be classes that the user is developing and needs to test, or classes that only one or a very few users need. Production classes that are to be usable by most users should be deployed to query servers by building them into JARs and placing these JARs under /etc/sysconfig/illumon.d/java_lib.

Default files

These are the default files that contain a list of Java classes that are added to the query library used by Deephaven queries:

• default_static_imports.txt
• default_class_imports.txt
• default_package_imports.txt

These file names can be overridden by the properties:

• IrisDB.queryLibrary.defaultPackageImportList=
• IrisDB.queryLibrary.defaultClassImportList=
• IrisDB.queryLibrary.defaultStaticImportList=

authusers.txt

/etc/sysconfig/illumon.d/resources/authusers.txt includes the set of Deephaven users, groups and ACLs are stored in MySQL database. Customers can choose to use LDAP for storing user credentials. In some situations, customers may wish to maintain the set of Deephaven users in a local file by setting the property:

authentication.server.localusers.enabled=true

In this case, the authusers.txt file contains the list of Deephaven users in htpasswd format.

superusers.txt

/etc/sysconfig/illumon.d/resources/superusers.txt lists the local superusers, and users allowed to operate as other users in the format <username>:<operate as user>. A * in the operate-as-user indicates that the username can operate as any user. This file is only used if the following property is set:

authentication.server.localsuperusers.enabled=true

The file name can be changed with the following property:

authentication.server.localsuperusers.file=<filename>

dsakeys.txt

The /etc/sysconfig/illumon.d/resources/dsakeys.txt file lists the DSA keys for Deephaven users as long as the property is set as follows:

authentication.server.authorizedkeys.enabled=true

DSA keys are used in place of passwords for non-interactive users in scripts or batch jobs or delegated API calls. See: Database Authentication to learn more.

This filename can be overridden by the property:

authentication.server.authorizedkeys.file=...

Passphrase files

db_acl_write_server_passphrase.txt and db_acl_ro_passphrase.txt are default files that contain the passwords for the MySQL database, used by Deephaven's internal processes when reading or writing user or ACL information.

YAML files

YAML is a human-readable data serialization language commonly used for configuration files. In Deephaven, a YAML file (which use the extension .yml) is used to configure the Data Routing Service.

XML files

XML (eXtensible Markup Language) files are also text files, and nominally human-readable, but, unlike configuration files and properties files, XML files include more requirements around how data is formatted and delimited. The main structure of XML files is based around elements, which can be single line or multi-line, and which can encapsulate other elements to form hierarchies. Elements can also contain embedded text data directly as content and/or in the form of named attributes.

The main places where XML is used to store Deephaven configuration settings are:

• Tailer configuration files, which tell a tailer what files and paths it should monitor.
• Import/Merge script configuration files, which provide one way of automating frequently executed data import and merge operations.
• Schema files, each of which is an XML file without a root element.
• Workspace files, which persist the layout of a Deephaven client application (tabs, queries, etc).

Environment Variables

Environment variables are used to set some of the global configuration for Deephaven, most commonly for bin utilities, which rely on JAVA_HOME, and also for the import/merge scripts.

In order to use Deephaven tools like iriscat (/usr/illumon/latest/bin/iriscat), JAVA_HOME must first be set to point to an installation directory that contains a Java 8 installation. This path will include a bin subdirectory that contains a Java runtime executable. Typically, this will be set to /usr/java/latest or /usr/java/latest-64. On Linux, this can be checked by running:

export | grep JAVA_HOME

It can be set similarly with:

export JAVA_HOME=path_to_java

The environment variable CUSTOMER_JAR_DIR can be used to add locations in which Deephaven will search for customer JAR files.

See Adding JAR Directories for further details.

Configuration tools

Installation

Installation configuration tools are included under the /usr/illumon/latest/install directory. Most of these need to be run as root (with sudo).

• iris_keygen.sh
• config_packager.sh
• set_iris_endpoints_prop.sh
• config_import.sh

If iris_keygen.sh is re-run to update some aspect of an installation (e.g., to change the certificate used by the web_api_service), the config_packager.sh will need to be used to package the updated authentication data, which will then need to be copied to the other nodes of the cluster and unpackaged.

dhconfig

The dhconfig script in /usr/illumon/latest/bin is the primary tool for managing configuration in Deephaven. This tool handles schema, data routing, properties, and service registry configuration, and can display table checkpoint files, adding some features to crcat.

In the text below, /usr/illumon/latest/bin/dhconfig is shortened to dhconfig for clarity, as if /usr/illumon/latest/bin has been added to the shell PATH.

Usage:

dhconfig [properties|schemas|routing|checkpoint] [import|export|list|delete|validate|help] [arguments]

The first argument is a data type:

• properties
• schemas
• routing
• checkpoint

The second argument is an action (not all actions are valid for all data types):

• import
  Add or update configuration to the system
• export
  Export or print configuration
• list
  Show what configuration exists in the system
• delete
  Remove configuration from the system
• validate
  Check a configuration file for errors (i.e. prior to importing)
• help
  Print usage information, as specific as possible given the arguments provided

dhconfig is designed to support exploration and provide context sensitive help. Users can get command details from the command itself. For example, dhconfig help will print top level help, which includes data types and common actions, and dhconfig schemas help will print usage specific to schemas. All actions, data types, and arguments allow prefixes as long as they are not ambiguous, but help and usage information will always print the full canonical values.

Some common options for the command follow. Note that not all arguments are applicable in all contexts.

All data type/action commands have further options. More detailed usage is provided if help is requested or if the arguments are incomplete or invalid.

Logging

The dhconfig script creates a log file in /var/log/deephaven/misc if the current user has write permission there, and in /tmp if not. Actions are logged to the DbInternal AuditEventLog table.

Authentication

All commands support a common authentication framework.

In general, authentication is not required for operations that don't change the system configuration (i.e. export, list, validate). If authentication options are provided, they will be validated whether they are required or not.

If --etcd is specified, authentication is implied by file system permissions and the authentication arguments do not apply. dhconfig will need to be run as a privileged user when using --etcd.

* You can create a password file with this command: echo -n "your-password" | base64 > password.txt.
Be sure to protect the file with appropriate filesystem permissions.

As a convenience, dhconfig will attempt to use the default iris private key file if it is readable. Executing with sudo or sudo -u irisadmin will automatically authenticate by setting -DWAuthenticationClientManager.defaultPrivateKeyFile=/etc/sysconfig/illumon.d/auth/priv-iris.base64.txt.

dhconfig properties

The properties (or property) data type is used to work with properties files stored in the system. The following additional arguments are supported for some actions:

/usr/illumon/latest/bin/dhconfig properties export --directory /tmp

/usr/illumon/latest/bin/dhconfig properties export --directory /tmp --etcd

/usr/illumon/latest/bin/dhconfig properties export iris-environment.prop

/usr/illumon/latest/bin/dhconfig properties export iris-environment.prop > /tmp/my_properties.prop

/usr/illumon/latest/bin/dhconfig properties export --directory /tmp iris-environment.prop iris-endpoints.prop

/usr/illumon/latest/bin/dhconfig properties export --directory . --file iris-environment.prop --file iris-endpoints.prop

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import /tmp/iris-environment.prop
/usr/illumon/latest/bin/dhconfig properties import /tmp/iris-environment.prop --user iris --pass mypassword

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import --file /tmp/my_properties.prop --rename iris-environment.prop

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import /tmp/p1.prop /tmp/p2.prop /tmp/p3.prop

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import --directory /tmp p1.prop p2.prop p3.prop

/usr/illumon/latest/bin/dhconfig properties list

/usr/illumon/latest/bin/dhconfig properties validate /tmp/my_properties.prop

dhconfig routing

The routing (or datarouting) data type is used to work with the data routing configuration.

/usr/illumon/latest/bin/dhconfig routing export

/usr/illumon/latest/bin/dhconfig routing export --file /tmp/routing.yml

/usr/illumon/latest/bin/dhconfig routing export /tmp/routing.yml

/usr/illumon/latest/bin/dhconfig routing validate --file /tmp/routing.yml

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig routing import --file /tmp/routing.yml

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig routing import --file /tmp/routing.yml --etcd

dhconfig schemas

The schemas data type is used to manage schemas in the system. The following additional arguments are supported for some actions:

/usr/illumon/latest/bin/dhconfig schemas list --namespace DbInternal

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schemas list --namespaceset System --etcd

/usr/illumon/latest/bin/dhconfig schemas export --namespace DbInternal --directory /tmp/schemas

/usr/illumon/latest/bin/dhconfig schemas export DbInternal.ProcessEventLog

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schemas import --file /tmp/schemas/LearnDeephaven.StockQuotes.schema

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schemas import --file /tmp/schemas/LearnDeephaven.StockQuotes.schema --no-compile --lenient

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schemas import --etcd --file /tmp/schemas/LearnDeephaven.StockQuotes.schema

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schemas import --etcd /tmp/schemas/LearnDeephaven.StockQuotes.schema

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schemas import --directory /tmp/schemas --namespace ExampleNamespace --force

/usr/illumon/latest/bin/dhconfig schemas delete --file MyNamespace.MyTable1 --file MyNamespace.MyTable2 --force --user some-admin-user --pwfile ~/pw.txt

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schemas delete MyNamespace.MyTable1 MyNamespace.MyTable2 --force

dhconfig checkpoint

The checkpoint data type is used to examine table checkpoint files on disk. This adds some features to crcat. Authentication options do not apply to this command (the executing user must have read access to the files to be examined). Checkpoint files are named table.size by default. File arguments may be the checkpoint file itself, or the directory containing table.size.

The following additional arguments are supported for some actions:

/usr/illumon/latest/bin/dhconfig checkpoint list --file /db/path.to.table/table.size

/usr/illumon/latest/bin/dhconfig checkpoint list --file /db/Intraday/DbInternal/ProcessEventLog/some_partition/some_date/ProcessEventLog

/usr/illumon/latest/bin/dhconfig checkpoint list /db/Intraday/DbInternal/ProcessEventLog/some_partition/some_date/ProcessEventLog

/usr/illumon/latest/bin/dhconfig checkpoint list --file /tmp/any.file.name

/usr/illumon/latest/bin/dhconfig checkpoint list --directory /db/Intraday/DbInternal/ProcessEventLog

/usr/illumon/latest/bin/dhconfig checkpoint list /db/Intraday/DbInternal/ProcessEventLog

/usr/illumon/latest/bin/dhconfig checkpoint list --directory /db/Intraday/DbInternal/ProcessEventLog --csv

find /db/Intraday/DbInternal/ProcessEventLog -name table.size > /tmp/list.txt
/usr/illumon/latest/bin/dhconfig checkpoint list --csv --list /tmp/list.txt

$ /usr/illumon/latest/bin/dhconfig checkpoint list /db/Intraday/DbInternal/ProcessEventLog/db_query_server_qa-treasureplus-cluster-query-1_int_illumon_com/2022-07-08/ProcessEventLog/
Checkpoint file '/db/Intraday/DbInternal/ProcessEventLog/db_query_server_qa-treasureplus-cluster-query-1_int_illumon_com/2022-07-08/ProcessEventLog/table.size':

CheckpointRecord[
	version=2,
	TableLocationState[size=7015, lastModificationTime=2022-07-08T17:32:28.691-0400],
	DataFileSizeRecords[
		[name=AuthenticatedUser.dat, size=28060],
		[name=AuthenticatedUser.sym, size=8],
		[name=AuthenticatedUser.sym.bytes, size=4],
		[name=EffectiveUser.dat, size=28060],
		[name=EffectiveUser.sym, size=8],
		[name=EffectiveUser.sym.bytes, size=4],
		[name=Host.dat, size=28060],
		[name=Host.sym, size=8],
		[name=Host.sym.bytes, size=46],
		[name=Level.dat, size=28060],
		[name=Level.sym, size=40],
		[name=Level.sym.bytes, size=24],
		[name=LogEntry.bytes, size=1107823],
		[name=LogEntry.dat, size=56120],
		[name=Process.dat, size=28060],
		[name=Process.sym, size=32],
		[name=Process.sym.bytes, size=32],
		[name=Timestamp.dat, size=56120]
	],
	SourceFileSizeRecord[name=DbInternal.ProcessEventLog.System.db_query_server_qa-treasureplus-cluster-query-1_int_illumon_com.2022-07-08.bin.2022-07-08.170000.000-0400, size=80605],
	ImportState[
		class=class com.illumon.iris.db.tables.dataimport.logtailer.ImportStateRowCounter,
		details=[
			[nRows, 7015]
		]
	]
]

$ /usr/illumon/latest/bin/dhconfig checkpoint list /db/Intraday/DbInternal/ProcessEventLog --csv --header

filename, version, tableSize, lastModificationTime, lastModificationTimeStr, columnCount, sourceFile, sourceBytes
"/db/Intraday/DbInternal/ProcessEventLog/db_query_server_qa-treasureplus-cluster-query-1_int_illumon_com/2022-07-08/ProcessEventLog/table.size", 2, 33919, 1657316070141, 2022-07-08T21:34:30.141Z, "DbInternal.ProcessEventLog.System.db_query_server_qa-treasureplus-cluster-query-1_int_illumon_com.2022-07-08.bin.2022-07-08.210000.000+0000", 351464
"/db/Intraday/DbInternal/ProcessEventLog/db_merge_server_qa-treasureplus-cluster-infra-1_int_illumon_com/2022-07-08/ProcessEventLog/table.size", 2, 4575, 1657316016391, 2022-07-08T21:33:36.391Z, "DbInternal.ProcessEventLog.System.db_merge_server_qa-treasureplus-cluster-infra-1_int_illumon_com.2022-07-08.bin.2022-07-08.210000.000+0000", 3355
"/db/Intraday/DbInternal/ProcessEventLog/qa-treasureplus-cluster-infra-1_int_illumon_com/2022-07-08/ProcessEventLog/table.size", 2, 50146, 1657316070557, 2022-07-08T21:34:30.557Z, "DbInternal.ProcessEventLog.System.qa-treasureplus-cluster-infra-1_int_illumon_com.2022-07-08.bin.2022-07-08.210000.000+0000", 475383
"/db/Intraday/DbInternal/ProcessEventLog/qa-treasureplus-cluster-query-1_int_illumon_com/2022-07-08/ProcessEventLog/table.size", 2, 474510, 1657316070432, 2022-07-08T21:34:30.432Z, "DbInternal.ProcessEventLog.System.qa-treasureplus-cluster-query-1_int_illumon_com.2022-07-08.bin.2022-07-08.210000.000+0000", 2731761
"/db/Intraday/DbInternal/ProcessEventLog/db_query_server_qa-treasureplus-cluster-query-2_int_illumon_com/2022-07-08/ProcessEventLog/table.size", 2, 46555, 1657316070029, 2022-07-08T21:34:30.029Z, "DbInternal.ProcessEventLog.System.db_query_server_qa-treasureplus-cluster-query-2_int_illumon_com.2022-07-08.bin.2022-07-08.210000.000+0000", 925046
"/db/Intraday/DbInternal/ProcessEventLog/qa-treasureplus-cluster-query-2_int_illumon_com/2022-07-08/ProcessEventLog/table.size", 2, 51152, 1657316073318, 2022-07-08T21:34:33.318Z, "DbInternal.ProcessEventLog.System.qa-treasureplus-cluster-query-2_int_illumon_com.2022-07-08.bin.2022-07-08.210000.000+0000", 184474