Java process launch configuration

Note

This document only applies to traditional installations.

The Enterprise Deephaven system is a complex and highly configurable platform. Most of its processes are Java-based and are launched using bash scripts. Proper configuration is essential to ensure optimal performance, such as assigning appropriate memory.

This document provides an overview of how Deephaven configures and starts its processes, and how you can customize these configurations to meet your specific needs.

This document discusses:

The various processes involved in the Deephaven system, including services managed by M/Monit, query workers, and command line utilities.
Configuration details set by the installer, such as environment variables, Java executable location, and host configurations.
Instructions on modifying the launch configurations for different processes, including services and interactive scripts.
How the Java CLASSPATH is constructed and where to place overrides and configuration changes.
The environment variables used in the Deephaven system and their purposes.

For tips on troubleshooting Deephaven services, refer to the Process Startup Troubleshooting guide.

Processes

Deephaven Java processes include the following:

Services: A service provides a set of functionalities together with policies that can be reused via a prescribed interface. A Deephaven Enterprise deployment is comprised of the following services:
- Configuration Server - provides configuration information to other Deephaven services and components. It is responsible for managing the configuration files and settings used by the Deephaven system.
- Authentication Server - handles user authentication and authorization. It is responsible for verifying user credentials and managing access control to Deephaven resources.
- Tailer - responsible for streaming data from a source to a destination. It handles the transfer of data in real-time, ensuring that the data is delivered to the appropriate location.
- Persistent Query Controller - manages the scheduling and execution of PQs in Deephaven.
- Remote Query Dispatcher (query server, merge server) - responsible for distributing queries across multiple servers. It handles the routing of queries to the appropriate server for execution.
- Table Data Cache Proxy - responsible for caching and managing table data in Deephaven.
- Data Import Server - handles the import of data from external sources into Deephaven. It is responsible for managing the data import process and ensuring that the data is properly formatted and stored in Deephaven.
Workers: A worker is a Deephaven Java process that performs the tasks for a PQ or Code Studio. Worker processes are started by Persistent Query Dispatchers.
Command line utilities: These programs are run interactively on a Deephaven server or non-interactively through scripts. Non-interactive usage includes installation scripts and any administrative automation you may implement.

These utilities are categorized into two types:
- Scripts located in /usr/illumon/latest/bin: These are designed for direct use from the command line. Examples include dhconfig, dhctl, iriscat, and etcdctl.sh. Each tool has its own command-line arguments and may rely on environment variables to modify default behavior. Most of these tools write log files to /var/log/deephaven/misc or /tmp.
- Actions invoked via iris_exec: These mimic the launch process used for Deephaven services. Actions can include predefined tasks or any Java class. Command-line options are the same as those used to configure Deephaven services in the hostconfig. Log files are written to /db/TempFiles/<user>/logs, /tmp/<user>/logs, or /var/log/deephaven/misc.

Configuration

Configuration properties must be set prior to launching Java processes. The launching script relies on settings from installer files and command-line options. Some options can be permanently modified for recurring processes, such as Deephaven services. Others can be adjusted temporarily for one-time invocations, like command-line utilities. The following items outline the required configurations:

The starting environment: The launching script sets the following via environment variables before invoking Java:
- devroot: The root directory of the Deephaven installation.
- workspace: A location where user-specific files will be written.
- logging: The names and locations of log files.
- The Deephaven properties file to load.
The Java executable: The launching script locates a Java executable, ensures it is valid for the environment, and sets environment variables JAVA, DH_JAVA, JAVA_FLAGS, and JAVA_VERSION. Any preexisting values in these variables are preserved.
- Identify the Java executable as follows:
  1. Use JAVA if it is set.
  2. Use DH_JAVA if it is set.
  3. Look for a system default under /usr/java.
- Verify the version matches system requirements:
  
  Look for a version file in $DEVROOT/configs/JAVA_VERSION and /usr/illumon/latest/etc/JAVA_VERSION.
- Add some command line flags that all processes require, specific to the Java version.
The CLASSPATH: The CLASSPATH environment variable governs how Java locates JAR, class, and resource files. This enumerates locations provided by Deephaven, plug-ins, and override locations.
Java process arguments: Some attributes of a Java process must be set when the process starts, and cannot be set later with a properties file. These include memory parameters, garbage collection settings, the system user, the name of the properties file to load, and connection bootstrapping. This information comes primarily from hostconfig and cluster.cnf. The Deephaven installer writes default hostconfig settings to /etc/sysconfig/illumon.confs/hostconfig.system. You can augment or override these by editing files in /etc/sysconfig/illumon.
Bootstrapping: This is the information required to connect to the system to get more configuration, such as how to connect to etcd and the configuration server.
Authentication: Some of the launching scripts will sudo to the appropriate system user and must be invoked by a user with the appropriate sudo privileges.
Environment variables: The launching script sets environment variables needed by other scripts and the Java process.

Process launch

The method for changing how a process is launched depends on the process type. The section below illustrates setting a Java option (-Dmy_property=my_value) and setting an environment variable (MY_ENV=my_value).

Deephaven services: Launch parameters for Deephaven services are set in the hostconfig files. Make changes in /etc/sysconfig/illumon, and refer to the included system file, /etc/sysconfig/illumon.confs/hostconfig.system. To make changes to a process, find the appropriate section and add settings. This example changes the first operation, auth_server_reload_tool:
```
    auth_server_reload_tool
      export MY_ENV=my_value
      export EXTRA_ARGS="$EXTRA_ARGS -j -Dmy_property=my_value"
      ;;
```
This examples adds to EXTRA_ARGS rather than replacing it.

Note

Java command line parameters are preceeded by the launch parameter -j.
Workers: Workers are launched by the Deephaven system. The UI for creating Persistent Queries and Code Studios provides options for adding environment variables, Java arguments, and more.
Interactive scripts: The interactive utility scripts provide command line arguments for most relevant settings. Provide additional Java arguments in the EXTRA_JAVA_ARGS environment variable.

To set an environment variable, preface your command with the setting:
```
MY_ENV=my_value EXTRA_JAVA_ARGS=-Dmy_property=my_value /usr/illumon/latest/bin/dhconfig properties list

sudo -u irisadmin MY_ENV=my_value EXTRA_JAVA_ARGS=-Dmy_property=my_value /usr/illumon/latest/bin/dhconfig properties list
```
You can change which properties file to load by setting DHCONFIG_ROOTFILE. For example, to bypass an invalid iris-common.prop properties file:
```
sudo -u irisadmin DHCONFIG_ROOTFILE=iris-defaults.prop /usr/illumon/latest/bin/dhconfig properties import --file /tmp/iris-common.prop
```
iris_exec: Actions invoked via iris_exec mimic the launch process used for the core Deephaven services. You can change the iris_exec) stanza in the hostconfig file as described above, but the changes will apply to all actions invoked this way. Most of the time, you should provide options directly on the command line.

To set an environment variable (and also set the Java property my_property=my_value), preface your command with the setting:
```
MY_ENV=my_value /usr/illumon/latest/bin/iris_exec run_local_script -j -Dmy_property=my_value [<other launch args>]

sudo -u irisadmin MY_ENV=my_value /usr/illumon/latest/bin/iris_exec run_local_script -j -Dmy_property=my_value [<other launch args>]
```

Note

When using sudo, you must define environment variables after the sudo, which discards the current environment before executing the given command.

Classpath

The classpath is a list of files and directories the Java runtime will use to locate classes and resources. The classpath can be specified in a command line argument (-cp or -classpath), or in the CLASSPATH environment variable. Deephaven sets the classpath using the environment variable because that method bypasses command line length limitations.

Deephaven uses several environment variables while constructing this path. Most have default values:

DEVROOT - the root of the Deephaven installation, usually /usr/illumon/latest.
WORKSPACE - a writable location for the user and process.
CUSTOMER_JAR_DIR - a comma-separated list of folders with JAR files.

The classpath consists of the following items:

$WORKSPACE/etc
/etc/sysconfig/illumon.d/override
/etc/sysconfig/illumon.d/resources
/etc/sysconfig/illumon.d/java_lib/*.jar
/etc/sysconfig/illumon.d/hotfixes/*.jar
plugin components
$DEVROOT/etc
$CLASSPATH - preexisting contents of the CLASSPATH environment variable
$DEVROOT/java_lib/*
$CUSTOMER_JAR_DIR/*.jar

Earlier files take precedence. The classpath is designed to support hotfixes.

Plugin components

Plugins are stored in /etc/sysconfig/illumon.d/plugins. Each plugin may include a global directory, a worker directory, or both (plugin_dir refers to these directories). Components in the worker directory are processed before those in the global directory.

Each plugin directory adds the following components to the classpath:

plugin_dir`/hotfixes/\*.jar
plugin_dir`/override
plugin_dir`/\* (These should be JARs or resource directories)

Environment variables

The following environment variables are relevant to this document, or to starting Deephaven processes:

Environment Variable	Description	Default
`DEVROOT`	Used to find files in the current Deephaven deployment.	`/usr/illumon/latest`
`WORKSPACE`	A location where user-specific files will be written.	`/db/TempFiles/<user>/<process>`, or a transient tmp location
`DH_JAVA`	Used to set the Java executable after `JAVA`. Will be equal to `JAVA` when Java process is started.
`JAVA`	Used to set the Java executable if set. Will be equal to `DH_JAVA` when Java process is started.

`JAVA_FLAGS`	Deephaven adds Java version-specific arguments to any existing value.
`JAVA_VERSION`	Deephaven sets this to the version set during installation.
`DH_ETCD_BOOTSTRAP_DIR`	Location of configuration files used to connect to etcd.	`/etc/sysconfig/deephaven/etcd/client`
`DHCONFIG_ROOTFILE`	Override the default properties file used by some utility scripts.
`EXTRA_JAVA_ARGS`	The specified Java arguments are used when invoking Java.
`CUSTOMER_JAR_DIR`	A comma-separated list of folders with JAR files to be added to the `CLASSPATH`.
`CLASSPATH`	Deephaven adds items to any preexisting value and uses the result when starting Java.