Java process launch configuration

Note

This document is focused on bare metal deployments.

The Enterprise Deephaven system is a complex and highly configurable platform. Most of its processes are Java-based and are launched using bash scripts. Proper configuration is essential to ensure optimal performance, such as assigning appropriate memory.

This document provides an overview of how Deephaven configures and starts its processes, and how you can customize these configurations to meet your specific needs.

This document discusses:

The various processes involved in the Deephaven system, including services managed by M/Monit, query workers, and command line utilities.
Configuration details set by the installer, such as environment variables, Java executable location, and host configurations.
Instructions on modifying the launch configurations for different processes, including services and interactive scripts.
Debugging tips, including where to find log files.
How the Java CLASSPATH is constructed and where to place overrides and configuration changes.
A table of environment variables used in the Deephaven system and their purposes.

What processes?

Deephaven Java processes include the following:

The services that comprise a Deephaven deployment, including:
- Configuration Server
- Authentication Server
- Tailer
- Persistent Query Controller
- Remote Query Dispatcher (query server, merge server)
- Table Data Cache Proxy
- Data Import Server
Workers

A worker is a Deephaven Java process that performs the tasks for a Persistent Query or Code Studio. Worker processes are started by Persistent Query Dispatchers.
Command line utilities

These are programs run interactively on a Deephaven server, and sometimes non-interactively by scripts. Non-interactive usage includes installation scripts and any administrative automation you might put in place.

These utilities can be divided into two types:
- Scripts in /usr/illumon/latest/bin are utilities intended for direct use from a command line. The set includes dhconfig, dhctl, iriscat, etcdctl.sh, dh_monit, and others. Each tool has its own command line arguments. Some are sensitive to environment variables to change default behavior. Most of these tools write log files to /var/log/deephaven/misc or /tmp.
- Actions invoked via iris_exec mimic the launch process used for the Deephaven services. The actions can be one of the defined set, or any Java class. Command line options are the same as those used to configure Deephaven services in the hostconfig. Log files are written to /db/TempFiles/<user>/logs, /tmp/<user>/logs, or /var/log/deephaven/misc.

What must be configured?

Before launching a Java process, certain configurations must be determined. The launching script uses settings from installer files and options on the command line. Some options can be permanently changed for recurring processes, such as Deephaven services, while others can be adjusted ad-hoc for specific invocations, like command line utilities. The following items outline the necessary configurations:

The starting environment:

The launching script sets the following via environment variables before invoking Java:
- devroot
  
  The root directory of the Deephaven installation.
- workspace
  
  A location where user-specific files will be written.
- logging
  
  The names and locations of log files.
- The Deephaven properties file to load.
The Java executable:

The launching script locates a Java executable, ensures it is valid for the environment, and sets environment variables JAVA, DH_JAVA, JAVA_FLAGS, and JAVA_VERSION. Any preexisting values in these variables are preserved.
- Identify the Java executable as follows:
  1. Use JAVA if it is set.
  2. Use DH_JAVA if it is set.
  3. Use ILLUMON_JAVA (deprecated) if it is set.
  4. Look for a system default under /usr/java.
- Verify the version matches system requirements:
  
  Look for a version file in $DEVROOT/configs/JAVA_VERSION and /usr/illumon/latest/etc/JAVA_VERSION.
- Add some command line flags that all processes require, specific to the Java version.
CLASSPATH:

The CLASSPATH environment variable governs how Java locates JAR, class, and resource files. This enumerates locations provided by Deephaven, plug-ins, and override locations.
Java process arguments:

Some attributes of a Java process must be set when the process starts, and cannot be set later with a properties file. These include memory parameters, garbage collection settings, the system user, the name of the properties file to load, and connection bootstrapping.

This information comes primarily from hostconfig and cluster.cnf. The Deephaven installer writes default hostconfig settings to /etc/sysconfig/illumon.confs/hostconfig.system. You can augment or override these by editing /etc/sysconfig/illumon.
Bootstrapping:

This is the information required to connect to the system to get more configuration, such as how to connect to etcd and the configuration server.
Authentication:

Some of the launching scripts will sudo to the appropriate system user and must be invoked by a user with the appropriate sudo privileges.
Environment variables:

The launching script sets environment variables needed by other scripts and the Java process.

How do you change how a process is launched?

The method for changing a process's command line depends on the process type. The section below illustrates setting a Java option (-Dmy_property=my_value) and setting an environment variable (MY_ENV=my_value).

Deephaven services:

Launch parameters for Deephaven services are set in the hostconfig files. Make changes in /etc/sysconfig/illumon, and refer to the included system file, /etc/sysconfig/illumon.confs/hostconfig.system. To make changes to a process, find the appropriate section and add settings. This example changes the first operation, auth_server_reload_tool:
```
    auth_server_reload_tool)
      export MY_ENV=my_value
      export EXTRA_ARGS="$EXTRA_ARGS -j -Dmy_property=my_value"
      ;;
```
This examples adds to EXTRA_ARGS rather than replacing it.

Note

Java command line parameters are preceeded by the launch parameter -j.
Workers:

Workers are launched by the Deephaven system. The UI for creating Persistent Queries and Code Studios provides options for adding environment variables, Java arguments, and more.
Interactive scripts:

The interactive utility scripts provide command line arguments for most relevant settings. Provide additional Java arguments in the EXTRA_JAVA_ARGS environment variable.

To set an environment variable, preface your command with the setting:
```
MY_ENV=my_value EXTRA_JAVA_ARGS=-Dmy_property=my_value /usr/illumon/latest/bin/dhconfig properties list

sudo -u irisadmin MY_ENV=my_value EXTRA_JAVA_ARGS=-Dmy_property=my_value /usr/illumon/latest/bin/dhconfig properties list
```
You can change which properties file to load by setting DHCONFIG_ROOTFILE. For example, to bypass an invalid iris-common.prop properties file:
```
sudo -u irisadmin DHCONFIG_ROOTFILE=iris-defaults.prop /usr/illumon/latest/bin/dhconfig properties import --file /tmp/iris-common.prop
```
iris_exec:

Actions invoked via iris_exec mimic the launch process used for the core Deephaven services. You can change the iris_exec) stanza in the hostconfig file as described above, but the changes will apply to all actions invoked this way. Most of the time, you should provide options directly on the command line.

To set an environment variable (and also set the Java property my_property=my_value), preface your command with the setting:
```
MY_ENV=my_value /usr/illumon/latest/bin/iris_exec run_local_script -j -Dmy_property=my_value [<other launch args>]

sudo -u irisadmin MY_ENV=my_value /usr/illumon/latest/bin/iris_exec run_local_script -j -Dmy_property=my_value [<other launch args>]
```

Note

When using sudo, you must define environment variables after the sudo it discards the current environment before executing the given command.

Debugging

For tips on troubleshooting Deephaven services, refer to the Process Startup Troubleshooting guide.

Classpath

The classpath is a list of files and directories the Java runtime will use to locate classes and resources. The classpath can be specified in a command line argument (-cp or -classpath), or in the CLASSPATH environment variable. Deephaven sets the classpath using the environment variable because that method bypasses command line length limitations.

Deephaven uses several environment variables while constructing this path. Most have default values:

DEVROOT - the root of the Deephaven installation, usually /usr/illumon/latest.
WORKSPACE - a writable location for the user and process.
CUSTOMER_JAR_DIR - a comma-separated list of folders with JAR files.

The classpath consists of the following items:

$WORKSPACE/etc
/etc/sysconfig/illumon.d/override
/etc/sysconfig/illumon.d/resources
/etc/sysconfig/illumon.d/java_lib/*.jar
/etc/sysconfig/illumon.d/hotfixes/*.jar
plugin components
$DEVROOT/etc
$CLASSPATH - preexisting contents of the CLASSPATH environment variable
$DEVROOT/java_lib/*
$CUSTOMER_JAR_DIR/*.jar

Earlier files take precedence. The classpath is designed to support hotfixes.

Plugin components

Plugins are located in /etc/sysconfig/illumon.d/plugins. Each plugin will have one or both of global and worker directories (plugin_dir below). worker components are before global components.

Each plugin directory will add the following components to the classpath:

plugin_dir`/hotfixes/\*.jar
plugin_dir`/override
plugin_dir`/\* (These should be JARs or resource directories)

Environment variables

The following environment variables are relevant to this document, or to starting Deephaven processes:

Environment Variable	Description	Default
`DEVROOT`	Used to find files in the current Deephaven deployment.	`/usr/illumon/latest`
`WORKSPACE`	A location where user-specific files will be written.	`/db/TempFiles/<user>/<process>`, or a transient tmp location
`DH_JAVA`	Used to set the Java executable after `JAVA`. Will be equal to `JAVA` when Java process is started.
`JAVA`	Used to set the Java executable if set. Will be equal to `DH_JAVA` when Java process is started.
`ILLUMON_JAVA`	Deprecated. Used to set the Java executable if neither `JAVA` nor `DH_JAVA` are set.
`JAVA_FLAGS`	Deephaven adds Java version-specific arguments to any existing value.
`JAVA_VERSION`	Deephaven sets this to the version set during installation.
`DH_ETCD_BOOTSTRAP_DIR`	Location of configuration files used to connect to etcd.	`/etc/sysconfig/deephaven/etcd/client`
`DHCONFIG_ROOTFILE`	Override the default properties file used by some utility scripts.
`EXTRA_JAVA_ARGS`	Java arguments specified will be used when invoking Java.
`CUSTOMER_JAR_DIR`	A comma-separated list of folders with JAR files to be added to the `CLASSPATH`.
`CLASSPATH`	Deephaven adds items to any preexisting value and uses the result when starting Java.