Install a Podman deployment

Deephaven's Podman-based distribution allows for very quick deployment of a Deephaven Enterprise cluster for test or production purposes. Podman is an open-source container management system that is largely compatible with Docker containers and syntax. The Deephaven Podman deployment can be used for single-node and multi-node installations, as well as with pre-built or custom-built images and deployment tooling.

The Deephaven Podman distribution is supported on Linux systems. MacOS and Windows are not supported at this time. For full listing of supported OS versions (including supported dependency versions), please refer to the Supported Versions page.

Deephaven offers two container-based deployment options: Podman and Kubernetes. Podman was chosen over Docker because it allows containers to run without requiring elevated user privileges and supports systemd, a feature needed in future Deephaven releases. Compared to Kubernetes, Podman provides a simpler deployment process for single or multiple hosts with fewer dependencies. However, Kubernetes excels in dynamic resource allocation and resilience through stateful sets, making it suitable for more complex and robust service management.

Pod user accounts

The Deephaven Podman deployment option supports two account handling configurations:

• Single-user - All processes within the pod run as root, which maps back to the account running the pod on the host system.
• Multi-user - Processes within the pod run under pod-local user and group accounts that match those used in other deployment models, like bare metal and Kubernetes. In this case, subuid and subgid are used to translate pod accounts and groups to host-level accounts and groups.

The single-user mode is easier to configure but, due to the elevated internal privileges of all processes, is recommended only in environments where all users with access to Deephaven have superuser access. The multi-user mode is more complex to configure, especially when granting Deephaven processes access to external storage. Still, it allows better stratification of user access rights within Deephaven.

Installation process

At a high level, the installation process consists of:

• Configuring prerequisites on one or more host machines, including creating the VOLUME_BASE_DIR.
• Building images or downloading and loading pre-built images.
• Configuring any additional host-managed volumes and external storage connections for persistent data storage and sharing.
• Running the start_command.sh script to start each node of the deployment.

Prerequisites

General prerequisites

For Deephaven Podman deployments in general, you need:

• One or more machines or VMs with Podman installed.
  • Each machine should have at least 24GB of RAM dedicated to running Deephaven on Podman. This is the bare minimum required to reliably run a Deephaven instance and execute some small queries for a single user. More memory is strongly recommended to ensure reliability.
  • If you use the deployment for larger-scale testing or production, each machine should have 64GB or more of RAM dedicated to running Deephaven on Podman.
  • The machine(s) should also be resolvable by DNS.
• The Deephaven Podman distribution package.
• A Web server certificate to use for the installation.
• A pre-built Deephaven Podman image package. If you want to build your own images, refer to the building Podman images section.

To extract the Deephaven Podman distribution archive (update this with the version you are using):

# Extract the podman deployment tar
tar -xf deephaven-podman-1.20240517.344.tar.gz
cd deephaven-podman-1.20240517.344

Set up the VOLUME_BASE_DIR

The Deephaven Podman distribution requires a directory structure on each host to persist data and configuration outside of the pod. For multi-node deployments, parts of this directory structure must be shared between the nodes.

In this example, /container_test is used as the root of this directory structure. The user running Podman must have permissions to create this directory and its subdirectories.

export VOLUME_BASE_DIR=/container-test
mkdir -pv "$VOLUME_BASE_DIR"/{db-intraday,db-systems,deephaven-tls,deephaven-shared-config,deephaven-etcd}
chcon -vR -u system_u -r object_r -t container_file_t "${VOLUME_BASE_DIR:-/container-test}"/{db-intraday,db-systems,deephaven-tls,deephaven-shared-config,deephaven-etcd}

The path to this directory structure is exported as VOLUME_BASE_DIR. This environment variable is used in the start_command.sh examples in this document when starting the cluster.

User setup requirements

It is strongly recommended that the user running Podman commands be the user that started the login session. In other words, executing sudo su -- <podman_user_name> and then executing Podman commands as the impersonated user is not recommended and usually will not work.

If the pod(s) will be running after the user logs out, linger will need to be enabled for the podman user:

sudo loginctl enable-linger <uid_of_podman_user>`

SELinux

If you are running on Linux machines with SELinux enabled, you may need sudo rights on the machines to set the necessary SELinux flags.

You can check whether SELinux is enabled with sestatus. If SELinux is installed and active, this will report SELinux status: enabled. If it shows disabled, or the command is not recognized, then SELinux is not in effect, and chcon statements will not be needed.

ID translation

The UID of the user account running Podman (through the start_command.sh script) must have appropriate entries in /etc/subuid and /etc/subgid. These files are used to configure how user and group accounts in the pod translate to user and group accounts on the host. These are required even when running the single-user variant of the Deephaven Podman deployment.

Architecture

Deephaven's Podman deployment model uses four images:

1. dh-base - contains the base Linux system and several prerequisites.
2. dh-stage - contains staged-but-not-initialized etcd and Deephaven installations.
3. dh-infra - contains the scripts required to initialize a cluster and start the required processes for an infrastructure node.
4. dh-queryserver — is also built from dh-stage (not from dh-infra) and contains the scripts required to start the required processes for a query server node.

The images are used in conjunction with the start_command.sh script and a directory structure on the host.

The directory structure is based at a path represented by the VOLUME_BASE_DIR environment variable used in Deephaven Podman scripts. This directory structure allows for shared configuration between nodes in the cluster, and persists configuration and data so Deephaven Podman deployments can be upgraded or redeployed (if needed) without losing their configuration.

The start_command.sh script validates arguments and creates Podman command lines to manage containers and pods. When it starts the dh-infra pod for the deployment, the entrypoint of the image's Containerfile effectively runs an installation or upgrade of Deephaven within the pod. Configuration for the cluster is persisted on the host in volume subdirectories. Various types of Deephaven data can also be persisted in mounted volumes. Because configuration (and, usually, data) are persisted, the containerized deployment can be restarted with different Podman options, while maintaining state, and can also be restarted with different image versions in order to upgrade the deployment.

Volumes, files, and configuration

Volumes

The following directories (inside the containers) are configured as Podman volumes:

Additional volumes can be mounted using the --extra-volume flag to start_command.sh. Multiple instances of this flag can be used.

Two special volumes can be attached to persist, and possibly share, the contents of ìllumon.db/java-lib and ìllumon.db/calendars. When preparing host directories for these volumes, ensure that the directory and its contents are owned by the user that will be starting the container, as the initialization process will change their ownership to the $DH_ADMIN_USER:$DH_ADMIN_GROUP that is configured in the container. If SELinux is in use, then the volumes should also have their SELinux labels set with:

sudo chcon -vR -u system_u -r object_r -t container_file_t "${VOLUME_BASE_DIR:?}"

Shared configuration

The /etc/sysconfig/deephaven directory consists of some items that must be shared across nodes in a Deephaven cluster (i.e., pods in a containerized deployment with podman; servers in a non-containerized deployment). In a non-containerized Deephaven installation, the appropriate files are automatically copied between nodes during the installation process. Since this is not feasible in a containerized deployment, the required shared files are stored in a directory mounted as a volume by all containers in the cluster. This shared configuration volume is mounted as read-write by all containers.

Build images

Prerequisites to build images

• Podman installed.
• A Deephaven product archive.
• A Deephaven installer jar.
• A Deephaven Core+ archive.
• An etcd installation archive. (etcd 3.5.12 or later is recommended)
• Optionally an Envoy installation archive.
  • (The version support matrix shows supported Envoy versions.)
  • Configuring images with Envoy allows for all traffic between users and the cluster to be over a single port.
• Internet connection or alternate repository configuration so the podman build, dnf, and pip commands can download needed packages during the build process.

The build_images.sh script

Below is an example of how to build the container images. The example command must be run from the extracted deephaven-podman-<version> directory (e.g. deephaven-podman-1.20240517.245); this is the directory that contains the build_images.sh and start_command.sh scripts.

When running build_images.sh, you must specify the location of the Deephaven installation artifacts to use. These can be specified via URLs or included in the dh-base/ subdirectory. See dh-base build arguments for more details.

# An example with the required files having been placed in the dh-base directory
├── dh-base
│   ├── Containerfile
│   ├── deephaven-coreplus-0.37.4-1.20240517.344-jdk17.tgz
│   ├── deephaven-enterprise-jdk17-1.20240517.344.tar.gz
│   ├── etcd-v3.5.13-linux-amd64.tar.gz
│   └── Installer-1.20240517.344.jar

If Envoy is configured, the Envoy binary must be placed in the dh-infra/ subdirectory. Envoy requires Rocky 9 or RHEL 9 as the base image family.

# Build and tag the images. The base image can also be specified with the BASE_IMAGE environment variable:
./build_images.sh \
--dh-file deephaven-enterprise-jdk17-1.20240517.245.tar.gz \
--etcd-file etcd-v3.5.13-linux-amd64.tar.gz \
--installer-file Installer-1.20240517.245.jar \
--coreplus-tar-file deephaven-coreplus-0.36.1-1.20240517.245-jdk17.tgz \
--jdk17 \
--python-package python3-devel.x86_64 \
--base-image docker.io/amd64/rockylinux:9.3-minimal

In this example, the prerequisite Deephaven installation archive, Core+ installation archive, etcd installation archive, and Deephaven installer JAR have already been downloaded and copied to the dh_base directory. This command line also specifies a particular Python package to install, a particular Rocky 8 base image to use, and to use the Java 17 JDK.

The images can be built by running build_images.sh, which will build the dh-base/dh-stage/dh-infra/dh-query images and tag them with those same names.

Built images can be viewed with podman image ls:

[user@podman-host dh-podman]$ podman image ls
REPOSITORY                  TAG                    IMAGE ID      CREATED        SIZE
localhost/dh-queryserver    latest                 cd1b83b47cea  4 days ago     5.81 GB
localhost/dh-infra          latest                 821738b91c07  4 days ago     5.96 GB
localhost/dh-stage          latest                 2bf8a0c9312c  4 days ago     5.81 GB
localhost/dh-base           latest                 d091604e94bb  4 days ago     2.29 GB

Other tested base image families are Rocky 9, RHEL 8, and RHEL 9. Using something other than these is not currently supported and may require significant changes to Containerfiles and dh_init.sh and related scripts.

Plugins

Plugins can be included in the images by placing the plugin archives into dh-stage/plugins/. The plugins to be installed must be listed in the plugins manifest file, dh-stage/plugins/plugins.txt. The SAML plug-in is built into the product itself.

The plugins manifest file contains entries with the following format:

<plugin-target-dir>=<plugin-file>
<plugin-target-dir>=<plugin-file>=<properties-file>

If the <properties-file> is specified, it is appended to iris-environment.prop after the plugin has been installed. (The properties are not added until after the plugin has been installed to ensure that the plugin's files are available on the classpath.) A plugin's properties file will only be added to the configuration the first time a cluster is initialized; if the container detects an existing configuration during initialization, the properties will not be appended to the configuration. This prevents duplication of the properties appended to iris-environment.prop.

Lines beginning with # are treated as comments and are ignored.

External storage

Paths on the host that are connected to volumes can, in turn, be mapped or linked to network storage. NFS should not be used for intraday data (/db/Intraday and /db/IntradayUser) or for binary log files such as /var/log/deephaven/binlogs, but NFS, or a similar shareable network storage solution, is required to share historical data under /db/Systems across nodes of a cluster, and can also be used to share historical data between multiple Deephaven installations.

Requirements for access to storage

Single-user installation

When running a single-user installation, all access operations originating in the pod are run on the host in the context of the user who ran the start_command.sh. Inside the pod this user is root/root - root user and root default group. For such installations, Deephaven services in the pod have access to any resources at the host that the Podman user has access to. In many cases, full control access will be required for the running user, as Deephaven does things like renaming directories during merge operations and replacing links when rolling log files.

Multi-user installation

In a multi-user installation, separate user and group accounts are used for different services and tasks inside the pod. These users and groups are translated to host-context UIDs and GIDs based on configured subuids and subgids on the host.

The translated UIDs and GIDs must be granted permissions to give the pod account access to resources attached to the host. The subuid and subgid files have entries for the name of each user who has authority to translate accounts from containers, along with an offset and a range of IDs to allocate.

For example, an entry like podman_user:100000:10000 allows podman_user to run containers that include their own internal user IDs, with the internal IDs as high as 10000. The translated ID on the host will be the internal UID plus the offset, minus one.

With this configuration, the default Deephaven dbmerge account, which has the default ID of 9001, will be represented on the host as UID 109000. When attaching volumes that are hosted off the host (e.g., NFS or SAN), check with your storage admin as to how best to grant the needed permissions to your translated user and group accounts. One option to make this easier to manage is creating local accounts on the host to represent the translated IDs.

Linux useradd and groupadd commands allow you to specify what UID or GID to use when creating an account. Although this is theoretically possible on OSX systems, support for it under the newer sysadminctl command is unclear. Check with your Apple support team to create accounts with specific IDs on an OSX system.

Start the cluster

The start_command.sh script creates a new pod with the required ports exposed, adds a container created from the appropriate image (either dh-infra or dh-queryserver), and starts the container. If another pod of the same name already exists, it is stopped and removed before the new pod is created.

Podman itself also has podman pod stop and podman pod start commands. These can be used to effectively pause and resume Podman pods, and are relatively quick operations compared to start_command.sh. These commands may be used as often as needed if you don't want to run the cluster continuously.

For example:

podman pod stop dh-infra-pod

The start_command.sh script is normally used only when starting a new Deephaven Podman cluster, or when changing start_command.sh options (such as mounted volumes) or upgrading the cluster. When the start_commands.sh script starts a pod, the pod's ENTRYPOINT runs, which invokes dh-init.sh. This script effectively runs the Deephaven installation/upgrade process for the node; in the case of an infrastructure node, it also runs the installation process for the cluster (setting up shared configuration, etc.).

export POD_IP=<IPv4 address of the Deephaven cluster>

Single-node clusters

Single-node clusters (where only one node runs all the Deephaven services) do not need shared storage and can be started with a single invocation of start_command.sh.

This example runs from the directory where the start_command.sh has been extracted:

POD_HOSTNAME= <URL name of the DH server>
./start_command.sh \
-e 8000 \
-h "${POD_HOSTNAME:?}" \
-t infra -T "${VOLUME_BASE_DIR:?}/deephaven-tls/" \
-d \
-n dh-infra \
-s \
-I "${VOLUME_BASE_DIR:?}/db-intraday/" \
-H "${VOLUME_BASE_DIR:?}/db-systems/"

Where the arguments are:

• -e 8000 - connect and forward host port 8000 to the pod's Envoy service.
• -h <URL name of the DH server> - the name by which you will connect to the server, which will be the same as the name in the Web certificate, such as mydeephaven.myorg.com.
• -t infra - the type of image to run, infra in this case.
• -T "${VOLUME_BASE_DIR:?}/deephaven-tls/" - path to the Web server certificate files.
• -d - delete any existing pod(s) running for this container name.
• -n dh-infra - the container name, and the base name to use for the main pod (dh-infra-pod, in this case).
• -s - single node Envoy instance, so, only forward -e 8000 into the container.
• -H "${VOLUME_BASE_DIR:?}/db-systems/" - mount the db-systems host directory to /db/Systems in the container for persistent storage of historical data.
• -I"${VOLUME_BASE_DIR:?}/db-intraday/" - mount the db-intraday host directory to /db/Intraday in the container for persistent storage of intraday data.

Without Envoy, the -e 8000 and -s options would be omitted.

Multi-node clusters

Multi-node clusters need the "${VOLUME_BASE_DIR:?}"/deephaven-shared-config path to be available to all nodes of the cluster.

To start a multi-node cluster, you first execute the start_command.sh script for each query node, and then for the infra node. Query pods wait to initialize until they see a COMPLETE status value in the cluster_status shared configuration file.

start_command.sh details

The script takes the following arguments:

The start_command.sh script also reads the following environment variables:

Changing properties

Most configuration of Deephaven Podman deployments is the same as for traditional deployments. Tasks like editing properties files or routing YAML are accomplished using the dhconfig tool. A key point to remember is that Deephaven-specific configuration changes do not require restarting the entire system; configuration changes that affect the behavior of a Deephaven service typically require just the service in question to be restarted with dh_monit.

For changes that affect configuration used by multiple services, it is recommended to /usr/illumon/latest/bin/dh_monit restart all, and then monitor service states as they restart with watch /usr/illumon/latest/bin/dh_monit summary.

Most property changes that affect only query workers will be in effect for any workers started after the change has been imported, without any service restarts needed.

Add nodes

Nodes can be added by updating the iris-endpoints.prop file to include the new node. See the configuration changes to define a new pod section in the appendix for an example of the required configuration changes.

After importing the updated iris-endponts.prop file, the new node can be used after reloading the Controller process's configuration:

/usr/illumon/latest/bin/iris controller_tool --reload

A new node's certificates must be signed by the same CA provided when creating the infra node, as with the existing nodes.

Appendix

Images

Images can be custom built using the build-images.sh script, or pre-built images can be downloaded from Deephaven.

dh-base

The dh-base image is built from a base Linux image(e.g. docker.io/redhat/ubi8-minimal or docker.io/amd64/rockylinux:8.7-minimal) and contains several installation files (copied in from the dh-infra directory) as well as prerequisites installed via dnf.

Base image build arguments

The following build arguments influence how dh-base is built:

These arguments can be configured automatically by build_images.sh.

Base image files

• The illumon-db-<VERSION>.tar.gz file is copied to /tmp/illumon-db.tar.gz.
• The Installer-<VERSION>.jar file is copied to /tmp/Installer.tar.
• The etcd-<VERSION>-linux-amd64.tar.gz file is copied to /tmp/etcd/etcd.tar.gz.

Packages

The following sets of packages are installed via dnf/microdnf:

1. Using microdnf:
   • dnf
   • subscription-manager
2. The EPEL repo, unless the image is built with --build-arg ADD_EPEL_REPO=N. The repo is installed from https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm or https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm, depending on OS major version (8 or 9).
3. The following Deephaven dependencies:
   • For OS major version 8:
     • redhat-lsb-core
     • initscripts
     • monit
     • wget
     • unzip
     • glibc
     • libgcc
     • libgomp
     • libstdc++
     • bzip2
     • git
     • which
     • openssl
     • rsync-3.1*
   • For OS major version 9:
     • initscripts
     • tar
     • hostname
     • monit
     • cronie
     • ed
     • wget
     • unzip
     • glibc
     • libgcc
     • libgomp
     • libstdc++
     • zeromq-devel
     • bzip2
     • git
     • which
     • openssl
     • rsync-3.*
4. sudo
5. The JDK. This is determined by the JDK_PACKAGE_NAME build argument. (See base image build arguments for more details.)
6. Helpful packages for debugging from within the container, unless the image is built with --build-arg ADD_DEBUG_TOOLS=N:
   • bind-utils (provides dig)
   • net-tools (provides netstat)

dh-stage

The dh-stage image is built from dh-base and has both Deephaven and etcd installed, but not initialized. The general steps to build dh-stage are:

1. Copy in the script with the common functions for container initialization (dh-init-common.sh).
2. Copy in the common cluster.cnf.base file, which contains environment variables that influence installation scripts. This is customized for the initial cluster layout (i.e., the specific infra/query nodes and their hostnames) when dh-init.sh runs (when the infra container first initializes). The final, customized cluster.cnf is stored in shared configuration and used by the other nodes as they initialize.
3. Update /etc/sudoers with the required permissions for the Deephaven user accounts.
4. Extract the Deephaven tar archive (to /usr/illumon and /etc/sysconfig/installer-deephaven).
5. Create several required directories and links.
6. Create the required user accounts and groups.
7. Extract the etcd tar and install etcd to /usr/bin/.
8. Add the DH username variables to cluster.cnf.base.
9. Set the OS name and version string in cluster.cnf.base (based on values extracted from /etc/os-release).
10. Copy in any plugins (from dh-stage/plugins/ to /tmp/plugins). The plugins are extracted and installed during container initialization.

Stage image build arguments

The following build arguments influence how dh-stage is built:

These arguments can be configured automatically by build_images.sh.

Use --use-non-root-accounts with build_images.sh to change the values of these arguments. --use-non-root-accounts instructs build_images.sh to check for account name environment variables; any that have not been set will use Deephaven defaults:

# Default users and groups, or values provided by environment variables
monit_user="${DH_MONIT_USER:-irisadmin}"
monit_group="${DH_MONIT_GROUP:-irisadmin}"
admin_user="${DH_ADMIN_USER:-irisadmin}"
admin_group="${DH_ADMIN_GROUP:-irisadmin}"
merge_user="${DH_MERGE_USER:-dbmerge}"
merge_group="${DH_MERGE_GROUP:-dbmerge}"
query_user="${DH_QUERY_USER:-dbquery}"
query_group="${DH_QUERY_GROUP:-dbquery}"
shared_group="${DH_SHARED_GROUP:-dbmergegrp}"
shared_query_group="${DH_SHARED_QUERY_GROUP:-dbquerygrp}"
etcd_user="${DH_ETCD_USER:-etcd}"

For example, if the environment variable DH_MONIT_USER is set when build_images.sh is run with --use-non-root-accounts, the set value will be used for the monit user. If it is not set, then irisadmin will be used for the monit user.

dh-infra

The dh-infra container is built from dh-stage and contains configuration files and scripts required to finalize the Deephaven installation and initialize etcd.

If USE_ENVOY=enabled, the dh-infra ContainerFile expects to copy an Envoy binary named envoy_binary from the dh-infra directory. build_images.sh can be passed a different name for the Envoy binary file, which it will copy to envoy_binary before building the dh-infra image. Envoy requires a major version of 9 for the base OS.

Infra image build arguments

The following build argument influences how dh-infra is built:

This argument can be configured automatically by build_images.sh.

dh-queryserver

The dh-queryserver container is built from dh-stage and contains configuration files and scripts required to start a query server node that uses the configuration initialized by the dh-infra node.

Image scripts and base configuration files

The following files are included in the dh-infra image:

The following files are included in the dh-queryserver image:

The following files are included in the dh-stage image (and thus are included in both dh-infra and dh-queryserver):

build_images.sh

This script automates the process of calling podman build for each of the containers (including setting build arguments and tagging the resulting images).

The script takes the following arguments:

Shared configuration directory contents

The following table provides an overview of the subdirectories of /deephaven-shared-config:

/etc/sysconfig/deephaven directory contents

The following table provides an overview of the subdirectories of /etc/sysconfig/deephaven and whether they are shared across containers in a Podman deployment of Deephaven:

For reference, here is an example ls from illumon.d.latest on a non-containerized Deephaven installation:

$ ls -l illumon.d.latest/
total 12
lrwxrwxrwx. 1 irisadmin irisadmin    29 Jun  9 18:07 auth -> /etc/sysconfig/deephaven/auth
drwx------. 2 irisadmin irisadmin     6 Apr  1 02:33 auth-backup
lrwxrwxrwx. 1 irisadmin irisadmin    34 Apr  1 02:31 auth-user -> /etc/sysconfig/deephaven/auth-user
drwxr-xr-x. 2 irisadmin irisadmin     6 Apr  1 02:31 calendars
drwxr-xr-x. 2 irisadmin irisadmin     6 Apr  1 02:31 chartthemes
drwxr-xr-x. 4 irisadmin irisadmin  4096 Jun  9 18:07 client_update_service
lrwxrwxrwx. 1 irisadmin irisadmin    34 Jun  9 18:07 dh-config -> /etc/sysconfig/deephaven/dh-config
lrwxrwxrwx. 1 irisadmin irisadmin    29 Jun  9 18:07 etcd -> /etc/sysconfig/deephaven/etcd
drwxr-xr-x. 2 irisadmin irisadmin     6 Jun  9 18:07 hotfixes
drwxr-xr-x. 2 irisadmin irisadmin     6 Apr  1 02:31 integrations
drwxr-xr-x. 2 irisadmin irisadmin   124 Apr 28 09:38 java_lib
lrwxrwxrwx. 1 irisadmin irisadmin    30 Jun  9 18:07 monit -> /etc/sysconfig/deephaven/monit
drwxr-xr-x. 3 irisadmin irisadmin  4096 Jun  9 18:07 monit.new
drwxr-xr-x. 2 irisadmin irisadmin     6 Jun  9 18:07 override
drwxrwxr-x. 2 irisadmin dbmergegrp   18 May  1 12:56 plugins
drwxr-xr-x. 2 irisadmin irisadmin  4096 May 28 22:03 resources
lrwxrwxrwx. 1 irisadmin irisadmin    31 Jun  9 18:07 schema -> /etc/sysconfig/deephaven/schema

Note that many of the directories in illumon.d.latest are simply links back to directories in /etc/sysconfig/deephaven. Of these directories, those that must be shared in a podman deployment (i.e., auth/dh-config/etcd/trust) are stored in a volume mounted at /deephaven-shared-config, and links are created from /etc/sysconfig/deephaven/<dir> to the corresponding directory in /deephaven-shared-config.

Configuration changes to define a new pod

The following two sections demonstrate the 'before and after' of configuration changes to iris-endpoints.prop to add a pod to a running cluster.

The iris-endpoints.prop file can be exported by connecting to the infra pod (e.g. podman exec dh-infra -it bash) and running:

/usr/illumon/lastest/bin/dhconfig properties export -f iris-endponts.prop -d .

After making the requisite changes to the iris-endpoints.prop file, the updated properties can be reimported with:

/usr/illumon/lastest/bin/dhconfig properties import -f iris-endponts.prop -d .

Example iris-endpoints.prop before adding a query pod

[service.name=iris_controller|controller_tool|configuration_server] {
iris.db.1.host=rhel8-test-infra.devrel.deephaven.io
iris.db.1.classPushList=
iris.db.1.class=Query

iris.db.2.host=rhel8-test-query.devrel.deephaven.io
iris.db.2.classPushList=
iris.db.2.class=Query

iris.db.3.host=rhel8-test-infra.devrel.deephaven.io
iris.db.3.classPushList=
iris.db.3.port=30002
iris.db.3.class=Merge
iris.db.3.websocket.port=22060

iris.db.4.host=rhel8-test-query.devrel.deephaven.io
iris.db.4.classPushList=
iris.db.4.port=30002
iris.db.4.class=Merge
iris.db.4.websocket.port=22060

iris.db.nservers=4
}

[host=dh-infra-pod] {
  RemoteQueryDispatcherParameters.host=rhel8-test-infra.devrel.deephaven.io
}
[host=dh-query1-pod] {
  RemoteQueryDispatcherParameters.host=rhel8-test-query.devrel.deephaven.io
}

Example iris-endpoints.prop after adding a query pod

The example configuration below shows how the above example should be modified to add a new host (in this case, rhel8-test-query2.devrel.deephaven.io, run as dh-query2-pod). Note the three critical changes to iris-endpoints.prop:

• New dispatchers defined (iris.db.5 and iris.db.6). Note that both a query dispatcher and a merge dispatcher are added for the new host (rhel8-test-query2.devrel.deephaven.io).

• The iris.db.nservers property is updated to match the number of iris.db.<N> sections.

• An additional host-scoped configuration section is added for the new pod, informing the query dispatcher of its external-facing hostname. If this step is skipped, the dispatcher will be unable to communicate with etcd. This may manifest in errors such as the following:

  java.lang.RuntimeException: Failed to get initial discovery watcher update
  at io.deephaven.enterprise.dispatcher.client.DispatcherClient.init(DispatcherClient.java:193)
  at io.deephaven.enterprise.dnd.Main.runDispatcherClient(Main.java:427)
  at io.deephaven.enterprise.dnd.Main.main(Main.java:164)
  

Additionally, be aware that Persistent Queries are assigned to a server based on the server ID number (e.g. 3 in iris.db.3). Changing a server (e.g. changing iris.db.3.host=host3.mycompany.net to iris.db.3.host=host6.mycompany.net) will cause PQs that were running on host3.mycompany.net to be assigned to host6.mycompany.net.

[service.name=iris_controller|controller_tool|configuration_server] {
iris.db.1.host=rhel8-test-infra.devrel.deephaven.io
iris.db.1.classPushList=
iris.db.1.class=Query

iris.db.2.host=rhel8-test-query.devrel.deephaven.io
iris.db.2.classPushList=
iris.db.2.class=Query

iris.db.3.host=rhel8-test-infra.devrel.deephaven.io
iris.db.3.classPushList=
iris.db.3.port=30002
iris.db.3.class=Merge
iris.db.3.websocket.port=22060

iris.db.4.host=rhel8-test-query.devrel.deephaven.io
iris.db.4.classPushList=
iris.db.4.port=30002
iris.db.4.class=Merge
iris.db.4.websocket.port=22060

iris.db.5.host=rhel8-test-query2.devrel.deephaven.io
iris.db.5.classPushList=
iris.db.5.class=Query

iris.db.6.host=rhel8-test-query2.devrel.deephaven.io
iris.db.6.classPushList=
iris.db.6.port=30002
iris.db.6.class=Merge
iris.db.6.websocket.port=22060

iris.db.nservers=6
}

[host=dh-infra-pod] {
  RemoteQueryDispatcherParameters.host=rhel8-test-infra.devrel.deephaven.io
}
[host=dh-query1-pod] {
  RemoteQueryDispatcherParameters.host=rhel8-test-query.devrel.deephaven.io
}
[host=dh-query2-pod] {
  RemoteQueryDispatcherParameters.host=rhel8-test-query2.devrel.deephaven.io
}

Related documentation

• Podman Quickstart
• Restart and update a Podman deployment
• Troubleshooting a Podman deployment
• Kubernetes