Install a Podman deployment

Deephaven's Podman-based distribution allows for very quick deployment of a Deephaven Enterprise cluster for test or production purposes. Podman is an open-source container management system that is largely compatible with Docker containers and syntax. The Deephaven Podman deployment can be used for single-node and multi-node installations, as well as with pre-built or custom-built images and deployment tooling.

The Deephaven Podman distribution is supported on Linux systems. MacOS and Windows are not supported at this time. For a full listing of supported OS versions (including supported dependency versions), please refer to the Supported Versions page.

Note

The Podman Quickstart provides simplified instructions. This guide covers all details of Deephaven Podman installation, including more complex configurations.

Deephaven offers two container-based deployment options: Podman and Kubernetes. Podman was chosen over Docker because it allows containers to run without requiring elevated user privileges and supports systemd, a feature needed in future Deephaven releases. Compared to Kubernetes, Podman provides a simpler deployment process for single or multiple hosts with fewer dependencies. However, Kubernetes excels in dynamic resource allocation and resilience through stateful sets, making it suitable for more complex and robust service management.

Pod user accounts

The Deephaven Podman deployment option supports two account handling configurations:

  • Single-user: all processes within the pod run as root, which maps back to the account running the pod on the host system.
  • Multi-user: processes within the pod run under pod-local user and group accounts that match those used in other deployment models, like bare metal and Kubernetes. In this case, subuid and subgid are used to translate pod accounts and groups to host-level accounts and groups.

Hosts in a multi-node cluster share files, and the shared storage records their owners as numeric IDs. Give the account that runs Podman the same UID on every host, and in a multi-user deployment the same subuid and subgid offset and range, or pods on one host cannot use files written on another. See Requirements for access to storage.

The single-user mode is easier to configure but, due to the elevated internal privileges of all processes, is recommended only in environments where all users with access to Deephaven have superuser access. The multi-user mode is more complex to configure, especially when granting Deephaven processes access to external storage. Still, it allows better stratification of user access rights within Deephaven.

The pre-built image package is single-user. For a multi-user deployment, build your own images with build_images.sh --use-non-root-accounts.

Caution

Switching between single-user and multi-user deployments while maintaining configuration is not supported.

Installation process

At a high level, the installation process consists of:

For how the images, volumes, and shared configuration fit together, see Architecture.

Prerequisites

General prerequisites

For Deephaven Podman deployments in general, you need:

  • One or more machines or VMs with Podman version 4.2.0 or later installed; Deephaven recommends the latest version your Linux distribution provides. Run podman --version to check the installed version.
    • Each machine must have at least 32GB of RAM dedicated to running Deephaven on Podman. This is the bare minimum required to reliably run a Deephaven instance and execute some small queries for a single user.
    • 64GB or more of RAM dedicated to running Deephaven on Podman is recommended. For production, size each machine according to the installation planning guide.
    • The machine(s) should also be resolvable by DNS.
    • The Deephaven Podman distribution package.
    • A TLS certificate to use for the installation. To generate a test certificate with the bundled create_dh_cert.sh script instead, the host needs openssl, keytool (from a JDK), and pwmake, which the script uses to generate its key store passwords. On RHEL and Rocky Linux, install them with sudo dnf install openssl java-17-openjdk-headless libpwquality; on Ubuntu and Debian, with sudo apt install openssl openjdk-17-jre-headless libpwquality-tools. The script stops and names any that are missing.
  • A pre-built Deephaven Podman image package. If you want to build your own images, refer to the building Podman images section.

To extract the Deephaven Podman distribution archive (update this with the version you are using):

Set up the VOLUME_BASE_DIR

The Deephaven Podman distribution requires a directory structure on each host to persist data and configuration outside of the pod. For multi-node deployments, parts of this directory structure must be shared between the nodes.

In this example, /container-test is used as the root of this directory structure. The user running Podman must have permissions to create this directory and its subdirectories.

The path to this directory structure is exported as VOLUME_BASE_DIR. This environment variable is used in the start_command.sh examples in this document when starting the cluster.

Important

export VOLUME_BASE_DIR=/<base_directory_for_external_configuration_and_data> should be added to the user's profile, or some sort of start_command.sh wrapper script, otherwise it will need to be re-executed each time the user logs in.

User setup requirements

It is strongly recommended that the user running Podman commands be the user that started the login session. In other words, executing sudo su -- <podman_user_name> and then executing Podman commands as the impersonated user is not recommended and usually will not work.

If the pod(s) will be running after the user logs out, linger will need to be enabled for the podman user:

SELinux

If you are running on Linux machines with SELinux enabled, you may need sudo rights on the machines to set the necessary SELinux flags.

You can check whether SELinux is enabled with sestatus. If SELinux is installed and active, this will report SELinux status: enabled. If it shows disabled, or the command is not recognized, then SELinux is not in effect, and chcon statements will not be needed.

Deephaven's start_command.sh mounts most volumes with the :z option, which tells Podman to relabel the volume so that containers can use it. Files that a pod's processes create in a volume carry that pod's SELinux Multi-Category Security (MCS) categories. When you delete and recreate a pod, the new pod receives different MCS categories, which can cause "Permission Denied" errors on files created by the previous pod.

For troubleshooting SELinux-related permission issues, see Troubleshooting SELinux in Podman.

ID translation

The UID of the user account running Podman (through the start_command.sh script) must have appropriate entries in /etc/subuid and /etc/subgid. These files are used to configure how user and group accounts in the pod translate to user and group accounts on the host. These are required even when running the single-user variant of the Deephaven Podman deployment.

Supplying signing certificates to Podman

When setting up a Podman cluster, each pod requires, at minimum, a certificate and private key that can sign for the hostname specified by the --hostname|-h flag passed to start_command.sh.

The certificate files must be on the Podman host machine in the directory specified by --tls-volume|-T.

At a minimum, you must supply an X.509 formatted PEM signing certificate named tls.crt and the PEM formatted signing key named tls.key.

When installing Envoy, you should also provide an envoy.pem file if you want Envoy to be reachable on a different hostname than the rest of the Infra pod's services. If you do not provide this file, start_command.sh creates it in the TLS directory from tls.crt and tls.key, so Envoy uses the same signing certificate as the Deephaven services.

Supplying trusted certificates to Podman

Provide the path on your Podman host machine to your X.509 bundle of trusted public certificates using the --ca-cert-file|-c flag. Deephaven transforms your certificate bundle into a truststore-iris.p12 truststore, and forces all TLS verification to use your CA certificates.

If you do not pass --ca-cert-file, no truststore is generated and all clients and servers rely on system trust materials, the operating system, Java, and Python bundles of publicly trusted CA certificates, for TLS hostname verification.

Caution

Supplying a truststore is optional only if your certificates were issued by a publicly trusted CA, or you built a custom image that embeds your CA. If your certificates were issued by an internal PKI (a private or corporate CA, which includes every certificate produced by create_dh_cert.sh) and you are running the pre-built images, you must pass --ca-cert-file|-c. Nothing in the system truststore knows your CA, so without it every internal TLS connection fails verification: the authentication server exits with PKIX path building failed: unable to find valid certification path to requested target, and the cluster stays at INITIALIZING.

Important

--ca-cert-file is not remembered between runs. It sets an environment variable on the container, so you must pass it on every start_command.sh invocation that creates a pod, including --delete-pod and upgrades. Omitting it on such a run rebuilds the deployment without a truststore and reintroduces the failure above. To avoid having to pass it every time, embed the CA into your images with build_images.sh --ca-cert-file instead.

Persisting authentication certificates

The /etc/sysconfig/deephaven/auth directory (linked from /etc/sysconfig/illumon.d/auth) is inside the container. It is not on a volume, so files you add to it, such as SAML or Active Directory certificates, are lost whenever the pod is recreated with --delete-pod or --upgrade.

To persist authentication files, keep them in a host directory and mount it in the infra pod with --additional-volume, described under Volumes. Only the authentication server, which runs in the infra pod, reads them, so query pods do not need the mount:

Like every other start_command.sh option, pass --additional-volume each time you create the infra pod. Then update the relevant properties to reference the new path:

FilePropertyExample
SAML IdP certificateauthentication.samlauth.idp.x509cert.file/deephaven-saml/idp-cert.pem
Active Directory trust storeauthentication.server.ldap.truststore.ad.path/deephaven-saml/ad-truststore.jks
SAML private keyauthentication.samlauth.sp.privatekey.file/deephaven-saml/sp-private-key.pem

The SAML private key is needed only if Deephaven signs or encrypts its SAML messages. Do not put private keys in the java_lib volume: every Deephaven process can read it, including query workers. In images built without --use-non-root-accounts, every process in the pod runs as the same account, so file permissions cannot keep a key in the infra pod from that pod's own workers either.

In images built with --use-non-root-accounts, the authentication server runs as the admin account (irisadmin unless DH_ADMIN_USER was set for the build), so a private key that only its owner can read must belong to that account. With rootless Podman, change the owner inside Podman's user namespace; with rootful Podman, use sudo chown with the same IDs:

Certificates are not secret and can stay readable by every account.

For example, change:

to:

For more information on LDAP trust stores, see Adding custom trust stores.

Build images

Prerequisites to build images

The build_images.sh script

Below is an example of how to build the container images. The example command must be run from the extracted deephaven-podman-<version> directory (e.g. deephaven-podman-2026.01.067); this is the directory that contains the build_images.sh and start_command.sh scripts.

When running build_images.sh, you must specify the location of the Deephaven installation artifacts to use. These can be specified via URLs or included in the dh-base/ subdirectory. See dh-base build arguments for more details.

If Envoy is configured, the Envoy binary must be placed in the dh-infra/ subdirectory. Envoy requires Rocky 9 or RHEL 9 as the base image family.

To add your own privately trusted certificate to the image's system truststore and Python environments, pass --ca-cert-file /path/to/ca.crt to build_images.sh. This embeds trust materials into the image, so you do not need to pass them every time you invoke start_command.sh.

In this example, the prerequisite Deephaven installation archive, Core+ installation archive, etcd installation archive, and Deephaven installer JAR have already been downloaded and copied to the dh-base directory. This command line also specifies a particular Python package to install, a particular Rocky 9 base image to use, and to use the Java 17 JDK.

The images can be built by running build_images.sh, which builds the dh-base/dh-stage/dh-infra/dh-queryserver images and tags them with those same names.

Built images can be viewed with podman image ls:

Other tested base image families are Rocky 9, RHEL 8, and RHEL 9. Using something other than these is not currently supported and may require significant changes to Containerfiles and dh-init.sh and related scripts.

Plugins

Plugins can be included in the images by placing the plugin archives into dh-stage/plugins/. The plugins to be installed must be listed in the plugins manifest file, dh-stage/plugins/plugins.txt. The SAML plug-in is built into the product itself.

The plugins manifest file contains entries with the following format:

FieldDescription
<plugin-target-dir>The target directory in /etc/sysconfig/deephaven/plugins into which this plugin will be installed. This must match the plugin's expected directory name, as included in the scripts built into the plugin archive (e.g., samlAuth for the DH SAML plugin).
<plugin-file>The filename of the plugin tar archive in dh-stage/plugins/.
<properties-file> (Optional)An optional file in dh-stage/plugins/ that includes additional properties to add to the system after installing the plugin.

If the <properties-file> is specified, it is appended to iris-environment.prop after the plugin has been installed. (The properties are not added until after the plugin has been installed to ensure that the plugin's files are available on the classpath.) A plugin's properties file is added to the configuration only the first time a cluster is initialized; if the container detects an existing configuration during initialization, it does not append the properties. This prevents duplication of the properties appended to iris-environment.prop.

Lines beginning with # are treated as comments and are ignored.

Custom JAR files

Custom JAR files can be added to a Podman deployment either at runtime (by mounting a volume) or at build time (by including them in the image). There are two locations for custom JARs:

  • Enterprise JARs (/etc/sysconfig/illumon.d/java_lib): Custom JARs for Deephaven Enterprise services, such as custom tailer configurations, authentication plugins, or other extensions.
  • Core+ JARs (/usr/illumon/coreplus/latest/custom_lib): Custom JARs for Core+ workers, such as custom libraries or dependencies for Persistent Queries.

Runtime: mount a volume

To add custom JARs at runtime, mount a host directory containing the JAR files as a volume when starting the pod.

For Enterprise JARs, use the --java-lib-volume option:

For Core+ JARs, use the --additional-volume option to mount a directory to /usr/illumon/coreplus/latest/custom_lib:

When preparing host directories for these volumes, ensure that the directory and its contents are owned by the user that will be starting the container. If SELinux is in use, set the SELinux labels with:

Build time: include in the image

To include custom JARs in the image, edit dh-stage/Containerfile to copy the JAR files into the appropriate directory.

  1. Create a directory for your custom JARs in the dh-stage/ directory:

  2. Copy your JAR files into the appropriate directory.

  3. Edit dh-stage/Containerfile to add COPY and RUN instructions. Add these lines at the end of the file (after all existing RUN commands):

    Note

    The Enterprise JARs are copied to /illumon-d-java_lib/ (the persistent mountpoint) rather than /etc/sysconfig/illumon.d/java_lib/. During container initialization, the init hooks replace /etc/sysconfig/illumon.d/java_lib with a symlink to /illumon-d-java_lib. Copying directly to the mountpoint ensures the JARs survive this symlink swap and seed the volume.

  4. Rebuild the images using build_images.sh.

Note

When building custom JARs into the image, the JARs are available on all nodes without requiring shared storage. However, updating the JARs requires rebuilding and redeploying the images. For JARs that change frequently, consider using the runtime volume mount approach instead.

External storage

Paths on the host that are connected to volumes can, in turn, be mapped or linked to network storage. NFS should not be used for intraday data (/db/Intraday and /db/IntradayUser) or for binary log files such as /var/log/deephaven/binlogs, but NFS, or a similar shareable network storage solution, is required to share historical data under /db/Systems across nodes of a cluster, and can also be used to share historical data between multiple Deephaven installations.

Requirements for access to storage

Single-user installation

When running a single-user installation, all access operations originating in the pod are run on the host in the context of the user who ran the start_command.sh. Inside the pod this user is root/root (the root user and root default group). For such installations, Deephaven services in the pod have access to any resources at the host that the Podman user has access to. In many cases, full control access will be required for the running user, as Deephaven does things like renaming directories during merge operations and replacing links when rolling log files.

Because that user is the Podman user itself rather than a translated account, the subuid and subgid offsets described below do not apply to a single-user installation. Its files on a shared volume still carry the Podman user's own UID, so give that user the same UID on every host that shares the volume.

Multi-user installation

In a multi-user installation, separate user and group accounts are used for different services and tasks inside the pod. These users and groups are translated to host-context UIDs and GIDs based on configured subuids and subgids on the host.

The translated UIDs and GIDs must be granted permissions to give the pod account access to resources attached to the host. The subuid and subgid files have entries for the name of each user who has authority to translate accounts from containers, along with an offset and a range of IDs to allocate.

For example, an entry like podman_user:100000:10000 allows podman_user to run containers that include their own internal user IDs, with the internal IDs as high as 10000. The translated ID on the host will be the internal UID plus the offset, minus one.

With this configuration, the default Deephaven dbmerge account, which has the default ID of 9001, will be represented on the host as UID 109000. When attaching volumes that are hosted off the host (e.g., NFS or SAN), check with your storage admin as to how best to grant the needed permissions to your translated user and group accounts. One option to make this easier to manage is creating local accounts on the host to represent the translated IDs.

Important

When several hosts share a volume, give the Podman user the same offset and range on each of them. The translated UID is what is recorded on the shared storage, so a file written by dbmerge on a host using offset 100000 is owned by 109000, which a host using a different offset cannot map back to an account. Deephaven services on that second host see the files as nobody and fail to start.

Linux useradd and groupadd commands allow you to specify what UID or GID to use when creating an account. Although this is theoretically possible on OSX systems, support for it under the newer sysadminctl command is unclear. Check with your Apple support team to create accounts with specific IDs on an OSX system.

Start the cluster

The start_command.sh script creates a new pod with the required ports exposed, adds a container created from the appropriate image (either dh-infra or dh-queryserver), and starts the container. If a pod of the same name already exists, the script starts that pod instead; with --delete-pod or --upgrade, it removes the existing pod and creates a new one.

When start_command.sh creates a pod, the pod's ENTRYPOINT runs dh-init.sh, which runs the Deephaven installation or upgrade process for the node; on the infrastructure node, it also sets up the cluster's shared configuration.

Important

Use start_command.sh only to create a new cluster, to change start_command.sh options (such as mounted volumes), or to upgrade the cluster with --upgrade. To stop and resume a working cluster, use podman pod stop and podman pod start, which are much quicker:

start_command.sh refuses to start a pod from an image whose Deephaven version differs from the deployed version unless --upgrade is passed. Like --delete-pod, --upgrade deletes the pod and creates a new one: the node is down while the installation runs again, which takes several minutes, and anything stored inside the pod rather than on a volume is deleted. On the infra pod, --upgrade also moves the generated configuration in the shared configuration volume into an upgrade-backup-<timestamp> directory there, which holds private keys, and generates it again. etcd data and mounted volumes are kept.

Note

The dig utility is used to resolve the IPv4 address from the URL name of the Deephaven server (passed as -h to start_command.sh). This will fail on systems where dig is not installed. A work-around to installing dig is to set the POD_IP environment variable before calling start_command.sh:

Single-node clusters

Single-node clusters (where only one node runs all the Deephaven services) do not need shared storage and can be started with a single invocation of start_command.sh.

Before the first start, put the host's certificate and key in the TLS directory as tls.crt and tls.key, as described in Supplying signing certificates to Podman.

This example runs from the directory where the start_command.sh has been extracted:

Where the arguments are:

  • -e 8000: connect and forward host port 8000 to the pod's Envoy service.
  • -h <URL name of the DH server>: the name by which you connect to the server, which is the same as the name in the Web certificate, such as mydeephaven.myorg.com.
  • -t infra: the type of image to run, infra in this case.
  • -T "${VOLUME_BASE_DIR:?}/deephaven-tls/": path to the TLS certificate files.
  • -c "${VOLUME_BASE_DIR:?}/deephaven-tls/ca.crt": path to the CA certificate that signed the TLS certificate. Required whenever the image does not already trust the issuing CA, which is the case for the pre-built images with a private or self-signed CA, including any certificate generated by create_dh_cert.sh. Omit it only if your certificate was issued by a publicly trusted CA, or if you built a custom image that embeds the CA. See Supplying trusted certificates to Podman.
  • --delete-pod: delete any existing pod with this pod name.
  • -n dh-infra: the container name, and the base name to use for the main pod (dh-infra-pod, in this case).
  • -s: single-node Envoy instance, so only the -e 8000 port is forwarded into the container.
  • -H "${VOLUME_BASE_DIR:?}/db-systems/": mount the db-systems host directory to /db/Systems in the container for persistent storage of historical data.
  • -I "${VOLUME_BASE_DIR:?}/db-intraday/": mount the db-intraday host directory to /db/Intraday in the container for persistent storage of intraday data.
  • -W, -U and -L: the same for centrally managed user tables (/db/IntradayUser), direct user tables (/db/Users) and logs (/var/log/deephaven).

Pass every volume option, with the same directories, each time you run start_command.sh for this pod: data whose option is left out is stored inside the pod and deleted when the pod is recreated.

Without Envoy, the -e 8000 and -s options would be omitted.

Multi-node clusters

A multi-node cluster has one infra pod and one or more query pods, each on its own host. Every host needs:

  • The same shared configuration directory (-C, default "${VOLUME_BASE_DIR:?}"/deephaven-shared-config), historical data directory (-H) and direct user table directory (-U), on shared storage.
  • Its own TLS certificate for its own hostname in its TLS directory (-T), signed by the same CA, and that CA passed with -c.

On the infra host, list every query pod with -q <pod name>=<hostname>. A query pod's name is its container name (-n) followed by -pod:

On each query host:

You can start the pods in either order. A query pod waits up to 15 minutes for the infra pod to set the COMPLETE status in the cluster_status shared configuration file, and exits if it does not.

If SELinux is in use, a query pod also cannot read the shared configuration until its MCS categories are cleared: the files the infra pod writes carry the infra pod's categories. Once the infra pod reaches COMPLETE, relabel the shared configuration on the infra host, then restart any query pod whose wait timed out:

When start_command.sh later recreates the infra pod, it clears these categories itself.

start_command.sh details

The script takes the following arguments:

ArgumentDefaultDescription
--additional-volume <options>N/A(Optional) Additional volume specifications. These flags are passed directly to the podman create command as --volume=<options>. You may need to include the :z option to configure SELinux labels appropriately. Can be specified multiple times. Example for Core+ custom JARs: --additional-volume ${VOLUME_BASE_DIR:?}/custom_lib/:/usr/illumon/coreplus/latest/custom_lib:z. See Custom JAR files for more information.
-a, --envoy-admin <port>N/A(Optional) Enable Envoy admin on specified port within the container. Only valid for "infra" pods where Envoy is enabled.
-c, --ca-cert-file <path to ca.crt>N/ARequired when the image does not already trust the CA that signed your certificates (pre-built images with a private or self-signed CA, including certificates from create_dh_cert.sh); optional only for publicly trusted CAs or images built with build_images.sh --ca-cert-file. One or more trusted certificates to verify TLS connections. A relative path is resolved from the TLS volume (-T). Must be passed on every invocation that creates a pod, including upgrades. See Supplying trusted certificates to Podman.
-d, --delete-podN/A(Optional) Whether to delete any existing pod with the same name. If not specified, an existing stopped pod is restarted. See -n for controlling container and pod name.
--discard-pod-datafalse(Optional) With -d or --upgrade, recreate the pod even though intraday data or user tables are stored inside it instead of on a host volume (-I, -W, -U), or are on a host volume the new pod would not mount. That data is deleted with the pod. Without this option, start_command.sh refuses to delete such a pod, changes nothing, and names the option that keeps the data. See Move pod data onto host volumes.
--calendars-volume <calendars dir>N/A (/etc/sysconfig/illumon.d/calendars is stored in the container)(Optional) Set the location of the volume where customer calendar definitions are stored.
-e, --envoy <port>N/A(Optional) Use Envoy on specified port within the container. Only valid for "infra" pods.
-h, --hostname <hostname>N/AHostname for the pod being created. <hostname> must resolve to an address of one of this host's network interfaces. The pod's ports are published on that address.
--java-lib-volume <java_lib dir>N/A (/etc/sysconfig/illumon.d/java_lib is stored in the container)(Optional) Set the location of the volume where custom Enterprise JARs are stored. See Custom JAR files for more information.
-n, --name <container name>dh-infra/dh-query(Optional) The container name. Defaults to dh-infra when type is "infra" and dh-query when type is "query". The pod name will be <container-name>-pod.
--nohupN/A(Optional) Use nohup when invoking podman pod start. This can be helpful to prevent the OS from removing the Podman process after the launching user has logged off the host.
-p, --port <port>N/A(Optional) Additional port numbers to expose (e.g., -p 9032 to expose port 9032 for the SAML ACS). Can be specified multiple times.
-q, --query <pod name>=<hostname>N/AA query pod and its hostname, such as -q dh-query1-pod=query1-host.mydomain.net. The pod name is the name passed with -n to start_command.sh on the query host, followed by -pod. Only applies to -t infra. Can be specified multiple times. List every query pod on every invocation passing -t infra: the supplied list is authoritative, and iris-endpoints.prop is regenerated from it whenever the infra pod is recreated (with --delete-pod or --upgrade).
-s, --single-node-envoyMulti-node(Optional) Only map Envoy port(s) into the container, for a single-node deployment. Without -s, all Deephaven service and worker ports are mapped. Ignored unless -e is also given. Only valid for "infra" pods.
-t, --type <infra | query>N/AContainer type (either "infra" or "query").
-C, --shared-config-volume <config volume dir>"${VOLUME_BASE_DIR:?}"/deephaven-shared-config(Optional) The directory (on the Podman host) containing shared configuration files.
-E, --etcd-volume <etcd dir>"${VOLUME_BASE_DIR:?}"/deephaven-etcd(Optional) The directory (on the Podman host) containing etcd data. Note that Deephaven schema, routing, and property files are centralized in etcd and shared across nodes/services.
-F, --client-files <client files dir>N/A(Optional) The directory (on the Podman host) where the DeephavenUpdater should stage files for clients.
-H, --hist-data-volume <hist data dir>"${VOLUME_BASE_DIR:?}"/db-systems(Optional) The directory (on the Podman host) where historical data is stored. If you pass -H, pass the same directory every time: a pod recreated without it mounts $VOLUME_BASE_DIR/db-systems instead of your directory.
-I, --intraday-data-volume <intraday data dir>N/A (/db/Intraday is stored in the container)(Optional) The directory (on the Podman host) where intraday data is stored.
-L, --logs-volume <DH logs dir>N/A (/var/log/deephaven is stored in the container)(Optional) The directory (on the Podman host) where Deephaven logs are stored.
-R, --container-registry <registry>localhost(Optional) The container registry used when retrieving images to run.
-T, --tls-volume <TLS dir>"${VOLUME_BASE_DIR:?}"/deephaven-tls(Optional) The directory (on the Podman host) containing TLS certificates.
-U, --users-volume <users data dir>N/A (/db/Users is stored in the container)(Optional) The directory (on the Podman host) where direct user table data is stored.
--etcd-config-overrides <file>N/A(Optional, infra only) Merge YAML overrides into generated etcd defaults and preserve them across restarts, recreation, and upgrades. Restarts an existing pod to apply them. See Customizing etcd configuration.
--keep-failed-containerfalse(Optional, infra only) Keep the container running after an installation-hook failure until you stop it, instead of exiting after ten minutes. Takes effect only when the pod is created: for an existing pod, also pass -d or --upgrade. See Troubleshooting.
--upgradeN/A(Optional) Upgrade (or reinitialize) an existing deployment. Implies --delete-pod. On infra pods, moves the regenerable shared configuration into a timestamped backup directory under the shared configuration volume (preserving dh_etcd_config.tgz, priv-keys, and etcd-overrides.yaml, so the existing etcd database is reused) and clears SELinux MCS labels. Required when the image version differs from the deployed version. See Upgrade with --upgrade.
-W, --intradayuser-volume <intradayuser data dir>N/A (/db/IntradayUser is stored in the container)(Optional) The directory (on the Podman host) where centrally managed user table data is stored.

The start_command.sh script also reads the following environment variables:

VariableDefaultDescription
IRIS_ACCT_PASSWORDN/AThe password set for the iris account after Deephaven is initialized and started.
VOLUME_BASE_DIR/container-testThe base directory from which volumes are mounted by default. Must be owned by the user starting the containers. For simplicity with SELinux, it is preferable that this be outside the home directory.
TLS_CERT_FILENAMEtls.crtThe X.509 PEM-format certificate file (in the TLS volume) that start_command.sh combines with the key to create envoy.pem when Envoy is enabled and envoy.pem does not exist. Deephaven services always use tls.crt.
TLS_KEY_FILENAMEtls.keyThe key file (in the TLS volume) used with TLS_CERT_FILENAME to create envoy.pem. Deephaven services always use tls.key.
DISABLE_SELINUX_CLIENT_FILESfalseIf set to true, the :z option is not set when creating the client files volume.
DISABLE_SELINUX_ETCDfalseIf set to true, the :z option is not set when creating the etcd volume.
DISABLE_SELINUX_INTRADAY_DATAfalseIf set to true, the :z option is not set when creating the intraday data volume.
DISABLE_SELINUX_INTRADAYUSERfalseIf set to true, the :z option is not set when creating the centrally managed user data volume.
DISABLE_SELINUX_USERSfalseIf set to true, the :z option is not set when creating the direct user data volume.
DISABLE_SELINUX_CALENDARSfalseIf set to true, the :z option is not set when creating the calendars volume.
DISABLE_SELINUX_JAVA_LIBfalseIf set to true, the :z option is not set when creating the java_lib volume.
DISABLE_SELINUX_DH_LOGSfalseIf set to true, the :z option is not set when creating the logs volume.
DISABLE_SELINUX_SHARED_CONFIGfalseIf set to true, the :z option is not set when creating the shared configuration volume.
DH_CA_CERTIFICATEN/A(Optional) The X.509 PEM-format CA bundle used to build the truststore, which then replaces the system truststore for nearly all TLS verification. --ca-cert-file sets it, and overrides a value from the environment.
POD_IPResolved from -h with digThe IPv4 address of this host that the pod's ports are published on. Set it if dig is not installed.

Customizing etcd configuration

Caution

etcd overrides are not validated: only their YAML syntax is checked. A wrong setting can stop etcd from starting, disconnect every Deephaven process, or expose data. Never change the cluster identity (name, initial-cluster*), data-dir, listen or advertise URLs, TLS, or authentication of an existing cluster. Keep a known working overrides file and a backup of the deployment.

To change etcd settings, add --etcd-config-overrides <file> to the infra node's usual start_command.sh command, keeping all its other options. The file holds only the settings you want to change. For example:

Any etcd configuration option can be set. Your values are merged into the configuration that Podman generates for the deployment: nested mappings merge, and other values, including lists, replace the generated value. Start from the etcd-overrides.yaml.example file in the distribution. To see the generated defaults, including your deployment's cluster ID, addresses, and certificate paths, copy them out of the infra container. Until overrides are applied, config.yaml is the generated configuration; after that, the generated version is kept as config.yaml.defaults:

Do not copy the whole generated file into your overrides; keeping only your changes lets later upgrades supply new defaults. Do not edit the generated config.yaml or dh_etcd_config.tgz directly.

start_command.sh checks the YAML syntax before it stops an existing pod, then saves a copy of your file as etcd-overrides.yaml in the shared configuration volume. That copy is reapplied every time the container starts, including after recreation and upgrade, even when later commands omit the flag. Editing or deleting your original file has no effect. On an existing pod, the command stops and restarts the pod so that etcd reads the new settings. --delete-pod and --upgrade also accept the flag.

To change or remove overrides, rerun the command with a new file. The replaced copy is kept as etcd-overrides.yaml.previous; to undo a change, rerun the command with that file. An empty file, or one that contains only comments, restores the generated defaults. The configuration etcd actually uses is /etc/etcd/dh/latest/config.yaml in the infra container.

etcd quota alarms and logs

If etcd reports a NOSPACE alarm, follow Recovering etcd after a quota alarm.

The etcd log is /var/log/deephaven/etcd/etcd.log in the infra container; its launcher writes launcher.log in the same directory. Each container start rotates both logs and keeps the previous five as .1 through .5. Logs are not rotated by size; to do that, configure etcd's own logging through overrides. Mount a host log directory with -L to keep the logs when the container is recreated.

Changing properties

Most configuration of Deephaven Podman deployments is the same as for traditional deployments. Tasks like editing properties files or routing YAML are accomplished using the dhconfig tool. A key point to remember is that Deephaven-specific configuration changes do not require restarting the entire system; configuration changes that affect the behavior of a Deephaven service typically require just the service in question to be restarted with dh_monit.

For changes that affect configuration used by multiple services, it is recommended to /usr/illumon/latest/bin/dh_monit restart all, and then monitor service states as they restart with watch /usr/illumon/latest/bin/dh_monit summary.

Most property changes that affect only query workers will be in effect for any workers started after the change has been imported, without any service restarts needed.

Adding your own properties

Put your own properties in iris-environment.prop. Upgrades keep your settings and change only the obsolete Grizzly TLS assignments described in Upgrading from Grizzly, saving the original first. Podman generates iris-endpoints.prop during initialization and upgrade and loads it after iris-environment.prop, so a generated endpoint setting overrides the same property in your file.

Properties are stored in etcd, not in a file on the host, so changing them is an export, edit, and import cycle. Run these on the Podman host as the user that owns the deployment. podman cp brings the file out of the container so you can edit it with your own tools, and puts it back:

Edit ./iris-environment.prop, then import it:

Important

Do not skip the podman cp steps by editing the file inside the container. The container's filesystem is discarded when the pod is recreated, which every upgrade does, so a file edited there is lost unless it has been imported into etcd.

Restart the affected services with dh_monit as described above. Do not set properties that iris-endpoints.prop sets under ### Overrides for podman deployment:; the generated values take precedence.

Caution

Do not recreate the pods (start_command.sh with --delete-pod or --upgrade) while the configuration prevents Deephaven services from starting or staying up. Recreating a pod reruns the installation, which fails with that configuration and leaves the cluster unusable. Fix the configuration first.

Add nodes

Nodes can be added by updating the iris-endpoints.prop file to include the new node. The configuration changes to define a new pod, below, show the required changes.

After importing the updated iris-endpoints.prop file, the new node can be used after reloading the Controller process's configuration:

Also add the new pod to the -q list of your infra start_command.sh command. iris-endpoints.prop is regenerated from that list whenever the infra pod is recreated, and every query pod that is not passed with -q is removed from the cluster configuration.

A new node's certificates must be signed by the same CA provided when creating the infra node, as with the existing nodes.

Configuration changes to define a new pod

The following two sections demonstrate the 'before and after' of configuration changes to iris-endpoints.prop to add a pod to a running cluster.

The iris-endpoints.prop file can be exported by connecting to the infra pod (e.g. podman exec -it dh-infra bash) and running:

After making the requisite changes to the iris-endpoints.prop file, the updated properties can be reimported with:

Example iris-endpoints.prop before adding a query pod
Example iris-endpoints.prop after adding a query pod

The configuration below shows how to modify the above example to add a new host (in this case, rhel8-test-query2.devrel.deephaven.io, run as dh-query2-pod). Note the three critical changes to iris-endpoints.prop:

  • New dispatchers defined (iris.db.5 and iris.db.6). Note that both a query dispatcher and a merge dispatcher are added for the new host (rhel8-test-query2.devrel.deephaven.io).

  • The iris.db.nservers property is updated to match the number of iris.db.<N> sections.

  • An additional host-scoped configuration section is added for the new pod, informing the query dispatcher of its external-facing hostname. If this step is skipped, the dispatcher will be unable to communicate with etcd. This may manifest in errors such as the following:

Additionally, be aware that Persistent Queries are assigned to a server based on the server ID number (e.g. 3 in iris.db.3). Changing a server (e.g. changing iris.db.3.host=host3.mycompany.net to iris.db.3.host=host6.mycompany.net) will cause PQs that were running on host3.mycompany.net to be assigned to host6.mycompany.net.

Architecture

Deephaven's Podman deployment model uses four images:

  1. dh-base contains the base Linux system and several prerequisites.
  2. dh-stage contains staged-but-not-initialized etcd and Deephaven installations.
  3. dh-infra contains the scripts required to initialize a cluster and start the required processes for an infrastructure node.
  4. dh-queryserver is also built from dh-stage (not from dh-infra) and contains the scripts required to start the required processes for a query server node.

The images are used in conjunction with the start_command.sh script and a directory structure on the host.

The directory structure is based at a path represented by the VOLUME_BASE_DIR environment variable used in Deephaven Podman scripts. This directory structure allows for shared configuration between nodes in the cluster, and persists configuration and data so Deephaven Podman deployments can be upgraded or redeployed (if needed) without losing their configuration.

The start_command.sh script validates arguments and creates Podman command lines to manage containers and pods. When it creates the dh-infra pod for the deployment, the entrypoint of the image's Containerfile effectively runs an installation or upgrade of Deephaven within the pod. Configuration for the cluster is persisted on the host in volume subdirectories. Various types of Deephaven data can also be persisted in mounted volumes. Because configuration (and, usually, data) are persisted, the pods can be recreated with different Podman options while maintaining state, and recreated from newer images to upgrade the deployment.

Volumes, files, and configuration

Volumes

The following directories (inside the containers) are configured as Podman volumes:

PathShared across pods?Pod typesPurpose
/db/IntradayNOInfra only(Optional) Intraday data. Only mounted on infra pod. The underlying storage for this volume should not be NFS.
/db/SystemsYESAll (infra/query)Historical data. This is shared across all pods. The underlying storage for this volume is typically NFS.
/db/UsersYESAll (infra/query)(Optional) Direct user table data. This volume may be stored on NFS.
/db/IntradayUserNOInfra only(Optional) Centrally managed user table data. Only mounted on infra pod. The underlying storage for this volume should not be NFS.
/deephaven-etcdNOInfra onlyThe etcd data and the etcd server configuration. The /var/lib/etcd and /etc/etcd directories are linked to /deephaven-etcd. Only mounted on infra pod. Note that etcd contains Deephaven schemas, Persistent Queries, access controls, and configuration files (i.e. .prop files).
/deephaven-client-filesNOInfra onlyStaging files (e.g., on NFS) for clients, so that clients do not need to run DeephavenUpdater themselves.
/deephaven-shared-configYESAll (infra/query)Deephaven configuration files and private keys. Mounted read-write on the infra pod and read-only on query pods. There are links in /etc/sysconfig/deephaven to individual shared configuration directories. See below for additional details.
/illumon-d-calendarsYESAll (infra/query)(Optional) Custom calendar files. This should be shared across pods.
/illumon-d-java_libYES*All (infra/query)(Optional) Custom JAR files and other files to persist across restarts. *This can optionally be shared across pods, depending on whether all pods should have access to the same contents.
/source-tlsNOAll (infra/query)TLS key and certificate (tls.key and tls.crt) for the host. The CA bundle passed with --ca-cert-file is copied to ca/ca.crt in this volume.
/var-log-deephavenNOAll (infra/query)(Optional) Log files, mounted with -L and linked from /var/log/deephaven. This volume may be stored on NFS. The binary log directory, /var/log/deephaven/binlogs, is a link to /var-log-deephaven-binlogs, so binary logs are not written to this volume.
/var-log-deephaven-binlogsNOAll (infra/query)(Optional) Binary log files that the tailer has not yet sent, and their recent history. There is no start_command.sh option for it: to keep binary logs when the pod is recreated, mount a host directory with --additional-volume <dir>:/var-log-deephaven-binlogs:z. This volume must not be stored on NFS.

Additional volumes can be mounted using the --additional-volume flag to start_command.sh. Multiple instances of this flag can be used.

Two special volumes can be attached to persist, and possibly share, the contents of illumon.d/java_lib and illumon.d/calendars. When preparing host directories for these volumes, ensure that the directory and its contents are owned by the user that will be starting the container, as the initialization process will change their ownership to the $DH_ADMIN_USER:$DH_ADMIN_GROUP that is configured in the container. If SELinux is in use, then the volumes should also have their SELinux labels set with:

Note

You can persist tailer config XML files for custom tailer configurations by adding a volume for illumon.d/java_lib using the --java-lib-volume start_command.sh option. The tailer process only scans JAR files in illumon.d/java_lib, so you must package your XML config files into a JAR before placing them in this directory:

For more information about adding custom JARs, including Core+ custom libraries, see Custom JAR files.

In-worker Data Import Servers

A Persistent Query that runs an in-worker Data Import Server, such as Kafka ingestion or a Core+ Data Import Server, writes intraday data to its own storage directory, usually /db/dataImportServers/<disName>. That directory is not a volume: it is stored inside the pod and deleted when the pod is recreated. Mount a host directory there with --additional-volume on every pod whose workers run an in-worker Data Import Server, and pass it each time you run start_command.sh for that pod. Like /db/Intraday, keep this storage off NFS for performance: a Data Import Server needs local or block storage with low-latency I/O.

Then create each Data Import Server's directory in the pod. In a multi-user deployment, the directory must belong to the merge account (dbmerge and dbmergegrp unless you changed them for the build); in a single-user deployment, leave out the chown:

Shared configuration

The /etc/sysconfig/deephaven directory consists of some items that must be shared across nodes in a Deephaven cluster (i.e., pods in a containerized deployment with podman; servers in a non-containerized deployment). In a non-containerized Deephaven installation, the appropriate files are automatically copied between nodes during the installation process. Since this is not feasible in a containerized deployment, the required shared files are stored in a directory mounted as a volume by all containers in the cluster. The infra container mounts this shared configuration volume read-write; query containers mount it read-only. See Shared configuration directory contents.

Appendix

Images

Images can be custom built using the build_images.sh script, or pre-built images can be downloaded from Deephaven.

dh-base

The dh-base image is built from a base Linux image (e.g., docker.io/redhat/ubi9-minimal or docker.io/amd64/rockylinux:9.3-minimal) and contains several installation files (copied in from the dh-base directory) as well as prerequisites installed via dnf.

Base image build arguments

The following build arguments influence how dh-base is built:

Example --build-arg commandDescription
--build-arg BASE_IMAGE=<my-image-name>Sets the base image from which the dh-base image is built. (Default is docker.io/amd64/rockylinux:9.3-minimal.)
--build-arg PACKAGE_SOURCE=localInstructs the build to copy the Deephaven, installer, Core+ and etcd files from the dh-base directory.
--build-arg PACKAGE_SOURCE=urlInstructs the build to download the Deephaven, installer, Core+ and etcd files from URLs.
--build-arg DH_TAR_URL=<url-for-deephaven-installation-tar>Sets the URL to download the Deephaven binaries from (when PACKAGE_SOURCE is url).
--build-arg DH_INSTALLER_URL=<url-for-deephaven-installer-jar>Sets the URL to download the Deephaven installer JAR from (when PACKAGE_SOURCE is url).
--build-arg COREPLUS_TAR_URL=<url-for-coreplus-tar>Sets the URL to download the Core+ tar from (when PACKAGE_SOURCE is url).
--build-arg ETCD_TAR_URL=<url-for-etcd-package>Sets the URL to download etcd from (when PACKAGE_SOURCE is url).
--build-arg DH_TAR_FILE=<file-for-deephaven-installation-tar>Sets the file name (in the dh-base directory) to use as the Deephaven installation archive (when PACKAGE_SOURCE is local). Example: deephaven-enterprise-2026.01.067.tar.gz.
--build-arg DH_INSTALLER_JAR=<file-for-deephaven-installer-jar>Sets the file name (in the dh-base directory) to use as the Deephaven installer JAR (when PACKAGE_SOURCE is local). Example: Installer-2026.01.067.jar.
--build-arg COREPLUS_TAR_FILE=<file-for-coreplus-tar>Sets the file name (in the dh-base directory) to use as the Core+ tar (when PACKAGE_SOURCE is local). Example: deephaven-coreplus-42.9-2026.01.067.tgz.
--build-arg ETCD_TAR_FILE=<file-for-etcd-package>Sets the file name (in the dh-base directory) to use for the etcd installation (when PACKAGE_SOURCE is local). Example: etcd-v3.5.17-linux-amd64.tar.gz.
--build-arg JDK_PACKAGE_NAME=<jdk-package-name>Sets the JDK package to install with dnf. build_images.sh --jdk17 sets temurin-17-jdk.
--build-arg JDK_REPO_BASEURL=<url>The baseurl of the RPM repository the JDK is installed from. Default: the Eclipse Adoptium (Temurin) repository.
--build-arg JDK_REPO_GPGKEY=<url>The GPG key URL for that repository. Default: the Adoptium public key.
--build-arg PYTHON_PACKAGE=<python-package-name>An additional Python package to install with dnf.
--build-arg ADD_EPEL_REPO=<Y or N>Determines whether the EPEL repo is added to the image. Setting this to N or n disables adding the EPEL repo. All other values are treated as Y. (Default is Y.)
--build-arg ADD_DEBUG_TOOLS=<Y or N>Determines whether additional debug packages are added to the image. Setting this to N or n disables adding additional debug packages. All other values are treated as Y. (Default is Y.)

These arguments can be configured automatically by build_images.sh.

Base image files
  • The deephaven-enterprise-<VERSION>.tar.gz file is copied to /dh/container-setup-files/deephaven-enterprise.tar.gz.
  • The Installer-<VERSION>.jar file is copied to /dh/container-setup-files/Installer.jar.
  • The Core+ tar file is copied to /dh/container-setup-files/coreplus/.
  • The etcd-<VERSION>-linux-amd64.tar.gz file is copied to /dh/container-setup-files/etcd/etcd.tar.gz.
Packages

The following are installed with dnf (microdnf installs dnf first if the base image lacks it):

  1. subscription-manager, if the base image lacks it.
  2. The EPEL repo, unless the image is built with --build-arg ADD_EPEL_REPO=N. The repo is installed from https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm or https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm, depending on OS major version (8 or 9).
  3. Updates to all installed packages (dnf -y update).
  4. Deephaven's OS dependencies:
    • On both OS major versions: findutils, vi, initscripts, monit, ed, wget, unzip, glibc, libgcc, libgomp, libstdc++, zeromq-devel, bzip2, git, which, openssl, time, rsync, diffutils, python3-virtualenv and python3-rpm-generators.
    • On OS major version 8: Python 3.10 (python310 with its -devel, -libs, -pip, -rpm-macros and -setuptools packages), redhat-lsb-core, and rsync 3.1.
    • On OS major version 9: Python 3.12 (python3.12), the system Python 3 (python3 with the same companion packages), tar, hostname and cronie.
  5. sudo
  6. The JDK, from the repository set by JDK_REPO_BASEURL. The package is determined by the JDK_PACKAGE_NAME build argument. (See base image build arguments for more details.)
  7. Helpful packages for debugging from within the container, unless the image is built with --build-arg ADD_DEBUG_TOOLS=N:
    • bind-utils (provides dig)
    • net-tools (provides netstat)
  8. The package named by PYTHON_PACKAGE, if set.

dh-stage

The dh-stage image is built from dh-base and has both Deephaven and etcd installed, but not initialized. The general steps to build dh-stage are:

  1. Copy in the script with the common functions for container initialization (dh-init-common.sh).
  2. Copy in the common cluster.cnf.base file, which contains environment variables that influence installation scripts. This is customized for the initial cluster layout (i.e., the specific infra/query nodes and their hostnames) when dh-init.sh runs (when the infra container first initializes). The final, customized cluster.cnf is stored in shared configuration and used by the other nodes as they initialize.
  3. Update /etc/sudoers with the required permissions for the Deephaven user accounts.
  4. Extract the Deephaven tar archive (to /usr/illumon and /etc/sysconfig/deephaven).
  5. Create several required directories and links.
  6. Create the required user accounts and groups.
  7. Extract the etcd tar and install etcd to /usr/bin/.
  8. Add the DH username variables to cluster.cnf.base.
  9. Set the OS name and version string in cluster.cnf.base (based on values extracted from /etc/os-release).
  10. Install Python, the Legacy Python virtual environment, and Core+.
  11. Copy in any plugins (from dh-stage/plugins/ to /dh/container-setup-files/plugins). The plugins are extracted and installed during container initialization.
Stage image build arguments

The following build arguments influence how dh-stage is built:

Example --build-arg commandDescription
--build-arg MONIT_USER=rootSets the account used to run the monit process. Default is root.
--build-arg MONIT_GROUP=rootSets the group account used to run the monit process. Default is root.
--build-arg ADMIN_USER=rootSets the account used to run admin processes. Default is root.
--build-arg ADMIN_GROUP=rootSets the group account used to run admin processes. Default is root.
--build-arg MERGE_USER=rootSets the account used to run merge processes. Default is root.
--build-arg MERGE_GROUP=rootSets the group account used to run merge processes. Default is root.
--build-arg QUERY_USER=rootSets the account used to run query processes. Default is root.
--build-arg QUERY_GROUP=rootSets the group account used to run query monit processes. Default is root.
--build-arg SHARED_GROUP=rootSets the group account used to obtain access to files shared by the admin, merge, and query users. Default is root.
--build-arg SHARED_QUERY_GROUP=rootSets the group account used for per-user workers to be able to run Deephaven workers. Default is root.
--build-arg ETCD_USER=rootSets the account used to run the etcd process. Default is root.
--build-arg JDK_VERSION=jdk17The Java version, written without the jdk prefix to /usr/illumon/latest/etc/JAVA_VERSION. Set from the build_images.sh --jdk* option.
--build-arg PIP_INDEX_URL=<url>The package index pip uses during the build and in the image. Set by build_images.sh --pip-index-url.
--build-arg PIP_TRUSTED_HOST=<host>A host pip trusts during the build and in the image. Set by build_images.sh --pip-trusted-host.
--build-arg ADD_RUAMEL_YAML=<Y or N>Determines whether python3-ruamel-yaml is installed. Setting this to N or n skips it, and etcd overrides then merge without the generated configuration's comments. All other values are treated as Y. (Default is Y; build_images.sh --disable-ruamel-yaml sets N.)

These arguments are what make an image multi-user. build_images.sh --use-non-root-accounts sets them to the standard Deephaven accounts (irisadmin, dbmerge, dbquery, etcd, and their groups), or to the names in the matching DH_MONIT_USER, DH_ADMIN_USER, DH_MERGE_USER, DH_QUERY_USER, DH_ETCD_USER and *_GROUP environment variables. Without it, every account is root, as in the pre-built single-user image.

These arguments can be configured automatically by build_images.sh.

Use --use-non-root-accounts with build_images.sh to change the values of these arguments. --use-non-root-accounts instructs build_images.sh to check for account name environment variables; any that have not been set will use Deephaven defaults:

For example, if the environment variable DH_MONIT_USER is set when build_images.sh is run with --use-non-root-accounts, the set value will be used for the monit user. If it is not set, then irisadmin will be used for the monit user.

dh-infra

The dh-infra container is built from dh-stage and contains configuration files and scripts required to finalize the Deephaven installation and initialize etcd.

If USE_ENVOY=enabled, the dh-infra Containerfile copies the Envoy binary named by the ENVOY_BINARY build argument from the dh-infra directory to /usr/bin/envoy. build_images.sh --envoy-file sets that name. Envoy requires a Rocky 9 or RHEL 9 base image (--base-image).

Infra image build arguments

The following build arguments influence how dh-infra is built:

Example --build-arg commandDescription
--build-arg USE_ENVOY=disabledIndicates whether to include the Envoy reverse proxy binary. Default is no Envoy.
--build-arg ENVOY_BINARY=<file>The Envoy binary file in the dh-infra directory. Default is envoy-1.36.4-linux-x86_64.

These arguments can be configured automatically by build_images.sh.

dh-queryserver

The dh-queryserver container is built from dh-stage and contains configuration files and scripts required to start a query server node that uses the configuration initialized by the dh-infra node.

Image scripts and base configuration files

The following files are included in the dh-infra image:

Path in dh-infraPath in containerPurpose
dh-init.sh/dh/bin/dh-init.shContainer entrypoint. Configures Deephaven (including by calling preinstall-hook.sh) and starts monit, which starts the Deephaven processes.
preinstall-hook.sh/dh/bin/preinstall-hook.shDeephaven cluster initialization script for containerized environments.
migrate-podman-properties.sh/dh/bin/migrate-podman-properties.shWhen an existing configuration is reinitialized, removes TLS settings left behind by a Grizzly Podman deployment: it comments out the keystore settings that point at /deephaven-tls/keystore.p12, which current deployments do not create, and points the SAML keystore settings at the web services keystore. Only those exact Grizzly values are changed, and the previous iris-environment.prop is saved first.

The two configuration files a Podman cluster runs on are not shipped in the dh-infra image. cluster.cnf.base comes from dh-stage (see below) and is completed during the image build. iris-endpoints.prop is generated at initialization: dh-init.sh calls /usr/illumon/latest/install/set_iris_endpoints_prop.sh with the pod hostnames, then appends the Podman deployment overrides to it. Neither is edited in the image.

The following files are included in the dh-queryserver image:

Path in dh-queryserverPath in containerPurpose
dh-init-queryserver.sh/dh/bin/dh-init-queryserver.shContainer entrypoint. Configures Deephaven (including by calling preinstall-hook-queryserver.sh) and starts monit, which starts the Deephaven processes.
preinstall-hook-queryserver.sh/dh/bin/preinstall-hook-queryserver.shDeephaven host initialization script for query server containers.

The following files are included in the dh-stage image (and thus are included in both dh-infra and dh-queryserver):

Path in dh-stagePath in containerPurpose
dh-init-common.sh/dh/bin/dh-init-common.shContains common functions used by both dh-infra and dh-queryserver.
dh-etcd-common.sh/dh/bin/dh-etcd-common.shContains the functions that configure and start etcd, including applying etcd overrides.
merge-etcd-config.py/dh/bin/merge-etcd-config.pyValidates etcd overrides and merges them into the generated etcd configuration.
dhc-common.sh/dh/bin/dhc-common.shContains the functions that configure Python and install Core+.
cluster.cnf.base/dh/cluster.cnf.baseBase installation configuration; the image build appends the account and version settings to it.

build_images.sh

This script automates the process of calling podman build for each of the containers (including setting build arguments and tagging the resulting images).

The script takes the following arguments:

ArgumentDefaultDescription
--base-image <image>docker.io/amd64/rockylinux:9.3-minimal(Optional) The base image to use for the dh-base container. The BASE_IMAGE environment variable also sets it.
--ca-cert-file <file>N/A(Optional) A PEM-format CA certificate bundle to add to the image's system truststore and Python environments, so that start_command.sh does not need --ca-cert-file.
--coreplus-tar-file <file>N/A(Required) Give all four --*-file options or all four --*-url options. The path (relative to the dh-base Containerfile) of the Core+ tar file.
--coreplus-tar-url <URL>N/A(Required) Give all four --*-file options or all four --*-url options. The URL from which to download the Core+ tar file.
--dh-file <file>N/A(Required) Give all four --*-file options or all four --*-url options. The path (relative to the dh-base Containerfile) of the Deephaven installation archive.
--dh-url <URL>N/A(Required) Give all four --*-file options or all four --*-url options. The URL from which to download the Deephaven installation archive.
--disable-debug-toolsfalse(Optional) Whether to disable the installation of debug tools in the dh-base image.
--disable-epelfalse(Optional) Whether to disable the installation of the Extra Packages for Enterprise Linux repository in the dh-base image.
--disable-ruamel-yamlfalse(Optional) Do not install python3-ruamel-yaml in the dh-stage image, for sites that do not want the dependency or whose package mirror does not provide it. etcd overrides still apply, but the merged configuration loses the generated file's comments.
--envoy-file <file>envoy-1.36.4-linux-x86_64(Optional) The path (relative to the dh-infra directory) of the Envoy binary file. Used with --include-envoy.
--etcd-file <file>N/A(Required) Give all four --*-file options or all four --*-url options. The path (relative to the dh-base Containerfile) of the etcd tar file.
--etcd-url <URL>N/A(Required) Give all four --*-file options or all four --*-url options. The URL from which to download the etcd tar file.
--include-envoyfalse(Optional) Whether to include the Envoy binary in the dh-infra image and configure it to run as a service. This option requires a RHEL 9 or Rocky 9 image as the --base-image.
--installer-file <file>N/A(Required) Give all four --*-file options or all four --*-url options. The path (relative to the dh-base Containerfile) of the Deephaven installer JAR.
--installer-url <URL>N/A(Required) Give all four --*-file options or all four --*-url options. The URL from which to download the Deephaven installer JAR.
--jdk17 | --jdk21 | --jdk25--jdk17(Optional) Which Java JDK version to install. Use at most one of these or --jdk-package.
--jdk-package <JDK package name>temurin-17-jdk, temurin-21-jdk, or temurin-25-jdk(Optional) The name of the JDK package to install (using DNF) when building the base image. Without it, the package corresponds to the --jdk* flag used.
--jdk-repo-baseurl <URL>Eclipse Adoptium RPM repository(Optional) The repository to install the JDK from.
--jdk-repo-gpgkey <URL>Adoptium public GPG key(Optional) The GPG key for --jdk-repo-baseurl.
--python-package <Python package name>None(Optional) The name of an additional Python package to install (using DNF) when building the base image. The chosen package must be available for the chosen base image. (For example, package names changed between Rocky 8/RHEL 8 and Rocky 9/RHEL 9.)
--pip-index-url <URL>N/A(Optional) The package index pip uses, for example a local cache.
--pip-trusted-host <host>N/A(Optional) A host pip trusts, for example a local cache.
--use-non-root-accountsfalse(Optional) Whether to use default Deephaven user and groups accounts, or values provided by environment variables, for image processes instead of running all as root (all as root is the default if this argument is omitted).
-h | --helpN/APrint usage information and exit.

Shared configuration directory contents

The following table provides an overview of the contents of /deephaven-shared-config:

ItemTypeNotes
cluster_statusFileContains a string describing the current status of the Podman deployment (e.g., INITIALIZING or COMPLETE). This refers to the state of the shared Deephaven configuration (except everything stored in etcd). It is only updated by the infra pod. Query pods wait for the cluster status to be COMPLETE before they initialize. If a default password for the iris account is configured, it is only set during cluster initialization.
dh-configDirectoryDirectory under /etc/sysconfig/deephaven that is shared across pods. See below for more details.
dh_etcd_config.tgzFileContains the etcd client configuration files used by Deephaven. This is extracted to /etc/sysconfig/deephaven/etcd on the infra node during cluster initialization (which is later moved to /deephaven-shared-config and replaced with a symlink). When redeploying a cluster (e.g., during an upgrade), this file must not be deleted, as it is re-extracted to regain access to the original etcd database.
etcdDirectoryDirectory under /etc/sysconfig/deephaven that is shared across pods. See below for more details.
etcd-overrides.yamlFileThe saved copy of the file last passed with --etcd-config-overrides. The copy it replaced is kept as etcd-overrides.yaml.previous.
installationDirectoryContains cluster.cnf and the installation scripts generated from the installer JAR (/dh/container-setup-files/Installer.jar on the infra pod) during initialization.
priv-keysDirectoryThe DSA/RSA private keys used by Deephaven services. The infra pod writes them here; every pod copies them into its own /etc/sysconfig/deephaven/priv-keys during initialization. Their public halves are registered in the etcd auth database, so do not delete this directory unless you also discard dh_etcd_config.tgz and the etcd database.
upgrade-backup-<timestamp>DirectoryCreated by --upgrade; holds the regenerable configuration that the upgrade moved aside.

/etc/sysconfig/deephaven directory contents

The following table provides an overview of the subdirectories of /etc/sysconfig/deephaven and whether they are shared across containers in a Podman deployment of Deephaven:

ItemTypeShared?Notes
authDirectoryNOContains signing certificates and private keys, including priv-tdcp.base64.txt (which is required to run the TDCP). The private keys are copied from the shared configuration's priv-keys directory, described in Shared configuration directory contents.
auth-userDirectoryNOContains webServices-keystore.p12, the web server certificate generated from the pod's tls.crt and tls.key.
backupsDirectoryNOContains backup files created by the Deephaven installer.
dh-configDirectoryYESContains the configuration files required for Deephaven services to communicate with the Deephaven configuration server.
etcdDirectoryYESContains the etcd client configuration, which is required for the Deephaven configuration server and command line tools to connect to etcd.
illumon.confs.<version>DirectoryNOContains the Deephaven hostconfig files, which set environment variables used by Deephaven services and utility scripts.
illumon.confs.latestSymlinkNOLink to latest illumon.confs.<version>.
illumon.d.<version>DirectoryNOContains java_lib/hotfixes/integrations/etc.
illumon.d.latestSymlinkNOLink to latest illumon.d.
illumon.iris.hostconfigFileNONode-specific hostconfig.
monitDirectoryNOContains Monit configuration files for Deephaven services.
pluginsDirectoryNOContains Deephaven plugins.
pythonDirectoryNO
schemaDirectoryNODeprecated. (Schema files are stored in etcd.)
priv-keysDirectoryCOPIEDContains the DSA/RSA private keys. Copied from /deephaven-shared-config/priv-keys during initialization.
trustDirectoryNOContains the truststore (truststore-iris.p12) that each pod generates from the CA passed with --ca-cert-file.

For reference, here is an example ls from illumon.d.latest on a non-containerized Deephaven installation:

Note that many of the directories in illumon.d.latest are links back to directories in /etc/sysconfig/deephaven. Of these directories, those that must be shared in a podman deployment (dh-config and etcd) are stored in a volume mounted at /deephaven-shared-config, and links are created from /etc/sysconfig/deephaven/<dir> to the corresponding directory in /deephaven-shared-config.