Podman

This guide provides step-by-step instructions for quickly deploying a Deephaven Enterprise cluster using Podman.

Deephaven's Podman-based distribution enables rapid deployment of a Deephaven Enterprise cluster. The single-user image used in this guide is ideal for testing purposes. Podman is an open-source container management system that is highly compatible with Docker containers and syntax.

About the Podman Deployment

Deephaven's Podman deployment includes:

• Tools for building custom images.
• A script (start_command.sh) to launch a Podman containerized cluster.
• Pre-built images that can be started immediately using the start_command.sh script.

The Deephaven Podman distribution is supported on Linux systems. At this time, MacOS and Windows are not supported. For a complete list of supported operating system versions and dependencies, refer to the Supported Versions page. Provided you have the prerequisites, this guide provides simplified instructions for running a Deephaven Enterprise deployment on Podman in just a few minutes.

Whether you build your own or use the pre-built ones, there are two images:

• an image that can be used for the infrastructure node or all-in-one node of a Deephaven cluster.
• an image that can be used to add one or more dedicated query server nodes to the cluster.

The Deephaven Podman deployment stores configuration data and, optionally, other data (such as table data and logs) in volume paths on the Podman host machines. When the start_command.sh script is executed, it initiates the installation or upgrade of Deephaven and starts the pods. The initial startup process may take a few minutes, as it runs the full Deephaven installation or upgrade process. Because configuration and other data are preserved on the host's filesystem, the pods themselves remain stateless and are easy to manage.

Prerequisites

General prerequisites

For Deephaven Podman deployments in general, you need:

• One or more machines or VMs with Podman installed.
  • Each machine should have at least 24GB of RAM dedicated to running Deephaven on Podman. This is the minimum to reliably run a Deephaven instance and execute some small queries for a single user.
  • If you are going to use the deployment for larger-scale testing or production, each machine should have 64GB or more of RAM that can be dedicated to running Deephaven on Podman.
  • The machine(s) should also be resolvable by DNS.
• The Deephaven Podman distribution package.
• A Web server certificate to use for the installation.
• A pre-built Deephaven Podman image package. If you want to build your own images, refer to the building Podman images section, and then skip the loading pre-built images step when executing the steps below.
• If you are running on Linux machines with SELinux enabled, you may need sudo rights on the machines to set the necessary SELinux flags.

Quickstart Deephaven Podman deployment

This section presents steps for deploying a single-node Deephaven Podman installation that uses the Envoy reverse proxy so that only one port (the default is 8000) needs to be accessible on the host to connect to the Deephaven Web UI and client APIs.

Load the pre-built images

Load the pre-built images into your local Podman image repository using:

IMAGE_ARCHIVE=<deephaven-podman-image.tar.gz>
podman load < "${IMAGE_ARCHIVE:?}"

Set up the host

1. Download the Deephaven Podman distribution package and extract it to a working path you have created:

   # from the directory where you downloaded the tar.gz
   mkdir ./dh-podman
   cd ./dh-podman
   tar -xvf ../deephaven-podman-1.20240517.298.tar.gz
   
   # this example creates a deephaven-podman-1.20240517.298 directory under <the download directory>/dh-podman/
   

2. Specify the path to the directory to store configuration information for the Podman instance:

   export VOLUME_BASE_DIR=/<root path of Deephaven Podman configuration>
   

   Note

   You must have rights to create a directory structure starting with VOLUME_BASE_DIRECTORY. If something like /container_test is used, then you must have rights to create a directory at the root of the filesystem. If you don't have sufficient rights, you might need an administrator to create the root directory for you and give you ownership before you run the next step.

3. Create needed subdirectories of VOLUME_BASE_DIR:

   mkdir -pv "${VOLUME_BASE_DIR:?}"/{db-intraday,db-systems,deephaven-tls,deephaven-shared-config,deephaven-etcd}
   

4. Obtain certificates Web server certificate. Use the following for a quickly-bootstrapped certificate for testing or demonstration purposes:

   ./create_dh_cert.sh -a "${VOLUME_BASE_DIR:?}"/deephaven-tls -d "${VOLUME_BASE_DIR:?}"/deephaven-tls -h "my-infra-hostname.mydomain.net,my-query-hostname.mydomain.net"
   

   Warning

   Do not use create_dh_cert.sh to generate certificates for production.

   In non-production scenarios, the certificates generated will be self-signed certificates, which will not be trusted by your local system. As a result, you will receive certificate warnings in the web browser when you connect to the Deephaven web UI.

5. When supplying your own certificates, copy certificate files to "${VOLUME_BASE_DIR:?}/deephaven-tls" as:

   1. Certificate file -> tls.crt
   2. Private key file -> tls.key
   3. Issuing certificate file -> ca.crt

6. For Linux hosts where SELinux is in use:

   chcon -vR -u system_u -r object_r -t container_file_t "${VOLUME_BASE_DIR:?}"
   

   This configures SELinux to allow the pod to access files in these paths.

   Note

   Use the sestatus command to check whether SELinux is enabled on a Linux machine. This command indicates whether SELinux is enabled or not. If sestatus is not recognized, this indicates that SELinux is not installed on the system.

Start the pod

From the directory where the start_commmand.sh has been unpacked, start the pod with:

POD_HOSTNAME= <URL name of the DH server>
./start_command.sh \
-e 8000 \
-h "${POD_HOSTNAME:?}" \
-t infra -T "${VOLUME_BASE_DIR:?}/deephaven-tls/" \
-d \
-n dh-infra \
-s \
-I "${VOLUME_BASE_DIR:?}/db-intraday/" \
-H "${VOLUME_BASE_DIR:?}/db-systems/"

Where the arguments are:

• -e 8000 - connect and forward host port 8000 to the pod's Envoy service.
• -h <URL name of the DH server> - the name by which you will connect to the server, which is the same as the name in the Web certificate, such as mydeephaven.myorg.com.
• -t infra - the type of image to run, infra in this case.
• -T "$VOLUME_BASE_DIR/deephaven-tls/" - path to the Web server certificate files.
• -d - delete any existing pod(s) running for this container name.
• -n dh-infra - the container name, and the base name to use for the main pod (dh-infra-pod in this case).
• -s - single node Envoy instance, so, only forward -e port into the container.
• -I $VOLUME_BASE_DIR/db-intraday/ - mount the db-intraday host directory to /db/Intraday in the container for persistent storage of intraday data.
• -H $VOLUME_BASE_DIR/db-systems/ - mount the db-systems host directory to /db/Systems in the container for persistent storage of historical data.

export POD_IP=<IPv4 address of the Deephaven cluster>

Monitor the startup process

The following command can be used immediately after executing start_commmand.sh to view the details of the startup process -- in this example dh-infra for the container name. If you used a different value for -n above, use that value for this command as well.

podman logs -f dh-infra

This should show a stream of log messages as the pod runs its startup process. The startup scripts complete the installation of Deephaven and do some initial configuration. The process should take about 5 to 10 minutes before the Deephaven installation is ready to be used. If something goes wrong, this log file will also be the best place to check for details of the failure. If the initialization process fails, the scripts will leave the pod running for 10 minutes before shutting it down.

Access a shell in the pod

Once the pod is running, use the following command to connect to a shell session in the dh-infra container. If you used a different container name when running the start_command.sh, then use that name here instead of dh-infra:

podman exec -it dh-infra bash

Connecting to a shell session is needed for much of Deephaven's configuration. It is also useful for troubleshooting.

Set the admin password and installing sample data

When a Podman deployment is first initialized, there will be one built-in Deephaven user account — iris — and its password will not be set. Before you can log in to Deephaven, you need to set a password for the iris account. This is done using a command line tool called iris_db_user_mod:

/usr/illumon/latest/bin/dhconfig acl user set-password --name iris

You will be prompted to enter a new password for the iris account.

While in the command shell, you can also install sample data in the LearnDeephaven namespace. This namespace and its tables are often used in Deephaven documentation examples.

/usr/illumon/latest/install/learn_install.sh

To exit the shell session, execute exit.

Log in to the Web UI

You can now log in to the Deephaven Web UI.

https://<URL name of the DH server>:<port number for envoy>/iriside

The typical port used for the cluster is 8000, so the URL will be something like https://mydeephaven.myorg.com:8000/iriside.

Start a worker and run a query

After logging in, you should see a dialog to create a Code Studio. If you do not, click the + New tab and select New Code Studio from the Advanced drop-down.

Options are available for Legacy or Core+ workers, as well as for Groovy or Python languages for query scripts. If you have installed the LearnDeephaven sample data, launch a Core+ worker for either Groovy or Python so you can try this test query.

Once the worker has been initialized, enter this line to run a simple query and view stock quote data from the sample data set:

t = db.historicalTable("LearnDeephaven","StockQuotes")
        .where("Date=`2017-08-25`")

t = db.historical_table("LearnDeephaven", "StockQuotes").where("Date=`2017-08-25`")

Stop, restart, or remove the pod

Use podman stop to shut down the dh-infra containers, giving them 90 seconds to shut down before aggressively killing the processes. 90 seconds is generally sufficient time for a clean shutdown. If you used a different container name when running the start_command.sh, use that name here instead of dh-infra:

podman stop dh-infra --time 90

Use podman start to re-start a deployment that has been stopped with podman stop:

podman start dh-infra

Use podman rm to delete a container that has already been stopped. This frees up space the pod and its internal storage were using. This does not remove the VOLUME_BASE_DIR directory structure or imported Deephaven Podman image files:

podman rm dh-infra

If the pod is in a bad state, as might happen if the host was rebooted without stopping the pod, it can be reinitialized with start_command.sh. This will preserve configuration and data that is in system intraday and historical tables.

Related documentation

• Attaching additional storage volumes to persist storage of user tables and log files
• Building your own images
• Podman troubleshooting