Restart and Upgrade a Podman deployment

This guide covers restarting and upgrading a Deephaven Podman deployment. It also covers exporting, importing, and backing up data and configuration.

Restart and upgrade

Restarting, in this context, means using the start_command.sh to initialize a cluster using persisted configuration data from the VOLUME_BASE_DIR directory structure.

Besides persisting configuration for the cluster, the VOLUME_BASE_DIR directory structure also stores shared running configuration for use in nodes across the cluster. This shared data must be moved aside before start_command.sh reinitializes or upgrades the cluster; the --upgrade flag handles this automatically by moving the data into a timestamped backup directory.

By default, Deephaven preserves all configuration stored in etcd (ACLs, Persistent Queries, property files, data routing YAML, and schema), as well as intraday and historical data, which also includes Web UI dashboards and notebooks. Deephaven does not preserve user tables, custom calendars, or contents of the /etc/sysconfig/illumon.d/java_lib path by default, but you can configure them to persist.

To upgrade, download, or build new images for the new version of Deephaven (this could be a point release update in the same major version or a new major version). Refer to the release notes of a new version to see if there are any manual steps or caveats associated with the upgrade process. You should also use the start_command.sh that comes with the new version Deephaven Podman package, as there may be fixes or new functionality in the start_command.sh script itself.

Upgrade with --upgrade

The --upgrade flag of start_command.sh automates the upgrade steps. Run start_command.sh --upgrade on each node, with the same arguments used to deploy the cluster — including the pod hostname (-h) and, on the infra node, the complete -q list of query pods:

For each pod, --upgrade:

• Uses the supplied machine names as the authoritative cluster topology. The iris-endpoints.prop file is regenerated from the supplied values during initialization, so the full machine list is required on every infra invocation. If the supplied list differs from the existing deployment, a notice showing both lists is printed.
• Stops the existing pod with a 90-second timeout — so that all Deephaven processes as well as monit and etcd exit cleanly — and then removes it.
• On the infra node, moves the regenerable shared configuration (auth, trust, dh-config, etcd, installation, and cluster_status) into a timestamped upgrade-backup-<timestamp> directory under the shared configuration volume. The dh_etcd_config.tgz file is preserved in place so the new deployment re-attaches to the existing etcd database, and the priv-keys directory is preserved because the public halves of the keypairs it holds are registered in that database — regenerating them would leave each Deephaven process holding a private key the auth database no longer recognizes.
• If SELinux is in use, clears the MCS labels from the shared configuration and etcd volumes. When Podman mounts volumes with the :z option, SELinux assigns Multi-Category Security (MCS) labels to restrict access. A new pod gets a different MCS category than the old pod, causing "Permission Denied" errors on files the previous pod created.
• Creates and starts a new pod from the new image.

start_command.sh refuses to start a pod from an image whose Deephaven version differs from the deployed version unless --upgrade is passed. The version is read from /usr/illumon/latest/etc/IRIS_GRADLE_VERSION inside the image.

Manually resetting shared state

If an upgrade or installation fails partway (for example, the cluster status is stuck at INITIALIZING), reset the shared state manually and retry. These steps are also the manual equivalent of the preparation performed by --upgrade:

• Stop the existing infra pod (e.g., podman pod stop dh-infra-pod -t 90). Specify a timeout of at least 90 seconds to ensure that all Deephaven processes as well as monit and etcd are able to exit cleanly.

• If SELinux is in use, clear the MCS labels from the shared configuration and etcd volumes:

• Remove all files from the shared config volume except for dh_etcd_config.tgz:

After resetting the shared state, re-run start_command.sh with the --delete-pod flag (and the full machine list) to recreate the pods. During initialization, the infra container detects the preserved dh_etcd_config.tgz archive and re-extracts the etcd configuration files, reusing the existing etcd database.

Export, import, and back up data and configuration

Export

General instructions for non-MariaDB/MySQL ACL systems backup apply directly to Deephaven Podman deployments as well.

The system may not persist some sections of the pod filesystem outside of the pod:

• /db/Users stores direct user table data. You can configure it to use a mounted volume, but if you have not, use podman cp to copy its data from a running or stopped pod. Copy the entire /db/Users directory tree so you can import it to bring direct user table data to a different deployment.
• /db/IntradayUser stores centrally managed user table data. You can configure it to use a mounted volume, but if you have not, use podman cp to copy its data from a running or stopped pod. Copy the entire /db/IntradayUser directory tree so you can import it to bring centrally managed user table data to a different deployment.
• /etc/sysconfig/illumon.d/java_lib stores custom JARs and additional JARs not delivered as part of the Deephaven deployment. You can configure it to use a mounted volume, but if you have not, use podman cp to copy its data from a running or stopped pod.
• /etc/sysconfig/illumon.d/calendars stores custom calendars not delivered as part of the Deephaven deployment. You can configure it to use a mounted volume, but if you have not, use podman cp to copy its data from a running or stopped pod.

An example of a podman cp command to export direct user table data from a pod:

You can also export or back up any other paths in the pod that you have not mapped to additional volumes, or that cannot be mapped (for instance, /etc/sysconfig/illumon.d/resources), using podman cp commands.

Import

You can import configuration, data, and content from exports taken from other Deephaven systems. In general, you can import exports from the same major version. If you took the export from a different major version, please check with Deephaven support for specifics. You can import Vermilion exports to Vermilion+ and Grizzly, Vermilion+ exports to Grizzly and Grizzly+, and Grizzly exports to Grizzly+.

Back up

You can back up configuration and at least most data of Deephaven Podman deployments with the VOLUME_BASE_DIR directory tree. However, if network storage hosts some subdirectories of this tree, you may want to omit these from backup, as the network storage system may already back them up and/or otherwise protect them.

Related documentation

• Podman Quickstart
• Podman Installation
• Troubleshooting a Podman deployment
• Kubernetes