Restart and Upgrade a Podman deployment

This guide covers restarting and upgrading a Deephaven Podman deployment. It also covers exporting, importing, and backing up data and configuration.

Restart and upgrade

Restarting, in this context, means using the start_command.sh to initialize a cluster using persisted configuration data from the VOLUME_BASE_DIR directory structure.

Besides persisting configuration for the cluster, the VOLUME_BASE_DIR directory structure also stores shared running configuration for use in nodes across the cluster. You must remove this shared data before using start_command.sh to reinitialize or upgrade the cluster.

By default, Deephaven preserves all configuration stored in etcd (ACLs, Persistent Queries, property files, data routing YAML, and schema), as well as intraday and historical data, which also includes Web UI dashboards and notebooks. Deephaven does not preserve user tables, custom calendars, or contents of the /etc/sysconfig/illumon.d/java_lib path by default, but you can configure them to persist.

Caution

Take a complete backup before upgrading. Optionally, export any pod-internal data — such as user tables, custom calendars, or custom JARs — before stopping the pod. You will lose any such data not mapped to a Persistent Volume when the pod is stopped or replaced. If the upgrade causes a critical issue, restoring from a pre-upgrade backup is your only supported recovery path. Deephaven does not support downgrading by switching to an older image without restoring from backup; doing so may result in data corruption or an unrecoverable state. See Reverting an upgrade for details.

To upgrade, download, or build new images for the new version of Deephaven (this could be a point release update in the same major version or a new major version). Refer to the release notes of a new version to see if there are any manual steps or caveats associated with the upgrade process. You should also use the start_command.sh that comes with the new version Deephaven Podman package, as there may be fixes or new functionality in the start_command.sh script itself.

The process for rerunning start_command.sh to reinitialize or upgrade a Deephaven Podman deployment is:

  • Stop the existing infra pod (e.g., podman pod stop dh-infra-pod -t 90). Specify a timeout of at least 90 seconds to ensure that all Deephaven processes as well as monit and etcd are able to exit cleanly.

  • If SELinux is in use, clear the MCS labels from the shared configuration and etcd volumes. When Podman mounts volumes with the :z option, SELinux assigns Multi-Category Security (MCS) labels to restrict access. A new pod gets a different MCS category than the old pod, causing "Permission Denied" errors on files the previous pod created.

    Caution

    The -l s0 option clears ALL SELinux labels, including any user-applied labels. See Troubleshooting SELinux in Podman for details on MCS labels and alternative approaches.

  • Remove all files from the shared config volume except for dh_etcd_config.tgz:

Caution

This command deletes the auth directory, which contains SAML and Active Directory certificates if you configured them. If you have not stored these certificates in a persistent location (such as the java_lib volume), you will lose them. See Persisting authentication certificates for how to configure certificate persistence before restarting.

Note

If SELinux is not in use, it should be sufficient to run the following without sudo:

The system may deny access when deleting some data, but this should remove data that would conflict with restarting the deployment.

Export, import, and back up data and configuration

Export

General instructions for non-MariaDB/MySQL ACL systems backup apply directly to Deephaven Podman deployments as well.

The system may not persist some sections of the pod filesystem outside of the pod:

  • /db/Users stores direct user table data. You can configure it to use a mounted volume, but if you have not, use podman cp to copy its data from a running or stopped pod. Copy the entire /db/Users directory tree so you can import it to bring direct user table data to a different deployment.
  • /db/IntradayUser stores centrally managed user table data. You can configure it to use a mounted volume, but if you have not, use podman cp to copy its data from a running or stopped pod. Copy the entire /db/IntradayUser directory tree so you can import it to bring centrally managed user table data to a different deployment.
  • /etc/sysconfig/illumon.d/java_lib stores custom JARs and additional JARs not delivered as part of the Deephaven deployment. You can configure it to use a mounted volume, but if you have not, use podman cp to copy its data from a running or stopped pod.
  • /etc/sysconfig/illumon.d/calendars stores custom calendars not delivered as part of the Deephaven deployment. You can configure it to use a mounted volume, but if you have not, use podman cp to copy its data from a running or stopped pod.

An example of a podman cp command to export direct user table data from a pod:

You can also export or back up any other paths in the pod that you have not mapped to additional volumes, or that cannot be mapped (for instance, /etc/sysconfig/illumon.d/resources), using podman cp commands.

Import

You can import configuration, data, and content from exports taken from other Deephaven systems. In general, you can import exports from the same major version. If you took the export from a different major version, please check with Deephaven support for specifics. You can import Vermilion exports to Vermilion+ and Grizzly, Vermilion+ exports to Grizzly and Grizzly+, and Grizzly exports to Grizzly+.

Important

Deephaven Podman deployments require special properties in the initial iris-environment.prop files. If you import iris-environment.prop from a system you did not deploy using the Deephaven Podman start_command.sh, you must add this set of properties to the imported iris-environment.prop:

Back up

You can back up configuration and at least most data of Deephaven Podman deployments with the VOLUME_BASE_DIR directory tree. However, if network storage hosts some subdirectories of this tree, you may want to omit these from backup, as the network storage system may already back them up and/or otherwise protect them.