Restart and upgrade a Podman deployment

This guide covers restarting and upgrading a Deephaven Podman deployment. It also covers exporting, importing, and backing up data and configuration.

Restart and upgrade

To stop and resume a working deployment, use podman pod stop -t 90 <pod name> and podman pod start <pod name>. Restarting, in this guide, means using start_command.sh to recreate the pods from the persisted configuration in the VOLUME_BASE_DIR directory structure, for example to upgrade or to change start_command.sh options.

Besides persisting configuration for the cluster, the VOLUME_BASE_DIR directory structure also stores shared running configuration for use in nodes across the cluster. The --upgrade flag moves the regenerable part of it into a timestamped backup directory before the cluster reinitializes.

To upgrade, download or build new images for the new version of Deephaven (a point release in the same major version or a new major version). Check the release notes of the new version for any manual steps or caveats. Use the start_command.sh from the new version's Deephaven Podman package, as there may be fixes or new functionality in the script itself.

Before you upgrade

Deephaven preserves all configuration stored in etcd (ACLs, Persistent Queries, property files, data routing YAML, and schema) and historical data.

Warning

Other data is preserved only if it is on a mounted volume, and only if you pass the same volume option, with the same directory, every time you run start_command.sh. Without the option, the data is stored inside the pod and is deleted when the pod is recreated:

  • Intraday data, which includes Web UI dashboards and notebooks: -I.
  • Centrally managed user tables: -W. Direct user tables: -U.
  • Logs: -L. Binary logs not yet loaded as intraday data: a volume mounted at /var-log-deephaven-binlogs with --additional-volume.
  • Custom JARs and calendars: --java-lib-volume and --calendars-volume.
  • Data written by an in-worker Data Import Server: --additional-volume.

Adding an option to an existing deployment does not move the data that is already inside the pod: the new pod starts with the new, empty directory. Move the data onto host volumes first.

For intraday data and user tables, start_command.sh checks before it deletes anything. It refuses to recreate a pod (--delete-pod or --upgrade) when that data is inside the pod and its volume option is missing or names an empty directory, or when the data is on a host volume the new command would not mount. A path that holds no data is not checked, so you need no option for data you do not have. It names the option to add and changes nothing. Pass --discard-pod-data only if you mean to delete that data.

Caution

Take a complete backup before upgrading. Optionally, export any pod-internal data, such as user tables, custom calendars, or custom JARs, before stopping the pod. You will lose any such data not on a mounted volume when the pod is replaced. If the upgrade causes a critical issue, restoring from a pre-upgrade backup is your only supported recovery path. Deephaven does not support downgrading by switching to an older image without restoring from backup; doing so may result in data corruption or an unrecoverable state. See Reverting an upgrade for details.

Upgrade with --upgrade

The --upgrade flag of start_command.sh automates the upgrade steps. Run start_command.sh --upgrade on each node with the same arguments used to deploy the cluster, including the pod hostname (-h), -c if you use it, every volume option, and, on the infra node, the complete -q list of query pods:

For each pod, --upgrade:

  • On the infra node, checks that the shared configuration archive (dh_etcd_config.tgz) and any etcd overrides are readable before stopping the existing pod. The pod may already be stopped. If its container was already removed, the upgrade uses the preserved shared configuration and reports the deployed version as unknown.
  • Uses the supplied machine names as the authoritative cluster topology. The iris-endpoints.prop file is regenerated from the supplied values during initialization, so the full machine list is required on every invocation passing -t infra. If the supplied list differs from the existing deployment, a notice showing both lists is printed.
  • Stops the existing pod with a 90-second timeout, so that all Deephaven processes as well as monit and etcd exit cleanly, and then removes it.
  • On the infra node, moves the regenerable shared configuration (auth, trust, dh-config, etcd, installation, and cluster_status, where present) into a timestamped upgrade-backup-<timestamp> directory under the shared configuration volume. Three items stay in place and are also copied into the backup:
    • dh_etcd_config.tgz, so the new deployment re-attaches to the existing etcd database.
    • priv-keys, because the public halves of its keypairs are registered in that database. Regenerated keys would not match them.
    • etcd-overrides.yaml, so your etcd overrides still apply.
  • If SELinux is in use, clears the Multi-Category Security (MCS) labels from the shared configuration and etcd volumes. Files the old pod created carry its MCS categories, and the new pod gets different ones, so without this the new pod gets "Permission Denied" errors on those files. This clears all MCS categories, including any you applied yourself. If the relabeling fails (for example, because it needs sudo), start_command.sh prints a warning; run the commands in Fixing MCS label conflicts yourself.
  • Creates and starts a new pod from the new image.

start_command.sh refuses to start a pod from an image whose Deephaven version differs from the deployed version unless --upgrade is passed. The version is read from /usr/illumon/latest/etc/IRIS_GRADLE_VERSION inside the image.

Upgrading from Grizzly

Upgrading a Grizzly (1.20240517.*) deployment to Grizzly+ requires two migrations, which start_command.sh --upgrade performs automatically; neither needs another flag. Both work from the preserved files and properties, so they also run when the previous container is stopped or removed. The rest of your command line still applies, including -c if your certificates come from an internal PKI (see the list below).

  • Private keys move. Grizzly shared the whole auth directory and kept its DSA/RSA private keys there. Grizzly+ shares only priv-keys. Because auth is regenerable and is moved into the backup directory, the upgrade first copies auth/priv*.txt into priv-keys so the keypairs registered in the etcd auth database keep working. Verify with md5sum that priv-iris.base64.txt is unchanged after the upgrade.
  • Obsolete TLS assignments are migrated. Grizzly's Podman block in iris-environment.prop points TLS settings at /deephaven-tls, which Grizzly+ does not create. The upgrade comments out only the assignments that exactly match those stock paths, so the current service-specific defaults apply, and updates the stock SAML keystore and passphrase paths in the authentication_server section. Your own values, other properties, comments, and scoped settings are kept. Before importing the change into etcd, the upgrade saves the original as iris-environment.prop.before-migration.<suffix> in the shared configuration volume. If the export or import fails, the upgrade stops; fix the cause and run it again.

Before you upgrade from Grizzly, also:

  • Pass -c if you use a private CA. Grizzly built a TLS truststore automatically from the CA in the TLS volume. Grizzly+ builds one only when you pass -c / --ca-cert-file, which is required whenever you generated certificates with create_dh_cert.sh. See Supplying trusted certificates to Podman.
  • Move SAML and Active Directory certificates out of auth. Grizzly kept the auth directory, including any certificates you added to it, in the shared configuration volume. Upgrading from Grizzly moves it into upgrade-backup-<timestamp>/auth. In Grizzly+, auth is inside the container and is not preserved when a pod is recreated, so keep such certificates in a host directory mounted with --additional-volume. See Persisting authentication certificates.

Manually resetting shared state

These steps are the manual equivalent of the preparation performed by --upgrade, which you should normally use instead; only --upgrade migrates Grizzly private keys. They apply only when dh_etcd_config.tgz exists in the shared configuration volume. If an installation that has never completed failed before dh_etcd_config.tgz was created, follow Retry a failed installation instead.

  • Stop the existing infra pod (e.g., podman pod stop dh-infra-pod -t 90). Specify a timeout of at least 90 seconds to ensure that all Deephaven processes as well as monit and etcd are able to exit cleanly.
  • If SELinux is in use, clear the MCS labels from the shared configuration and etcd volumes, as described in Fixing MCS label conflicts.
  • Remove only the regenerable paths listed below. Preserve dh_etcd_config.tgz, priv-keys, etcd-overrides.yaml, and all backups:

Note

With rootless Podman, files written by the container's accounts belong to subordinate UIDs, so remove them inside Podman's user namespace instead of with sudo. Either way, the removal must succeed completely; leftover files conflict with the reinitialized deployment.

After resetting the shared state, re-run start_command.sh with the --delete-pod flag (and the full machine list) to recreate the pods. During initialization, the infra container detects the preserved dh_etcd_config.tgz archive and re-extracts the etcd configuration files, reusing the existing etcd database.

Export, import, and back up data and configuration

Export

General instructions for non-MariaDB/MySQL ACL systems backup apply directly to Deephaven Podman deployments as well.

The system may not persist some sections of the pod filesystem outside of the pod:

  • /db/Users stores direct user table data. You can configure it to use a mounted volume, but if you have not, use podman cp to copy its data from a running or stopped pod. Copy the entire /db/Users directory tree so you can import it to bring direct user table data to a different deployment.
  • /db/IntradayUser stores centrally managed user table data. You can configure it to use a mounted volume, but if you have not, use podman cp to copy its data from a running or stopped pod. Copy the entire /db/IntradayUser directory tree so you can import it to bring centrally managed user table data to a different deployment.
  • /etc/sysconfig/illumon.d/java_lib stores custom JARs and additional JARs not delivered as part of the Deephaven deployment. You can configure it to use a mounted volume, but if you have not, use podman cp to copy its data from a running or stopped pod.
  • /etc/sysconfig/illumon.d/calendars stores custom calendars not delivered as part of the Deephaven deployment. You can configure it to use a mounted volume, but if you have not, use podman cp to copy its data from a running or stopped pod.

An example of a podman cp command to export direct user table data from a pod:

You can also export or back up any other paths in the pod that you have not mapped to additional volumes, or that cannot be mapped (for instance, /etc/sysconfig/illumon.d/resources), using podman cp commands.

Move pod data onto host volumes

To start keeping data that is stored inside the infra pod, copy it into host directories, then recreate the pod with the matching volume options. Use podman unshare cp -a, as below: it keeps each file's owner and permissions. In a multi-user deployment, Deephaven's accounts (such as dbmerge for intraday data) must own their files to keep writing them, and podman cp would make every copied file belong to the Podman user instead.

Run these commands as the user that runs the pods (rootless Podman), from the extracted deephaven-podman-<version> directory. Each copy prints nothing when it succeeds, and podman umount prints the container name, dh-infra. If a copy prints an error, do not recreate the pod: the data is still in the stopped pod.

  1. Stop the pod and create the host directories:

  2. Copy /db/Intraday. Skip this step if the pod already uses -I (--intraday-data-volume). /db/Intraday is a Podman volume of the pod:

  3. Copy /db/Users. Skip this step if the pod already uses -U (--users-volume). /db/Users is in the container's own file system, which podman mount makes readable:

  4. Copy /db/IntradayUser. Skip this step if the pod already uses -W (--intradayuser-volume):

  5. Recreate the infra pod with --delete-pod (or --upgrade), your usual options, and the new volume options:

    Pass these options every time you run start_command.sh for this pod from now on.

Import

You can import configuration, data, and content from exports taken from other Deephaven systems. In general, you can import exports from the same major version. If you took the export from a different major version, please check with Deephaven support for specifics. You can import Vermilion exports to Vermilion+ and Grizzly, Vermilion+ exports to Grizzly and Grizzly+, and Grizzly exports to Grizzly+.

Important

Podman deployment overrides live in iris-endpoints.prop, which is regenerated and re-imported every time the infra pod is recreated. Do not copy them into iris-environment.prop: that file holds your custom settings, which are carried forward unchanged through every upgrade, so a copied override stays even after a later release needs a different value.

An iris-environment.prop taken from a Grizzly or earlier Podman deployment carries a ### Overrides for podman deployment: block. Before importing the file, comment out the stock TLS assignments in that block that point at /deephaven-tls, which does not exist in Grizzly+, and keep any settings you added. If SAML is configured, also change SAMLAuth.tls.keystore to /etc/sysconfig/deephaven/auth-user/webServices-keystore.p12 and SAMLAuth.tls.passphrase.file to /etc/sysconfig/deephaven/auth-user/.webapi_passphrase wherever they point at /deephaven-tls. The automatic migration runs only when the infra pod is recreated, not when you import a file yourself. A file taken from a system that was not deployed with start_command.sh carries no such block and needs no edit.

Back up

You can back up configuration and most data of Deephaven Podman deployments with the VOLUME_BASE_DIR directory tree. If network storage hosts some subdirectories of this tree, you may want to omit them from the backup, as the network storage system may already protect them. See File system backup and restoration (Podman) for the procedure.