Restart and upgrade a Podman deployment
This guide covers restarting and upgrading a Deephaven Podman deployment. It also covers exporting, importing, and backing up data and configuration.
Restart and upgrade
To stop and resume a working deployment, use podman pod stop -t 90 <pod name> and podman pod start <pod name>. Restarting, in this guide, means using start_command.sh to recreate the pods from the persisted configuration in the VOLUME_BASE_DIR directory structure, for example to upgrade or to change start_command.sh options.
Besides persisting configuration for the cluster, the VOLUME_BASE_DIR directory structure also stores shared running configuration for use in nodes across the cluster. The --upgrade flag moves the regenerable part of it into a timestamped backup directory before the cluster reinitializes.
To upgrade, download or build new images for the new version of Deephaven (a point release in the same major version or a new major version). Check the release notes of the new version for any manual steps or caveats. Use the start_command.sh from the new version's Deephaven Podman package, as there may be fixes or new functionality in the script itself.
Before you upgrade
Deephaven preserves all configuration stored in etcd (ACLs, Persistent Queries, property files, data routing YAML, and schema) and historical data.
Warning
Other data is preserved only if it is on a mounted volume, and only if you pass the same volume option, with the same directory, every time you run start_command.sh. Without the option, the data is stored inside the pod and is deleted when the pod is recreated:
- Intraday data, which includes Web UI dashboards and notebooks:
-I. - Centrally managed user tables:
-W. Direct user tables:-U. - Logs:
-L. Binary logs not yet loaded as intraday data: a volume mounted at/var-log-deephaven-binlogswith--additional-volume. - Custom JARs and calendars:
--java-lib-volumeand--calendars-volume. - Data written by an in-worker Data Import Server:
--additional-volume.
Adding an option to an existing deployment does not move the data that is already inside the pod: the new pod starts with the new, empty directory. Move the data onto host volumes first.
For intraday data and user tables, start_command.sh checks before it deletes anything. It refuses to recreate a pod (--delete-pod or --upgrade) when that data is inside the pod and its volume option is missing or names an empty directory, or when the data is on a host volume the new command would not mount. A path that holds no data is not checked, so you need no option for data you do not have. It names the option to add and changes nothing. Pass --discard-pod-data only if you mean to delete that data.
Caution
Take a complete backup before upgrading. Optionally, export any pod-internal data, such as user tables, custom calendars, or custom JARs, before stopping the pod. You will lose any such data not on a mounted volume when the pod is replaced. If the upgrade causes a critical issue, restoring from a pre-upgrade backup is your only supported recovery path. Deephaven does not support downgrading by switching to an older image without restoring from backup; doing so may result in data corruption or an unrecoverable state. See Reverting an upgrade for details.
Upgrade with --upgrade
The --upgrade flag of start_command.sh automates the upgrade steps. Run start_command.sh --upgrade on each node with the same arguments used to deploy the cluster, including the pod hostname (-h), -c if you use it, every volume option, and, on the infra node, the complete -q list of query pods:
For each pod, --upgrade:
- On the infra node, checks that the shared configuration archive (
dh_etcd_config.tgz) and any etcd overrides are readable before stopping the existing pod. The pod may already be stopped. If its container was already removed, the upgrade uses the preserved shared configuration and reports the deployed version as unknown. - Uses the supplied machine names as the authoritative cluster topology. The
iris-endpoints.propfile is regenerated from the supplied values during initialization, so the full machine list is required on every invocation passing-t infra. If the supplied list differs from the existing deployment, a notice showing both lists is printed. - Stops the existing pod with a 90-second timeout, so that all Deephaven processes as well as
monitandetcdexit cleanly, and then removes it. - On the infra node, moves the regenerable shared configuration (
auth,trust,dh-config,etcd,installation, andcluster_status, where present) into a timestampedupgrade-backup-<timestamp>directory under the shared configuration volume. Three items stay in place and are also copied into the backup:dh_etcd_config.tgz, so the new deployment re-attaches to the existing etcd database.priv-keys, because the public halves of its keypairs are registered in that database. Regenerated keys would not match them.etcd-overrides.yaml, so your etcd overrides still apply.
- If SELinux is in use, clears the Multi-Category Security (MCS) labels from the shared configuration and etcd volumes. Files the old pod created carry its MCS categories, and the new pod gets different ones, so without this the new pod gets "Permission Denied" errors on those files. This clears all MCS categories, including any you applied yourself. If the relabeling fails (for example, because it needs
sudo),start_command.shprints a warning; run the commands in Fixing MCS label conflicts yourself. - Creates and starts a new pod from the new image.
start_command.sh refuses to start a pod from an image whose Deephaven version differs from the deployed version unless --upgrade is passed. The version is read from /usr/illumon/latest/etc/IRIS_GRADLE_VERSION inside the image.
Upgrading from Grizzly
Upgrading a Grizzly (1.20240517.*) deployment to Grizzly+ requires two migrations, which start_command.sh --upgrade performs automatically; neither needs another flag. Both work from the preserved files and properties, so they also run when the previous container is stopped or removed. The rest of your command line still applies, including -c if your certificates come from an internal PKI (see the list below).
- Private keys move. Grizzly shared the whole
authdirectory and kept its DSA/RSA private keys there. Grizzly+ shares onlypriv-keys. Becauseauthis regenerable and is moved into the backup directory, the upgrade first copiesauth/priv*.txtintopriv-keysso the keypairs registered in the etcd auth database keep working. Verify withmd5sumthatpriv-iris.base64.txtis unchanged after the upgrade. - Obsolete TLS assignments are migrated. Grizzly's Podman block in
iris-environment.proppoints TLS settings at/deephaven-tls, which Grizzly+ does not create. The upgrade comments out only the assignments that exactly match those stock paths, so the current service-specific defaults apply, and updates the stock SAML keystore and passphrase paths in theauthentication_serversection. Your own values, other properties, comments, and scoped settings are kept. Before importing the change into etcd, the upgrade saves the original asiris-environment.prop.before-migration.<suffix>in the shared configuration volume. If the export or import fails, the upgrade stops; fix the cause and run it again.
Before you upgrade from Grizzly, also:
- Pass
-cif you use a private CA. Grizzly built a TLS truststore automatically from the CA in the TLS volume. Grizzly+ builds one only when you pass-c/--ca-cert-file, which is required whenever you generated certificates withcreate_dh_cert.sh. See Supplying trusted certificates to Podman. - Move SAML and Active Directory certificates out of
auth. Grizzly kept theauthdirectory, including any certificates you added to it, in the shared configuration volume. Upgrading from Grizzly moves it intoupgrade-backup-<timestamp>/auth. In Grizzly+,authis inside the container and is not preserved when a pod is recreated, so keep such certificates in a host directory mounted with--additional-volume. See Persisting authentication certificates.
Manually resetting shared state
These steps are the manual equivalent of the preparation performed by --upgrade, which you should normally use instead; only --upgrade migrates Grizzly private keys. They apply only when dh_etcd_config.tgz exists in the shared configuration volume. If an installation that has never completed failed before dh_etcd_config.tgz was created, follow Retry a failed installation instead.
- Stop the existing infra pod (e.g.,
podman pod stop dh-infra-pod -t 90). Specify a timeout of at least 90 seconds to ensure that all Deephaven processes as well asmonitandetcdare able to exit cleanly. - If SELinux is in use, clear the MCS labels from the shared configuration and etcd volumes, as described in Fixing MCS label conflicts.
- Remove only the regenerable paths listed below. Preserve
dh_etcd_config.tgz,priv-keys,etcd-overrides.yaml, and all backups:
Note
With rootless Podman, files written by the container's accounts belong to subordinate UIDs, so remove them inside Podman's user namespace instead of with sudo. Either way, the removal must succeed completely; leftover files conflict with the reinitialized deployment.
After resetting the shared state, re-run start_command.sh with the --delete-pod flag (and the full machine list) to recreate the pods. During initialization, the infra container detects the preserved dh_etcd_config.tgz archive and re-extracts the etcd configuration files, reusing the existing etcd database.
Export, import, and back up data and configuration
Export
General instructions for non-MariaDB/MySQL ACL systems backup apply directly to Deephaven Podman deployments as well.
The system may not persist some sections of the pod filesystem outside of the pod:
/db/Usersstores direct user table data. You can configure it to use a mounted volume, but if you have not, usepodman cpto copy its data from a running or stopped pod. Copy the entire/db/Usersdirectory tree so you can import it to bring direct user table data to a different deployment./db/IntradayUserstores centrally managed user table data. You can configure it to use a mounted volume, but if you have not, usepodman cpto copy its data from a running or stopped pod. Copy the entire/db/IntradayUserdirectory tree so you can import it to bring centrally managed user table data to a different deployment./etc/sysconfig/illumon.d/java_libstores custom JARs and additional JARs not delivered as part of the Deephaven deployment. You can configure it to use a mounted volume, but if you have not, usepodman cpto copy its data from a running or stopped pod./etc/sysconfig/illumon.d/calendarsstores custom calendars not delivered as part of the Deephaven deployment. You can configure it to use a mounted volume, but if you have not, usepodman cpto copy its data from a running or stopped pod.
An example of a podman cp command to export direct user table data from a pod:
You can also export or back up any other paths in the pod that you have not mapped to additional volumes, or that cannot be mapped (for instance, /etc/sysconfig/illumon.d/resources), using podman cp commands.
Move pod data onto host volumes
To start keeping data that is stored inside the infra pod, copy it into host directories, then recreate the pod with the matching volume options. Use podman unshare cp -a, as below: it keeps each file's owner and permissions. In a multi-user deployment, Deephaven's accounts (such as dbmerge for intraday data) must own their files to keep writing them, and podman cp would make every copied file belong to the Podman user instead.
Run these commands as the user that runs the pods (rootless Podman), from the extracted deephaven-podman-<version> directory. Each copy prints nothing when it succeeds, and podman umount prints the container name, dh-infra. If a copy prints an error, do not recreate the pod: the data is still in the stopped pod.
-
Stop the pod and create the host directories:
-
Copy
/db/Intraday. Skip this step if the pod already uses-I(--intraday-data-volume)./db/Intradayis a Podman volume of the pod: -
Copy
/db/Users. Skip this step if the pod already uses-U(--users-volume)./db/Usersis in the container's own file system, whichpodman mountmakes readable: -
Copy
/db/IntradayUser. Skip this step if the pod already uses-W(--intradayuser-volume): -
Recreate the infra pod with
--delete-pod(or--upgrade), your usual options, and the new volume options:Pass these options every time you run
start_command.shfor this pod from now on.
Import
You can import configuration, data, and content from exports taken from other Deephaven systems. In general, you can import exports from the same major version. If you took the export from a different major version, please check with Deephaven support for specifics. You can import Vermilion exports to Vermilion+ and Grizzly, Vermilion+ exports to Grizzly and Grizzly+, and Grizzly exports to Grizzly+.
Important
Podman deployment overrides live in iris-endpoints.prop, which is regenerated and re-imported every time the infra pod is recreated. Do not copy them into iris-environment.prop: that file holds your custom settings, which are carried forward unchanged through every upgrade, so a copied override stays even after a later release needs a different value.
An iris-environment.prop taken from a Grizzly or earlier Podman deployment carries a ### Overrides for podman deployment: block. Before importing the file, comment out the stock TLS assignments in that block that point at /deephaven-tls, which does not exist in Grizzly+, and keep any settings you added. If SAML is configured, also change SAMLAuth.tls.keystore to /etc/sysconfig/deephaven/auth-user/webServices-keystore.p12 and SAMLAuth.tls.passphrase.file to /etc/sysconfig/deephaven/auth-user/.webapi_passphrase wherever they point at /deephaven-tls. The automatic migration runs only when the infra pod is recreated, not when you import a file yourself. A file taken from a system that was not deployed with start_command.sh carries no such block and needs no edit.
Back up
You can back up configuration and most data of Deephaven Podman deployments with the VOLUME_BASE_DIR directory tree. If network storage hosts some subdirectories of this tree, you may want to omit them from the backup, as the network storage system may already protect them. See File system backup and restoration (Podman) for the procedure.