--- title: Restart and Upgrade a Podman deployment sidebar_label: Podman --- This guide covers restarting and upgrading a Deephaven Podman deployment. It also covers exporting, importing, and backing up data and configuration. ## Restart and upgrade Restarting, in this context, means using the `start_command.sh` to initialize a cluster using persisted configuration data from the [`VOLUME_BASE_DIR`](./podman-installation.md#architecture) directory structure. Besides persisting configuration for the cluster, the [`VOLUME_BASE_DIR`](./podman-installation.md#architecture) directory structure also stores shared running configuration for use in nodes across the cluster. This shared data must be removed before `start_command.sh` can be used to reinitialize or upgrade the cluster. By default, all configuration stored in [etcd](../core-components/etcd.md) (ACLs, Persistent Queries, property files, data routing YAML, and schema) is preserved, as is intraday and historical data, which also includes Web UI dashboards and notebooks. What is not preserved by default, but can be [configured to be persistent](./podman-installation.md#volumes), is user tables, custom calendars, and contents of the `/etc/sysconfig/illumon.d/java_lib` path. > [!CAUTION] > Take a complete [backup](#back-up) before upgrading. Optionally, [export](#export) any pod-internal data — such as user tables, custom calendars, or custom JARs — before stopping the pod. Any such data not mapped to a Persistent Volume will be lost when the pod is stopped or replaced. If the upgrade causes a critical issue, restoring from a pre-upgrade backup is your only supported recovery path. Downgrading by switching to an older image without restoring from backup is not supported and may result in data corruption or an unrecoverable state. See [Reverting an upgrade](../backup-restore-migrate/backup-and-restore.md#reverting-an-upgrade) for details. To upgrade, download, or build new images for the new version of Deephaven (this could be a point release update in the same major version or a new major version). Refer to the release notes of a new version to see if there are any manual steps or caveats associated with the upgrade process. You should also use the `start_command.sh` that comes with the new version Deephaven Podman package, as there may be fixes or new functionality in the `start_command.sh` script itself. The process for rerunning `start_command.sh` to reinitialize or upgrade a Deephaven Podman deployment is: - Stop the existing infra pod (e.g., `podman pod stop dh-infra-pod -t 90`). Specify a timeout of at least 90 seconds to ensure that all Deephaven processes as well as `monit` and `etcd` are able to exit cleanly. - If [SELinux is in use](./podman-installation.md#selinux) Ensure that the MCS labels have been cleared from the shared configuration volume (the directory mounted at `/deephaven-shared-config` in the containers) from the `etcd` volume (the directory mounted at `/deephaven-etcd` in the containers). For example: ```shell chcon -vR -u system_u -r object_r -t container_file_t -l s0 "${VOLUME_BASE_DIR:?}"/deephaven-shared-config/ chcon -vR -u system_u -r object_r -t container_file_t -l s0 "${VOLUME_BASE_DIR:?}"/deephaven-etcd/{etc-etcd,var-lib-etcd} ``` This ensures that the newly created infra container can to write to these directories. - Remove all files from the shared config volume except for `dh_etcd_config.tgz`: ```shell sudo rm -vrf "${VOLUME_BASE_DIR:?}"/deephaven-shared-config/{auth,dh-config,etcd,installation,trust,cluster_status} ``` > [!NOTE] > If [SELinux is not in use](./podman-installation.md#selinux), it should be sufficient to run the following without `sudo`: > > ```shell > rm -vrf "${VOLUME_BASE_DIR:?}"/deephaven-shared-config/{auth,dh-config,etcd,installation,trust,cluster_status} > ``` > > This may result in access being denied when deleting some data, but it should remove data that would conflict with restarting the deployment. ## Export, import, and back up data and configuration ### Export [General instructions for non-MariaDB/MySQL ACL systems backup](../backup-restore-migrate/backup-and-restore.md) apply directly to Deephaven Podman deployments as well. Some sections of the pod filesystem which may not be persisted outside of the pod: - `/db/Users` stores direct user table data. It [can be configured to use a mounted volume](./podman-installation.md#start_commandsh-details), but if it is not, `podman cp` can be used to copy its data from a running or stopped pod. The entire `/db/Users` directory tree should be copied so it can be imported to bring direct user table data to a different deployment. - `/db/IntradayUser` stores centrally managed user table data. It [can be configured to use a mounted volume](./podman-installation.md#start_commandsh-details), but if it is not, `podman cp` can be used to copy its data from a running or stopped pod. The entire `/db/IntradayUser` directory tree should be copied so it can be imported to bring centrally managed user table data to a different deployment. - `/etc/sysconfig/illumon.d/java_lib` stores custom JARS and additonal JARs not delivered as part of the Deephaven deployment. It [can be configured to use a mounted volume](./podman-installation.md#start_commandsh-details), but if it is not, `podman cp` can be used to copy its data from a running or stopped pod. - `/etc/sysconfig/illumon.d/calendars` stores custom calendars not delivered as part of the Deephaven deployment. It [can be configured to use a mounted volume](./podman-installation.md#start_commandsh-details), but if it is not, `podman cp` can be used to copy its data from a running or stopped pod. An example of a `podman cp` command to export direct user table data from a pod: ```shell podman cp dh-infra:/db/Users/ ./BackupUsers/ ``` Any other paths in the pod that have not been mapped to [additional volumes](./podman-installation.md#start_commandsh-details), or maybe cannot be (for instance, `/etc/sysconfig/illumon.d/resources`), can also be exported/backed up using `podman cp` commands. ### Import Configuration, data, and content can be imported from exports taken from other Deephaven systems. In general, exports from the same major version can be imported. If the export was taken from a different major version, please check with Deephaven support for specifics. Vermilion exports can be imported to Vermilion+ and Grizzly, and Vermilion+ exports can be imported to Grizzly. > [!IMPORTANT] > There are special properties added to the initial `iris-environment.prop` files that are needed for Deephaven Podman deployments. If you import `iris-environment.prop` from a system that was not deployed using the Deephaven Podman `start_command.sh`, you must add this set of properties to the imported `iris-environment.prop`: > > ```properties > ### Overrides for Podman deployment: > > RemoteQueryDispatcher.workerStartupTimeoutMS=90000 > EmbeddedDbConsole.remoteDatabaseRequestTimeoutMS=60000 > > dbaclwriter.usessl=true > dbaclwriter.acceptplaintext=false > dbaclwriter.acceptssl=true > > Webapi.server.cus.enabled=true > Webapi.server.cus.home=/etc/sysconfig/deephaven/cus > > tls.keystore=/deephaven-tls/keystore.p12 > tls.passphrase.file=/deephaven-tls/.keystore_passphrase > > tls.truststore=/deephaven-tls/truststore.p12 > tls.truststore.passphrase.file=/deephaven-tls/truststore_passphrase > > WebServer.tls.keystore=/deephaven-tls/keystore.p12 > WebServer.tls.passphrase.file=/deephaven-tls/.keystore_passphrase > > RemoteQueryDispatcher.tls.keystore=/deephaven-tls/keystore.p12 > RemoteQueryDispatcher.tls.passphrase.file=/deephaven-tls/.keystore_passphrase > > WSCommServer.tls.keystore=/deephaven-tls/keystore.p12 > WSCommServer.tls.passphrase.file=/deephaven-tls/.keystore_passphrase > > authentication.server.overrideAuthority= > PersistentQueryController.overrideAuthority= > ``` ### Back up Configuration and at least most data of Deephaven Podman deployments can be backed up with the [`VOLUME_BASE_DIR`](./podman-installation.md#architecture) directory tree. However, if some subdirectories of this tree are actually hosted on network storage, you may want to omit these from backup, as they may already be backed up and/or otherwise protected by the network storage system. ## Related documentation - [Podman Quickstart](../../quickstarts/podman-quickstart.md) - [Podman Installation](./podman-installation.md) - [Troubleshooting a Podman deployment](../troubleshooting/podman-troubleshooting.md) - [Kubernetes](../kubernetes/kubernetes-install-guide.md)