---
title: Restart and Upgrade a Podman deployment
sidebar_label: Podman
---

This guide covers restarting and upgrading a Deephaven Podman deployment. It also covers exporting, importing, and backing up data and configuration.

## Restart and upgrade

Restarting, in this context, means using the `start_command.sh` to initialize a cluster using persisted configuration data from the [`VOLUME_BASE_DIR`](./podman-installation.md#architecture) directory structure.

Besides persisting configuration for the cluster, the [`VOLUME_BASE_DIR`](./podman-installation.md#architecture) directory structure also stores shared running configuration for use in nodes across the cluster. This shared data must be moved aside before `start_command.sh` reinitializes or upgrades the cluster; the [`--upgrade` flag](#upgrade-with---upgrade) handles this automatically by moving the data into a timestamped backup directory.

By default, Deephaven preserves all configuration stored in [etcd](../core-components/etcd.md) (ACLs, Persistent Queries, property files, data routing YAML, and schema), as well as intraday and historical data, which also includes Web UI dashboards and notebooks. Deephaven does not preserve user tables, custom calendars, or contents of the `/etc/sysconfig/illumon.d/java_lib` path by default, but you can [configure them to persist](./podman-installation.md#volumes).

> [!CAUTION]
> Take a complete [backup](#back-up) before upgrading. Optionally, [export](#export) any pod-internal data — such as user tables, custom calendars, or custom JARs — before stopping the pod. You will lose any such data not mapped to a Persistent Volume when the pod is stopped or replaced. If the upgrade causes a critical issue, restoring from a pre-upgrade backup is your only supported recovery path. Deephaven does not support downgrading by switching to an older image without restoring from backup; doing so may result in data corruption or an unrecoverable state. See [Reverting an upgrade](../backup-restore-migrate/backup-and-restore.md#reverting-an-upgrade) for details.

To upgrade, download, or build new images for the new version of Deephaven (this could be a point release update in the same major version or a new major version). Refer to the release notes of a new version to see if there are any manual steps or caveats associated with the upgrade process. You should also use the `start_command.sh` that comes with the new version Deephaven Podman package, as there may be fixes or new functionality in the `start_command.sh` script itself.

### Upgrade with `--upgrade`

The `--upgrade` flag of `start_command.sh` automates the upgrade steps. Run `start_command.sh --upgrade` on each node, with the same arguments used to deploy the cluster — including the pod hostname (`-h`) and, on the infra node, the complete `-q` list of query pods:

```shell
./start_command.sh --upgrade -t infra -h infra-host.mydomain.net \
    -q dh-query1-pod=query1-host.mydomain.net \
    -q dh-query2-pod=query2-host.mydomain.net
```

For each pod, `--upgrade`:

- Uses the supplied machine names as the authoritative cluster topology. The `iris-endpoints.prop` file is regenerated from the supplied values during initialization, so the full machine list is required on every infra invocation. If the supplied list differs from the existing deployment, a notice showing both lists is printed.
- Stops the existing pod with a 90-second timeout — so that all Deephaven processes as well as `monit` and `etcd` exit cleanly — and then removes it.
- On the infra node, moves the regenerable shared configuration (`auth`, `trust`, `dh-config`, `etcd`, `installation`, and `cluster_status`) into a timestamped `upgrade-backup-<timestamp>` directory under the shared configuration volume. The `dh_etcd_config.tgz` file is preserved in place so the new deployment re-attaches to the existing etcd database, and the `priv-keys` directory is preserved because the public halves of the keypairs it holds are registered in that database — regenerating them would leave each Deephaven process holding a private key the auth database no longer recognizes.
- If [SELinux is in use](./podman-installation.md#selinux), clears the MCS labels from the shared configuration and etcd volumes. When Podman mounts volumes with the `:z` option, SELinux assigns Multi-Category Security (MCS) labels to restrict access. A new pod gets a different MCS category than the old pod, causing "Permission Denied" errors on files the previous pod created.
- Creates and starts a new pod from the new image.

### Upgrading from Grizzly

Upgrading a Grizzly (`1.20240517.*`) deployment to Grizzly+ requires two migrations that no later upgrade does. `start_command.sh` detects the Grizzly version on the running pod and performs both automatically; neither needs a flag. Note that the rest of your command line still applies, including `-c` if your certificates come from an internal PKI — see below.

- **Private keys move.** Grizzly shared the whole `auth` directory and kept its DSA/RSA private keys there. Grizzly+ shares only `priv-keys`. Because `auth` is regenerable and is moved into the backup directory, the upgrade first copies `auth/priv*.txt` into `priv-keys` so the keypairs registered in the etcd auth database keep working. Verify with `md5sum` that `priv-iris.base64.txt` is unchanged after the upgrade.
- **Grizzly's property overrides are removed.** Grizzly wrote its Podman deployment overrides into `iris-environment.prop`, which Deephaven never rewrites during an upgrade. Those overrides point the TLS keystore and truststore at absolute paths under `/deephaven-tls`, a directory Grizzly+ does not create, so leaving them in place stops the cluster from starting. The upgrade removes only the properties Deephaven itself wrote into that block from the copy held in [etcd](../core-components/etcd.md); any property it does not recognize is kept and logged. Before changing anything, it saves the original as `iris-environment.prop.before-scrub` in the `upgrade-backup-<timestamp>` directory and logs that path. Grizzly+ writes the equivalent settings to the generated `iris-endpoints.prop` instead.

> [!NOTE]
> Grizzly built a TLS truststore automatically from the CA in the TLS volume. Grizzly+ builds one only when you pass `-c` / `--ca-cert-file`. If your deployment uses a private CA — which is the case whenever you generated certificates with `create_dh_cert.sh` — pass `-c` on the upgrade command line, and on every subsequent `start_command.sh` invocation:
>
> ```shell
> ./start_command.sh --upgrade -c "${VOLUME_BASE_DIR:?}/deephaven-tls/ca.crt" ...
> ```

`start_command.sh` refuses to start a pod from an image whose Deephaven version differs from the deployed version unless `--upgrade` is passed. The version is read from `/usr/illumon/latest/etc/IRIS_GRADLE_VERSION` inside the image.

> [!CAUTION]
> The upgrade moves the `auth` directory into the backup directory. The `auth` directory contains SAML and Active Directory certificates if you configured them. If you have not stored these certificates in a persistent location (such as the `java_lib` volume), restore them from the backup directory after the upgrade. See [Persisting authentication certificates](./podman-installation.md#persisting-authentication-certificates) for how to configure certificate persistence.

> [!CAUTION]
> The SELinux relabeling performed by `--upgrade` uses `chcon ... -l s0`, which clears ALL SELinux MCS categories, including any user-applied categories. If the relabeling fails (for example, because it requires `sudo`), `start_command.sh` prints a warning; run the commands below manually. See [Troubleshooting SELinux in Podman](../troubleshooting/podman-troubleshooting.md#troubleshooting-selinux-in-podman) for details on MCS labels and alternative approaches.

### Manually resetting shared state

If an upgrade or installation fails partway (for example, the cluster status is stuck at `INITIALIZING`), reset the shared state manually and retry. These steps are also the manual equivalent of the preparation performed by `--upgrade`:

- Stop the existing infra pod (e.g., `podman pod stop dh-infra-pod -t 90`). Specify a timeout of at least 90 seconds to
  ensure that all Deephaven processes as well as `monit` and `etcd` are able to exit cleanly.
- If [SELinux is in use](./podman-installation.md#selinux), clear the MCS labels from the shared configuration and etcd volumes:

  ```shell
  chcon -vR -u system_u -r object_r -t container_file_t -l s0 "${VOLUME_BASE_DIR:?}"/deephaven-shared-config/
  chcon -vR -u system_u -r object_r -t container_file_t -l s0 "${VOLUME_BASE_DIR:?}"/deephaven-etcd/{etc-etcd,var-lib-etcd}
  ```

- Remove all files from the shared config volume except for `dh_etcd_config.tgz`:
  ```shell
  sudo rm -vrf "${VOLUME_BASE_DIR:?}"/deephaven-shared-config/{auth,dh-config,etcd,installation,trust,cluster_status}
  ```

> [!NOTE]
> If [SELinux is not in use](./podman-installation.md#selinux), it should be sufficient to run the following without `sudo`:
>
> ```shell
> rm -vrf "${VOLUME_BASE_DIR:?}"/deephaven-shared-config/{auth,dh-config,etcd,installation,trust,cluster_status}
> ```
>
> The system may deny access when deleting some data, but this should remove data that would conflict with restarting the deployment.

After resetting the shared state, re-run `start_command.sh` with the `--delete-pod` flag (and the full machine list) to recreate the pods. During initialization, the infra container detects the preserved `dh_etcd_config.tgz` archive and re-extracts the etcd configuration files, reusing the existing etcd database.

## Export, import, and back up data and configuration

### Export

[General instructions for non-MariaDB/MySQL ACL systems backup](../backup-restore-migrate/backup-and-restore.md) apply directly to Deephaven Podman deployments as well.

The system may not persist some sections of the pod filesystem outside of the pod:

- `/db/Users` stores direct user table data. You [can configure it to use a mounted volume](./podman-installation.md#start_commandsh-details), but if you have not, use `podman cp` to copy its data from a running or stopped pod. Copy the entire `/db/Users` directory tree so you can import it to bring direct user table data to a different deployment.
- `/db/IntradayUser` stores centrally managed user table data. You [can configure it to use a mounted volume](./podman-installation.md#start_commandsh-details), but if you have not, use `podman cp` to copy its data from a running or stopped pod. Copy the entire `/db/IntradayUser` directory tree so you can import it to bring centrally managed user table data to a different deployment.
- `/etc/sysconfig/illumon.d/java_lib` stores custom JARs and additional JARs not delivered as part of the Deephaven deployment. You [can configure it to use a mounted volume](./podman-installation.md#start_commandsh-details), but if you have not, use `podman cp` to copy its data from a running or stopped pod.
- `/etc/sysconfig/illumon.d/calendars` stores custom calendars not delivered as part of the Deephaven deployment. You [can configure it to use a mounted volume](./podman-installation.md#start_commandsh-details), but if you have not, use `podman cp` to copy its data from a running or stopped pod.

An example of a `podman cp` command to export direct user table data from a pod:

```shell
podman cp dh-infra:/db/Users/ ./BackupUsers/
```

You can also export or back up any other paths in the pod that you have not mapped to [additional volumes](./podman-installation.md#start_commandsh-details), or that cannot be mapped (for instance, `/etc/sysconfig/illumon.d/resources`), using `podman cp` commands.

### Import

You can import configuration, data, and content from exports taken from other Deephaven systems. In general, you can import exports from the same major version. If you took the export from a different major version, please check with Deephaven support for specifics. You can import Vermilion exports to Vermilion+ and Grizzly, Vermilion+ exports to Grizzly and Grizzly+, and Grizzly exports to Grizzly+.

> [!IMPORTANT]
> Podman deployment overrides live in `iris-endpoints.prop`, which `start_command.sh` regenerates and re-imports every time the cluster initializes or upgrades. Do not copy them into `iris-environment.prop`: that file is yours and Deephaven never rewrites it, so an override placed there outlives the release that needed it. Releases up to Grizzly did write these overrides into `iris-environment.prop`; upgrading such a deployment removes that block automatically.
>
> When you import an `iris-environment.prop` taken from another system, never add Podman overrides to it. `start_command.sh` supplies them through `iris-endpoints.prop`.
>
> A file taken from a Grizzly or earlier Podman deployment carries a `### Overrides for podman deployment:` block. Delete that block before importing the file. Its properties point at absolute paths under `/deephaven-tls`, a directory Grizzly+ does not create, and a cluster that reads them fails to start. A file taken from a system that was not deployed with `start_command.sh` carries no such block and needs no edit.

### Back up

You can back up configuration and at least most data of Deephaven Podman deployments with the [`VOLUME_BASE_DIR`](./podman-installation.md#architecture) directory tree. However, if network storage hosts some subdirectories of this tree, you may want to omit these from backup, as the network storage system may already back them up and/or otherwise protect them.

## Related documentation

- [Podman Quickstart](../../quickstarts/podman-quickstart.md)
- [Podman Installation](./podman-installation.md)
- [Troubleshooting a Podman deployment](../troubleshooting/podman-troubleshooting.md)
- [Kubernetes](../kubernetes/kubernetes-install-guide.md)