Restart and Upgrade a Podman deployment

This guide covers restarting and upgrading a Deephaven Podman deployment. It also covers exporting, importing, and backing up data and configuration.

Restart and upgrade

Restarting, in this context, means using the start_command.sh to initialize a cluster using persisted configuration data from the VOLUME_BASE_DIR directory structure.

Besides persisting configuration for the cluster, the VOLUME_BASE_DIR directory structure also stores shared running configuration for use in nodes across the cluster. This shared data must be removed before start_command.sh can be used to reinitialize or upgrade the cluster.

By default, all configuration stored in etcd (ACLs, Persistent Queries, property files, data routing YAML, and schema) is preserved, as is intraday and historical data, which also includes Web UI dashboards and notebooks. What is not preserved by default, but can be configured to be persistent, is user tables, custom calendars, and contents of the /etc/sysconfig/illumon.d/java_lib path.

Important

It is recommended to backup and/or export configuration data before restarting a deployment.

To upgrade, download, or build new images for the new version of Deephaven (this could be a point release update in the same major version or a new major version). Refer to the release notes of a new version to see if there are any manual steps or caveats associated with the upgrade process. You should also use the start_command.sh that comes with the new version Deephaven Podman package, as there may be fixes or new functionality in the start_command.sh script itself.

The process for rerunning start_command.sh to reinitialize or upgrade a Deephaven Podman deployment is:

  • Stop the existing infra pod (e.g., podman pod stop dh-infra-pod -t 90). Specify a timeout of at least 90 seconds to ensure that all Deephaven processes as well as monit and etcd are able to exit cleanly.
  • If SELinux is in use Ensure that the MCS labels have been cleared from the shared configuration volume (the directory mounted at /deephaven-shared-config in the containers) from the etcd volume (the directory mounted at /deephaven-etcd in the containers). For example: 
    chcon -vR -u system_u -r object_r -t container_file_t -l s0 "${VOLUME_BASE_DIR:?}"/deephaven-shared-config/
chcon -vR -u system_u -r object_r -t container_file_t -l s0 "${VOLUME_BASE_DIR:?}"/deephaven-etcd/{etc-etcd,var-lib-etcd}
    This ensures that the newly created infra container can to write to these directories.
  • Remove all files from the shared config volume except for dh_etcd_config.tgz: 
    sudo rm -vrf "${VOLUME_BASE_DIR:?}"/deephaven-shared-config/{auth,dh-config,etcd,installation,trust,cluster_status}

Note

If SELinux is not in use, it should be sufficient to run the following without sudo:

rm -vrf "${VOLUME_BASE_DIR:?}"/deephaven-shared-config/{auth,dh-config,etcd,installation,trust,cluster_status}

This may result in access being denied when deleting some data, but it should remove data that would conflict with restarting the deployment.

Export, import, and back up data and configuration

Export

General instructions for non-MariaDB/MySQL ACL systems backup apply directly to Deephaven Podman deployments as well.

Some sections of the pod filesystem which may not be persisted outside of the pod:

  • /db/Users stores direct user table data. It can be configured to use a mounted volume, but if it is not, podman cp can be used to copy its data from a running or stopped pod. The entire /db/Users directory tree should be copied so it can be imported to bring direct user table data to a different deployment.
  • /db/IntradayUser stores centrally managed user table data. It can be configured to use a mounted volume, but if it is not, podman cp can be used to copy its data from a running or stopped pod. The entire /db/IntradayUser directory tree should be copied so it can be imported to bring centrally managed user table data to a different deployment.
  • /etc/sysconfig/illumon.d/java_lib stores custom JARS and additonal JARs not delivered as part of the Deephaven deployment. It can be configured to use a mounted volume, but if it is not, podman cp can be used to copy its data from a running or stopped pod.
  • /etc/sysconfig/illumon.d/calendars stores custom calendars not delivered as part of the Deephaven deployment. It can be configured to use a mounted volume, but if it is not, podman cp can be used to copy its data from a running or stopped pod.

An example of a podman cp command to export direct user table data from a pod:

podman cp dh-infra:/db/Users/ ./BackupUsers/

Any other paths in the pod that have not been mapped to additional volumes, or maybe cannot be (for instance, /etc/sysconfig/illumon.d/resources), can also be exported/backed up using podman cp commands.

Import

Configuration, data, and content can be imported from exports taken from other Deephaven systems. In general, exports from the same major version can be imported. If the export was taken from a different major version, please check with Deephaven support for specifics. Vermilion exports can be imported to Vermilion+ and Grizzly, and Vermilion+ exports can be imported to Grizzly.

Important

There are special properties added to the initial iris-environment.prop files that are needed for Deephaven Podman deployments. If you import iris-environment.prop from a system that was not deployed using the Deephaven Podman start_command.sh, you must add this set of properties to the imported iris-environment.prop:

### Overrides for Podman deployment:

RemoteQueryDispatcher.workerStartupTimeoutMS=90000
EmbeddedDbConsole.remoteDatabaseRequestTimeoutMS=60000

dbaclwriter.usessl=true
dbaclwriter.acceptplaintext=false
dbaclwriter.acceptssl=true

Webapi.server.cus.enabled=true
Webapi.server.cus.home=/etc/sysconfig/deephaven/cus

tls.keystore=/deephaven-tls/keystore.p12
tls.passphrase.file=/deephaven-tls/.keystore_passphrase

tls.truststore=/deephaven-tls/truststore.p12
tls.truststore.passphrase.file=/deephaven-tls/truststore_passphrase

WebServer.tls.keystore=/deephaven-tls/keystore.p12
WebServer.tls.passphrase.file=/deephaven-tls/.keystore_passphrase

RemoteQueryDispatcher.tls.keystore=/deephaven-tls/keystore.p12
RemoteQueryDispatcher.tls.passphrase.file=/deephaven-tls/.keystore_passphrase

WSCommServer.tls.keystore=/deephaven-tls/keystore.p12
WSCommServer.tls.passphrase.file=/deephaven-tls/.keystore_passphrase

authentication.server.overrideAuthority=
PersistentQueryController.overrideAuthority=

Back up

Configuration and at least most data of Deephaven Podman deployments can be backed up with the VOLUME_BASE_DIR directory tree. However, if some subdirectories of this tree are actually hosted on network storage, you may want to omit these from backup, as they may already be backed up and/or otherwise protected by the network storage system.