Upgrading Deephaven in a Kubernetes environment

This guide covers upgrading Deephaven in a Kubernetes environment using Helm. Commands are shown for running in a Unix-like environment. It is very similar to the procedure described in the Kubernetes installation guide, minus some steps.

Note

The kubectl and helm commands shown in this guide do not explicitly set a namespace and assume that your environment is configured to operate in your intended namespace by default.

Prerequisites

You must meet the following prerequisites to upgrade Deephaven in Kubernetes:

  • An existing Deephaven deployment running in a Kubernetes cluster.
  • kubectl, docker and helm command line tools.
  • An artifact repository to which Docker images can be stored and from which Kubernetes pods may pull them. In this example, my-repo.dev/my-project/images/deephaven-install is used; replace this with your image repository URL.
  • A Deephaven distributable package containing a helm chart, support scripts, and Dockerfiles, e.g. deephaven-helm-1.20240517.344.tar.gz.

If using Deephaven pre-built images:

  • A Deephaven distributable package containing pre-built Docker images, e.g., deephaven-containers-1.20240517.344.tar.gz.

If building your own images:

  • A distributable containing Deephaven Enterprise, e.g., deephaven-enterprise-jdk17-1.20240517.344.tar.gz.
  • A distributable containing Deephaven Core+ worker, e.g., deephaven-coreplus-0.37.4-1.20240517.344-jdk17.tgz.

Note

You can either use pre-built container images from Deephaven or build your own images. Building images allows for customizing them to make your JARs available to workers or to add plugins.

Unzip the Deephaven helm chart

Unpack the deephaven-helm package:

tar -xzf deephaven-helm-1.20240517.344.tar.gz

Push the Deephaven images to your image repository

If you are using Deephaven's pre-built images, load them into your local Docker repository.

docker image load -i deephaven-containers-1.20240517.344.tar.gz
./deephaven-helm-1.20240517.344/docker/pushAll.sh --source-tag 1.20240517.344 my-repo.dev/my-project/images/deephaven-install 1.20240517.344

To build images from Deephaven Docker files, first change the directory to the docker subdirectory within the unzipped helm distribution and copy the Enterprise and Core+ distributions to the deephaven_base and db_query_worker_coreplus directories, respectively.

  • To build custom JARs into your images and make them available in Persistent Queries and Code Studios, create a zipped tar file containing your JARs, copy it to the deephaven_customer directory, and add a --customer-coreplus-jar flag to the buildAllForK8s.sh command.
  • To build plugin JARs into your images and make them available to non-worker processes, create zipped tar files containing your plugins, copy them to the deephaven_customer directory, and add up to 10 --customer-plugin flags to the buildAllForK8s.sh command.
# Example: Building Docker images with a custom plugin in my-plugins.tgz and custom jars in my-custom-jars.tgz

# Change to the Docker directory within the unzipped helm distribution.
cd deephaven-helm-1.20240517.344/docker

# Copy the enterprise and coreplus JARs to these directories.
cp /your/path/to/deephaven-enterprise-1.20240517.344 deephaven_base/
cp /your/path/to/deephaven-coreplus-0.37.4-1.20240517.344 db_query_worker_coreplus/

# Optional: if building a custom JAR into a Core+ worker image
cp /your/path/to/my-custom-jars.tgz deephaven_customer/

# Optional: if building a plugin into your image
cp /your/path/to/my-plugins.tgz deephaven_customer/

# Run the docker build scripts. The --customer-coreplus-jar and --customer-plugin flags are optional.
./buildAllForK8s.sh --jdk17 \
    --version 1.20240517.344 \
    --coreplus-tar deephaven-coreplus-0.37.4-1.20240517.344-jdk17.tgz \
    --customer-coreplus-jar my-custom-jars.tgz \
    --customer-plugin my-custom-plugins.tgz

# Push the Docker images to your image repository
./pushAll.sh --source-tag 1.20240517.344 my-repo.dev/my-project/images/deephaven-install 1.20240517.344

Change directory

The rest of the commands in this guide must be run from the helm subdirectory of the unpackaged helm distribution.

cd deephaven-helm-1.20240517.344/helm

Install the Deephaven helm chart

The Deephaven helm upgrade requires the name of your Deephaven helm installation. Run helm list to find it if you do not know what it is. Substitute your installation name in place of my-deephaven-deployment-name in the commands shown below.

The upgrade also requires the override YAML file used for the last upgrade or installation. If unavailable, you can run helm get values my-deephaven-deployment-name to get the values configured previously. Substitute your override YAML file name for my-override-values.yaml in the commands below.

Note

Properties for a helm chart are typically stored in one or more YAML files. If more than one is provided to the helm command, priority is given to the last (right-most) file specified with -f. Properties can also be provided with --set flags, which will take precedence over YAML settings.

Scale down your deployment and upgrade the Deephaven helm chart using your override YAML file with these commands:

# Scale down your deployment..
./setupTools/scaleAll.sh 0

# And run the upgrade..
helm upgrade --install my-deephaven-deployment-name deephaven \
    -f my-override-values.yaml \
    --set image.tag="1.20240517.344" \
    --debug

The installation takes a couple of minutes. You can see progress by tailing the log output of the upgrade job with the command kubectl logs -f job/my-deephaven-deployment-name-pre-release-hook.