Kubernetes troubleshooting

This guide helps you troubleshoot a Deephaven system deployed in Kubernetes, especially if you have limited experience with Kubernetes. It assumes you are familiar with concepts such as Pods, Deployments, StatefulSets, and Secrets.

For general system troubleshooting guidance applicable to all platforms, refer to the main troubleshooting guide. Much of that guidance still applies on Kubernetes. For example, Persistent Queries run in workers, and worker information is available in the Process Event Log, which you can query from a Code Studio as described in Debugging Code Studio.

How do I see the deployment names I used to install Deephaven, etcd, or NFS?

Run helm list to see the deployment names. A typical installation has three Helm releases: the Deephaven application, the deephaven-etcd chart (the etcd cluster), and the deephaven-nfs chart (the shared NFS server). The etcd resources are named after the etcd release (StatefulSet/service/secret <etcd-release>, or <etcd-release>-etcd if the name does not already end in -etcd), and the NFS server Deployment is deephaven-nfs-server behind a service named deephaven-nfs.

How do I see the Kubernetes objects in the Deephaven cluster?

You can use kubectl get to see the list of objects of a certain type.

Example usages of kubectl get

On OpenShift, how do I check which Security Context Constraint admitted a pod?

OpenShift records the SCC that admitted each pod in an annotation. Every Deephaven pod, including worker pods started at runtime, should show nonroot-v2; the bundled NFS server shows privileged in its own namespace.

A pod admitted under restricted-v2 runs as a namespace-allocated user ID rather than Deephaven's own, and fails with permission errors on its volumes: the nonroot-v2 RoleBinding is missing. A pod admitted under anyuid or privileged where you expected nonroot-v2 means one of those SCCs is granted to a ServiceAccount the pod uses; OpenShift prefers them because they carry a higher priority. See OpenShift installation.

How do I uninstall a Deephaven deployment?

There is a dh_uninstall.sh script in the setupTools directory of the Deephaven helm chart distribution. Running this script with no arguments tries to determine the Helm release names and the persistent volume prefix that was configured when Deephaven was installed, or you may provide them explicitly. Run dh_uninstall.sh -h or dh_uninstall.sh --help for usage information. The script asks for confirmation before proceeding with an uninstall.

How do I view logs for a container?

Process logs are sent to standard out, so you can view them using the kubectl logs command. First, identify the containers in the pod, then you can tail the log of the specific container. Use the -f option to tail the log, not just print it.

Note that you can use the object type and static name instead of the pod name, which can change. However, if multiple pods exist, it selects only one to operate on. Deephaven deployments have one pod per deployment, so this approach is safe:

Depending on your version, some Deephaven components may be a statefulset instead of a deployment, and those may have multiple pods.

Pod names for statefulset objects are predictably named, so you can easily get logs for them by pod name without first listing pod names.

How do I view Envoy logs?

Envoy logs to /tmp/envoy-rds.log on the Envoy deployment's pod, but Envoy is deployed with a distroless image that does not allow an interactive session. To view the logs, patch the pod to add an ephemeral container. Follow these steps:

Proxy the Kubernetes API server to your local machine:

Run this command in another terminal, substituting your namespace and pod name where indicated, and using an image of your choice that has bash, or another shell. You can use any image available to you in your environment, such as busybox or ubuntu, or you can use the image for the Deephaven management shell pod which may be found by running kubectl get deploy/management-shell -o jsonpath='{.spec.template.spec.containers[].image}'.

This command patches the pod to add an ephemeral container named "debugger" that you can use to access the logs.

You can then exec into the ephemeral container to view the logs:

How do I run commands on the command line as shown elsewhere in the documentation?

The Deephaven Kubernetes deployment includes a management-shell deployment with a single pod for this purpose. You can use the kubectl exec command to open a session on the pod and run commands.

Here is an example using a management shell to access a Deephaven config file.

How do I scale my Deephaven deployment down or up?

Scale your deployment with the script provided in the setupTools directory of the Deephaven Helm chart distribution, providing an argument of down or up.

Why do I get permission ("forbidden") errors when installing, upgrading, or scaling?

helm, kubectl, and the admin scripts (dh_helm, scaleAll.sh, restartAll.sh) act on Kubernetes objects using your own credentials, not the chart's ServiceAccounts. A forbidden error means the identity running the command is missing a permission — commonly on statefulsets/statefulsets/scale, deployments, persistentvolumes, or the chart's RBAC objects. The simplest fix is to run as a cluster (or namespace) administrator; if you must use a least-privilege identity, grant it the full set of permissions listed in Permissions required to install, upgrade, and administer the chart. This is separate from — and should not be worked around by editing — the runtime Roles that the chart binds to its own pods.

How do I update the deephaven-tls webserver certificate?

The webserver certificate for the front url of your Deephaven deployment is stored in a Kubernetes secret named deephaven-tls. You can delete the secret with the command kubectl delete secret deephaven-tls and recreate it with a new certificate as described in this step of the install guide.

After the new secret is created, you must scale down the deployment and run a Helm upgrade as described in Install the Deephaven Helm chart before it is effective. The Helm upgrade process runs a script that applies the new certificate to internal Deephaven trust stores.

How can I tell if my persistent volume storage class allows for volume expansion?

First, get a listing of your persistent volume claims (PVCs). The output includes a STORAGECLASS column.

You can then get information about that storage class. If ALLOWVOLUMEEXPANSION is listed as true, then it is expandable.

How do I increase the DIS volume size used for /db/Intraday or /db/IntradayUser?

First check to see that your volume's storage class allows volume expansion. For storage classes that allow it, you can increase the volume sizes by adding or editing this section of the override YAML file you reference in your Helm upgrade command. You can increase the size of one or both, as shown here.

Then scale down your deployment with the script provided in the Deephaven Helm chart and run your Helm upgrade as documented in the upgrade guide.

How do I increase the volume size used for /var/log/deephaven/binlogs in a deployment?

First check to see that your volume's storage class allows volume expansion. The binlogs used in the Deephaven Helm chart for a deployment are a persistent volume claim that can be expanded by adding or editing the configuration in the override YAML file you reference in your Helm upgrade command.

Then scale down your deployment with the script provided in the Deephaven Helm chart and run your Helm upgrade as documented in the upgrade guide.

How do I increase the volume size used for /var/log/deephaven/binlogs in a statefulset?

First check to see that your volume's storage class allows volume expansion. The binlogs persistent volume claim (PVC) is part of the definition of the statefulset, so it is handled differently than a binlog PVC used for a deployment. In this example, we upgrade the capacity of the controller's binlogs.

Identify the binlog PVC names by running kubectl get pvc. If you have two pods for the controller, there are two PVCs named binlogs-controller-0 and binlogs-controller-1. To edit a PVC, use the command kubectl edit pvc binlogs-controller-0.

Change the YAML value at spec.resources.requests.storage to your desired capacity, then save and exit. In this YAML snippet, we increase a volume from its previous value to 10Gi.

Edit all PVCs in the statefulset before continuing. The volume should resize asynchronously, and you can check progress with kubectl get pvc binlogs-controller-0 or kubectl describe pvc binlogs-controller-0.

You should know the override YAML used to install or upgrade the Helm chart. If you don't have it available, create a new file by running helm get values my-deephaven-deployment-name. Add or edit the resources section to include the new capacity value of your PVCs. If this is not updated, future Helm upgrades may result in an error.

The next step is deleting the statefulset.

Then scale down your deployment with the script provided in the Deephaven Helm chart and run your Helm upgrade as documented in the upgrade guide, using the override YAML file edited as part of this process.

How do I decrease the size of a persistent volume?

There is no direct support for decreasing a volume's capacity in Kubernetes.

How do I set up git SSH credentials for Persistent Queries in a Kubernetes deployment?

The Git authentication documentation describes creating a key pair and known_hosts file in the ~irisadmin/.ssh directory on a Deephaven server. On Kubernetes, there is no server to run those commands on. Instead, you create a Kubernetes secret that holds the keys and the known_hosts file, and the Helm chart mounts that secret into the Controller pods. The Controller runs /usr/bin/ssh for Git operations, so the standard OpenSSH rules for key names and known_hosts entries apply.

Create the keys and known_hosts file

The Helm chart mounts the secret read-only at /db/TempFiles/irisadmin/.ssh, so ssh cannot add new host keys to known_hosts when the Controller connects. The known_hosts file must contain every host key the Controller needs before you create the secret.

Run ssh-keyscan from any machine that can reach the Git server. If the Git server listens on a port other than 22, pass it with -p:

Check the fingerprints against the ones your Git administrator publishes before you trust them:

Keep the following in mind:

• When the port is not 22, ssh looks up the host as [host]:port (for example, [git.mycompany.net]:7999). ssh-keyscan -p writes entries in this form, so do not remove the brackets.
• ssh matches known_hosts entries against the host exactly as written in the repository uri. If the uri uses an IP address, known_hosts needs an entry for that IP address, separate from any entry for the hostname. Run ssh-keyscan against each name or address the Controller uses.

There is no SSH config file in the Controller pod, so the private key must use one of the default names that ssh looks for, such as id_ed25519 or id_rsa.

Configure the repository on the Git volume

When gitCreateVolume is true, the Helm chart creates a persistent volume for each Controller pod and mounts it at the path set by controller.gitMountPath, which defaults to /db/TempFiles/irisadmin/git. Set the iris.scripts.repo.<repo_name>.root property to a directory on this volume before installing or upgrading the Helm release. For an existing deployment, change this setting before upgrading, because applying the volume mount can restart the Controller pods:

Use a path that resolves to the Git volume, rather than one that remains inside the Controller workspace. For example, git/team1 resolves inside the workspace at /db/TempFiles/irisadmin/iris_controller, which is an emptyDir and is removed when the pod is recreated. With the default Git mount path, ../git/team1 resolves to /db/TempFiles/irisadmin/git/team1 and is persistent. If your existing repository root resolves inside the workspace, change it before restarting the Controllers.

Create the secret and mount it

Each key in the secret becomes a file in /db/TempFiles/irisadmin/.ssh:

Add the following to your Helm chart override YAML file, then install or upgrade the Helm release to mount the secret in the Controller pods:

Test the connection

The Controller runs ssh as irisadmin in the controller container, so running the same command there tests the same key and known_hosts file. Run it in every Controller pod, replacing the port and host with the values from your repository uri:

-o BatchMode=yes makes ssh fail instead of prompting, as it does for the Controller. A successful test prints a greeting or a message that shell access is not available. If the test fails, see Fix connection failures.

Fix connection failures

If ssh reports Host key verification failed, known_hosts does not contain a key that matches what the server presented for that host and port. The entry may be missing, or it may be stale because the Git server's host key changed. Rebuild the file with ssh-keyscan and verify the new fingerprints with your Git administrator, as described in Create the keys and known_hosts file.

If ssh reports that it could not resolve the hostname, the Git server's hostname is not in the cluster's DNS, even if it resolves from your workstation. Either use the Git server's IP address in the repository uri (with a matching IP address entry in known_hosts), or have your cluster administrator add the hostname to the cluster DNS.

To replace only the known_hosts file in an existing secret, without needing the private key on your local machine, patch the secret:

Secret volume updates are asynchronous. Kubernetes documents the delay as the kubelet sync period plus its Secret cache propagation delay, so do not assume the new file is visible immediately. Because the chart mounts the SSH secret as a directory rather than with subPath, the updated file becomes visible without restarting the pod. Once it is visible, rerun the connection test.

Repositories that already failed to clone when a Controller started are not retried. After the test succeeds, restart every Controller pod by following the suggested Controller restart procedure, restarting non-leaders before the leader.

How do I prevent a Persistent Query's worker pod from getting removed immediately after the process terminates?

In the Settings tab of the Persistent Query configuration editor, use the Extra Env Variables input box to define WORKER_POD_EXIT_WAIT_SEC to delay the pod removal; for example, WORKER_POD_EXIT_WAIT_SEC=600.

Use this judiciously — only when required for troubleshooting — because extended delays to pod deletions cause query restarts and system shutdowns to wait until the processes terminate.

How do I troubleshoot Deephaven service pods that get killed by Kubernetes?

When Kubernetes kills and restarts a pod due to resource constraints or other issues, you'll see a non-zero RESTARTS count in the pod listing. To identify problematic pods, first check which ones have been restarted:

You'll see output like this:

Note the three restarts above. There are a couple of common possibilities causing this that can be corrected by reconfiguring the deployments.

To determine why a pod was terminated and restarted, examine its details using kubectl describe:

Most Deephaven pods contain two containers (a service container and a tailer container). The kubectl describe output shows separate sections for each. While it's more common for service containers to be killed due to resource constraints, check both sections for the Last State information shown above.

In this example, the Reason: OOMKilled indicates the configuration-server container was terminated due to exceeding its memory limit. The chart's default limits (set under resources in the chart's values.yaml) suit typical deployments, but heavily used systems or some platforms may need more.

To address this, you can increase the resource allocation using kubectl set resources:

• Specify both the workload (statefulset/<name> or deployment/<name>) and the container name (-c parameter). The aclwriter, authserver, configuration-server, and controller services are StatefulSets; the other Deephaven services are Deployments.
• This patch is temporary; the next helm upgrade overwrites it.
• Applying the patch causes Kubernetes to replace the affected pod with one using the new settings.

Set values persistently

For a permanent solution that persists across Helm upgrades, add resource overrides to your values YAML file (commonly named my-values.yaml):

These overrides take effect the next time you run a Helm upgrade that references this values file:

Known changes from defaults needed on platforms other than GKE

Ephemeral storage allocations

Another possibility for Kubernetes killing a container is because it exceeded its ephemeral storage. Ephemeral storage in Deephaven Kubernetes containers is largely used for log files. If logging activity is high, due to heavy use, or debug logging enabled, it may be necessary to increase the ephemeral-storage requests and limits settings. These can be patched or overridden the same way as memory requests and limits were modified in the examples above.

How do I debug a process using a remote debugger?

Follow the steps below to debug a non-worker Deephaven Java process with a remote debugger.

First, configure your override file with port and debug information by adding configurations for the process you want to debug.

Then scale down your deployment with the script provided in the Deephaven Helm chart and run your Helm upgrade as documented in the upgrade guide, using the override YAML file edited as part of this process.

Set up port forwarding from your local machine to the pod using the debugPort configured earlier. If the debugPort is not known, find the name of the pod you want to debug by running kubectl get pods.

After your pod has restarted, you can run a remote debugger as normal.

How do I make a certificate or other piece of external data available within my Persistent Queries?

You can create a secret and mount its data in workers by adding a workerExtraSecrets section to your chart override YAML and running a Helm upgrade as documented in the upgrade guide. All keys of the secret are mapped as files under the workerExtraSecrets.mountPath on your worker pods.

Similarly, you can also mount an existing persistent volume claim to worker pods with a workerExtraVolumes section like this.

To add JARs from a mounted volume to the classpath, see Add custom JARs using a persistent volume.

How do I add custom JARs using a persistent volume?

To add JARs without rebuilding your container images, stage them on a persistent volume, mount it to pods, and add the mounted path to the classpath. For the complete procedure, see Add custom JARs using a persistent volume.

How do I look at the disks used by the Data Import Server?

You may want to look at the mounted disks used by the Data Import Server (DIS) to see how much space is available or perform other actions such as removing old data. You can do this by examining the filesystem on the DIS pod. For example, to see the available space on the disks, run:

What if the DIS isn't running?

If the DIS isn't running (which can happen if the volumes are full), you'll need to create a temporary pod which mounts the same volumes as the DIS. This procedure is only helpful to debug and resolve issues when the DIS is down, because the new pod won't be able to mount the PVCs while the DIS is using them.

First, find the names of the PVCs used by the DIS:

The output looks like this:

Then, create a pod with the same PVCs mounted. You can use the following YAML to create a pod that mounts the same volumes as the DIS, using the names from the previous command.

You'll need the image name currently deployed in the management shell pod. You can find it with a command like this:

Then create a file called my-temp-pod.yaml. You can use the following YAML as a template, but be sure to change the image to the one you found in the previous command. If the disk PVC names are different, you'll also need to update both those sections.

Create the pod:

Now, log on to it with a bash shell and execute commands like df to view the disks, and purge data if needed:

When you're done, delete the pod. Don't forget this or your system won't start correctly due to the mounted PVCs:

How do I tell if I am using the Deephaven or the Bitnami etcd chart?

Run helm list | grep etcd to see etcd release information. If the value in the chart column begins with deephaven-etcd, you are using the Deephaven etcd chart. If it begins with etcd, it is the legacy Bitnami chart. To migrate from a legacy Bitnami chart to the deephaven-etcd chart, see migrating from Bitnami etcd to deephaven-etcd.

How do I check the health of the etcd cluster?

The deephaven-etcd pods are labeled app.kubernetes.io/component=etcd. Retrieve the root password and check cluster health and membership:

The startup and liveness probes use the metrics port (8080, endpoints /readyz and /livez) and don't require credentials. The readiness probe runs etcdctl endpoint health against the local member and authenticates as root with the password from the etcd secret. A pod stuck at 0/1 Running therefore indicates either an etcd startup or clustering problem or an authentication problem, such as a root password that no longer matches the secret. Check the pod logs with kubectl logs ${ETCD_POD}.

How do I find the etcd root password?

The etcd root password is stored in the secret named after the etcd release (key etcd-root-password), and is preserved across helm uninstall/upgrade:

How do I check or trigger etcd backups?

Backups run as a CronJob named <etcd-release>-backup (or <etcd-release>-etcd-backup if the release name does not end in -etcd; fullnameOverride replaces this prefix when set) only when backup.enabled=true. The chart does not create a backup volume: snapshots are written to the existing PVC named in backup.pvc.existingClaim, mounted on each etcd pod at backup.snapshotsDir (default /snapshots).

For full backup and restore procedures, see the Kubernetes etcd backup and recovery guide.

How do I troubleshoot the Deephaven NFS server?

The deephaven-nfs chart deploys an NFS server (Deployment deephaven-nfs-server, pods labeled role=deephaven-nfs-server) behind a ClusterIP service named deephaven-nfs.

If Deephaven pods fail to mount shared storage, verify the Deephaven chart's nfs.server value matches the deephaven-nfs service cluster IP (kubectl get svc deephaven-nfs) and that the NFS pod is Running.

Related documentation

• Kubernetes installation guide
• Kubernetes upgrade guide
• Kubernetes configuration settings
• Kubernetes process launch configuration
• Add custom JARs using a persistent volume
• Kubernetes etcd backup and recovery
• Data Import Server