Troubleshooting a Podman deployment
For standard Podman support questions, refer directly to the Podman troubleshooting docs.
The status of Podman pods can be viewed with podman pod ls:
The status of Podman containers can be viewed with podman container ls:
Multiple containers can run in a pod. Containers share access to pod resources. The podman-pause labeled container is the infra container, which coordinates the shared kernel namespace of the pod.
podman pod ls lists every pod, including stopped ones. podman container ls shows only running containers; add -a to also see failed or stopped containers.
Common problems
Unable to podman load images
This is typically a permissions or configuration issue. Ensure that the user running Podman has configured entries in /etc/subuid and /etc/subgid.
Unable to build images
Images must be built on an Intel-type system. Generally, podman build provides a good indication of what went wrong in its output.
Some common causes of build failure are:
- No Internet access and no alternate repositories configured for needed DNF packages being used in the build process.
- Insufficient disk space because of old images or build cache. To delete images not currently being used by containers/pods and to clear the build cache, run:
start_command.sh invocation hangs at admin_init
There are two known possibilities here: a problem resolving the name of the pod from within the pod, or "extra" instances of etcd left running from a previous pod.
Name resolution within the pod
When processes within the pod connect to other processes in the same pod, they resolve the pod's hostname. The admin_init.sh script gets the etcd endpoint to connect to from /etc/sysconfig/deephaven/etcd/client/root/endpoints. This is a text file you can open with cat or vi. The endpoint URI has the form https://mydeephaven.myorg.com:2379. The admin_init.sh script resolves the IP address of mydeephaven.myorg.com from /etc/hosts.
The resolution process, and the presence of etcd, can be checked from within the pod with:
The latter part of the command is the endpoint URI from the endpoints file above. It should return 404 page not found. If it instead returns curl: (6) Could not resolve host: ..., then there is a problem with the entry in /etc/hosts.
The /etc/hosts file should map the pod hostname passed to start_command.sh with -h (e.g. mydeephaven.myorg.com) to the loopback address, 127.0.0.1. start_command.sh adds this entry when it creates the pod.
Pods created by early Deephaven Podman releases with the -i option mapped the hostname to the pod's internal IP address instead (typically 10.0.2.100 on Linux). -i now has no effect. If configuration you added refers to that internal address, replace it with the pod hostname.
Orphaned etcd processes
In some cases, Podman may stop a pod without stopping all the processes used by the pod. The cause of this is not currently known. These "orphaned" processes can cause problems when launching a Deephaven Podman pod.
If start_command.sh is executed for an infra pod on a host that has leftover etcd processes running, admin_init will hang while attempting to manage the old etcd process.
To diagnose this situation, stop the pod:
Then, check running processes on the host with:
If no infra pods are running on the host, and no other etcd instances are running on the host, this should return only one line for grep itself. If other process entries are returned, these orphaned processes must be killed before start_command.sh is tried again.
Important
Before killing any suspected orphaned etcd processes, double-check that they are not related to other instances of etcd or Deephaven Podman infra pods running correctly on the host. One item to verify is the ownership of the parent process (deephav... in the example above). Also, if there are multiple processes, verify that they are related. In the example above, the first is the parent of the second, and the second is the parent of the third.
Pauses for a relatively long time (60 seconds plus) at two points during initialization
This can be seen when viewing the initialization logs interactively with:
as relatively long pauses where nothing happens.
Once the script continues after the first long pause, it will generally show a message like: root_prepare execution time: 208.554s, where 208.554s is some amount of time in excess of 60 seconds. These pauses are caused by large numbers of old log files in a mounted log volume. Deleting no-longer needed logs from the log volume will reduce or eliminate the pause.
System not usable after ungraceful shutdown and restart with podman pod start
This has been seen intermittently when the host or the podman process has been terminated ungracefully/abruptly.
If the system is not usable from the Web UI after such a restart scenario:
-
Log into each of the containers with:
where container name is
dh-infra,dh-query, etc. -
In each node, run:
to reinitialize monit itself, and run:
to ensure that all Deephaven processes are restarted.
Avoid machine start on Linux
On Linux, a separately-spawned VM is not required. If Podman is installed, simply loading an image and starting a system is sufficient.
For anyone used to using Podman on MacOS, doing the following cycle may have become second nature. These commands are not required on Linux.
If run unintentionally on Linux, the following error appears:
start_command.sh invocation does not result in a usable pod
To tail the log file of a running (or most recently running) infra container, use:
Or for a query container:
The infra container is usually the place to look, because it does most of the cluster configuration.
dh-init.sh, which initializes, installs, or upgrades a Deephaven Podman deployment, takes a few minutes to run on the Deephaven infra node. Viewing the logs during this time shows it doing things like installing and configuring Python, configuring etcd (admin_init.sh), and verifying started status of key services like the configuration_server and authentication_server.
podman logs also works on a container that has stopped. How long a failed infra container stays up depends on where it failed:
- If the installation step (the preinstall hook) fails, the container waits 10 minutes before it exits, so you can open a shell in it and inspect log files and configuration.
- If Deephaven services fail to start after installation, the container prints up to 250 lines from each failed process's logs and exits.
- Other errors, such as a problem with something configured on the host, stop the container immediately. Correct the cause before you retry. In rare cases, an early failure may also keep
podman logsfrom showing the logs; if so, runpodman logs -f dh-infraimmediately afterstart_command.sh.
Restarting a failed container with podman pod start does not resume the installation: the container stops again with Container initialization already started but not complete!.
Keep a failed container for diagnosis
To keep an infra container running after its preinstall hook (the installation step) fails, until you stop it, prepare the retry as described in Retry a failed installation, and add --keep-failed-container to the retry command it gives. Failures after the installation, such as services that do not start, still stop the container. The deployment stays failed: fixing the problem inside the container does not resume installation. Open a shell in a running container with:
Stop the container with podman stop dh-infra when you are done.
Retry a failed installation
--delete-pod alone does not retry a failed installation: the new container stops with Cluster initialization already started but not complete! because the cluster status in the shared configuration volume is still INITIALIZING. After fixing the cause, retry as follows:
-
If
dh_etcd_config.tgzexists in the shared configuration volume, rerun the deployment command with--upgrade. -
Otherwise, an installation that has never completed failed before etcd was configured, so the volumes hold nothing to keep. Remove the pod, empty the shared configuration and etcd volumes, and rerun the original deployment command. Never do this on a deployment that has completed: it erases its etcd database.
Common failure modes
-
Fails with
Cluster initialization already started but not complete!in the logs. An earlier initialization failed; follow Retry a failed installation. -
The
authentication_serverfails withPKIX path building failed: unable to find valid certification path to requested target, and the cluster stays atINITIALIZING. The CA that signed your certificates was not passed with-c/--ca-cert-file. Retry the installation with-cadded to the command. See Supplying trusted certificates to Podman. -
Fails with
access deniedorFile existsaccessing some file or directory during initialization. This is usually caused by leftover configuration from an earlier failed initialization (see Retry a failed installation) or by missing SELinux labels. If SELinux is in use, relabel the volumes beforestart_command.shwithpodman unshare chcon -vRH --user=system_u --role=object_r --type=container_file_t "${VOLUME_BASE_DIR:?}"(rootless Podman) or the samechconcommand withsudo(rootful Podman). -
Initialization does not complete, and while the container is still running, most services show "Start Pending" in
dh_monit summary. Theconfiguration_serverprocess starts and stops in a loop. This can be caused by attaching ajava_libvolume whose contents Deephaven processes in the container cannot read because of incorrect permissions. -
Fails with a message stating
find: 'UNKNOWN' is not the name of a known user. This is caused bydh_root_preparebeing unable to recognize the owner or group of a file or directory in a volume, typically a directory under the logs volume (-L) that was created or copied in from outside the container. Fix only the files it reports, then rerun the deployment command with--upgrade:For images built without
--use-non-root-accounts,<owner>isroot:root. For images built with it, use the account of the service that owns the directory:irisadmin:irisadminfor most log directories,dbmerge:dbmergefordisandmerge_server, anddbquery:dbqueryforquery_serverandltds. Runningchowninside the image resolves those names to the image's own accounts.Warning
Never change the owner of the whole
VOLUME_BASE_DIRin a deployment whose images were built with--use-non-root-accounts. Its files belong to several container accounts, such asetcd,irisadmin,dbmergeanddbquery; giving them all one owner leaves services unable to read their own files, for example the etcd private keys.
General Deephaven troubleshooting procedures can also help to isolate Deephaven Podman initialization issues.
Recovering etcd after a quota alarm
When the etcd database reaches its quota, etcd raises a NOSPACE alarm and rejects writes, so Deephaven services that write to etcd fail. During initialization, admin_init reports the alarm in its etcd diagnostics, and initialization fails.
Run dh_etcd_recover_nospace.sh from the extracted deephaven-podman-<version> directory on the infra host, as the user that owns the pod, while the infra container is running. If initialization failed, run it within the 10-minute window or use --keep-failed-container. Use a maintenance window: defragmentation blocks etcd requests while it runs.
The script changes nothing unless etcd reports a NOSPACE alarm. It then asks for confirmation, saves a snapshot of the etcd database to the current directory (readable only by you), compacts away old revisions while keeping current key values, defragments the database, disarms the alarm, and checks that no alarm remains. The snapshot contains sensitive deployment data; store it securely. Options:
-n <container name>: the infra container, if it is not nameddh-infra.--snapshot-dir <dir>: where to save the snapshot.--yes: skip the confirmation prompt.
If the alarm persists, or returns soon after recovery, the current data exceeds the quota. Increase quota-backend-bytes through etcd overrides, then run the script again. Do not delete the etcd data directory. See etcd maintenance for background.
To inspect etcd directly, dh_etcdctl.sh runs etcdctl inside the infra container as the container's administrator account, with the existing etcd root credentials. Add -n <container name> before the etcdctl arguments if your infra container is not named dh-infra.
Troubleshooting SELinux in Podman
When running Podman on Linux with SELinux enabled, you may encounter "Permission Denied" errors when accessing files in mounted volumes. This is often caused by SELinux Multi-Category Security (MCS) labels.
Understanding MCS labels
SELinux assigns each container a set of MCS categories, such as c43,c512, to keep containers apart. Deephaven mounts volumes with the :z option, which relabels them so containers can use them. Files that a container's processes create in a volume carry that container's categories, and only a container with matching categories can access them.
The problem occurs when you delete and recreate a pod: the new pod gets different categories (e.g., c123,c456), but files created by the old pod still have the old ones. This causes "Permission Denied" errors.
Copied files retain the source SELinux type
If you copy files into a volume using cp -a, or copy them from a home directory, the original SELinux type (commonly user_home_t) can be preserved. A file with user_home_t is not readable inside a container, even if the surrounding directory is container_file_t.
To avoid this:
-
Use
cpwithout-awhen copying into a volume, so the file inherits the target directory'scontainer_file_ttype. -
If a file already has the wrong type, relabel it explicitly:
chcon -R does not follow a symbolic link named on its command line: it relabels only the link and still reports success. The chcon commands in this guide pass -H, which follows a symbolic link named on the command line; keep it if you adapt them.
Diagnosing SELinux issues
To check the SELinux context of files:
Example output showing a container MCS label:
The c43,c512 portion is the MCS category. If this doesn't match the current container's category, access is denied.
Fixing MCS label conflicts
The recommended approach resets the MCS level to s0 (no categories) on the affected directories:
Note
There is no way to selectively remove only container-assigned MCS categories: SELinux does not distinguish between categories assigned by containers and those applied manually. The -l s0 option resets the MCS level to s0, clearing all categories. If you have intentionally applied custom MCS labels, you must re-apply them after running this command.
After clearing the labels, recreate the pod with start_command.sh.
start_command.sh does this itself whenever it recreates the infra pod, with --upgrade or --delete-pod: it runs chcon ... --range=s0 on the shared configuration and etcd volumes. If the relabeling fails (for example, because it needs sudo), it prints a warning; run the commands above yourself.
When MCS label issues occur
MCS label conflicts typically happen when:
- Deleting and recreating a pod (the new pod gets a different MCS category)
- Reusing a test machine for multiple deployments
- Upgrading Deephaven by deleting the old pod and creating a new one
See Upgrade with --upgrade for how an upgrade handles these labels.
Preventing MCS label issues
The DISABLE_SELINUX_* environment variables skip the :z mount option for specific volumes, for storage that does not support SELinux relabeling. Podman then does not relabel that volume, so its files must already carry a label that containers can use. See start_command.sh details for the variables.