--- title: Troubleshooting a Podman deployment sidebar_label: Podman --- For standard Podman support questions, refer directly to the [Podman troubleshooting docs](https://github.com/containers/podman/blob/main/troubleshooting.md). The status of Podman pods can be viewed with `podman pod ls`: ``` POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS 765f71d7996e dh-infra-pod Running 2 weeks ago 694adc64e997 2 ``` The status of Podman containers can be viewed with `podman container ls`: ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 694adc64e997 localhost/podman-pause:4.9.4-rhel-1730457905 2 weeks ago Up 2 weeks 10.128.1.118:8000->8000/tcp 765f71d7996e-infra 882b0bea033b localhost/dh-infra:latest 2 weeks ago Up 2 weeks 10.128.1.118:8000->8000/tcp dh-infra ``` Multiple containers can run in a pod. Containers share access to pod resources. The `podman-pause` labeled container is the [infra](https://docs.podman.io/en/stable/markdown/podman-pod-create.1.html#infra) container, which coordinates the shared kernel namespace of the pod. These commands show only running pods or containers. To also see failed or stopped pods or containers, add `-a` to the commands. ### Unable to `podman load` images This is typically a permissions or configuration issue. Ensure that the user running Podman has configured entries in [`/etc/subuid` and `/etc/subgid`](https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md#etcsubuid-and-etcsubgid-configuration). ### Unable to build images Images must be built on an Intel-type system. Generally, `podman build` provides a good indication of what went wrong in its output. Some common causes of build failure are: - No Internet access and no alternate repositories configured for needed DNF packages being used in the build process. - Insufficient disk space because of old images or build cache. To delete images not currently being used by containers/pods and to clear the build cache, run: ```bash podman system prune -a ``` ### `start_command.sh` invocation hangs at `admin_init` There are two known possibilities here - a problem resolving the name of the pod from within the pod, or "extra" instances of etcd left running from a previous pod. #### Name resolution within the pod When processes within the pod attempt to connect to other processes also running within the pod, they need to resolve the pod's internal IP address from the pod's DNS name. The `admin_init.sh` gets the etcd endpoint to connect to from `/etc/sysconfig/deephaven/etcd/client/root/endpoints`. This is a text file that can be viewed with `cat` or `vi`, etc. The endpoint URI will be of the form, `https://mydeephaven.myorg.com:2379`. The `admin_init.sh` script will resolve the IP address of `mydeephaven.myorg.com` from `/etc/hosts`. The resolution process, and the presence of etcd, can be checked from within the pod with: ```bash curl --cacert /etc/sysconfig/deephaven/etcd/client/root/cacert https://mydeephaven.myorg.com:2379 ``` The latter part of the command is the endpoint URI from the `endpoints` file above. It should return `404 page not found`. If it instead returns `curl: (6) Could not resolve host: ...`, then there is a problem with the entry in `/etc/hosts`. The `/etc/hosts` file should contain entries for short and long names of the pod (e.g. `mydeephaven` and `mydeephaven.myorg.com`) with the IP loopback address of localhost (127.0.0.1). Normally this should be sufficient. In some early deployments of Deephaven Podman, the pods internal IP address was used instead of the loopback address. If there is configuration that is expecting this internal address, it can be found in the output of `ifconfig`: ```shell ifconfig lo: flags=73 mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10 loop txqueuelen 1000 (Local Loopback) RX packets 1168377 bytes 671693078 (640.5 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1168377 bytes 671693078 (640.5 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 tap0: flags=67 mtu 65520 inet 10.0.2.100 netmask 255.255.255.0 broadcast 10.0.2.255 inet6 fe80::f0c6:f1ff:fe13:5487 prefixlen 64 scopeid 0x20 inet6 fd00::f0c6:f1ff:fe13:5487 prefixlen 64 scopeid 0x0 ether f2:c6:f1:13:54:87 txqueuelen 1000 (Ethernet) RX packets 27564 bytes 4297036 (4.0 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 27408 bytes 2125868 (2.0 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ``` `10.0.2.100` is typical for Podman on Linux. Podman on other platforms may be different. #### Orphaned etcd processes In some cases, Podman may stop a pod without stopping all the processes used by the pod. The cause of this is not currently known. These "orphaned" processes can cause problems when launching a Deephaven Podman pod. If `start_command.sh` is executed for an infra pod on a host that has leftover etcd processes running, `admin_init` will hang while attempting to manage the old etcd process. To diagnose this situation, stop the pod: ```bash podman pod stop --time 90 dh-infra ``` Then, check running processes on the host with: ```bash ps -ef | grep "etcd " ``` If no infra pods are running on the host, and no other etcd instances are running on the host, this should return only one line for `grep` itself. If other process entries are returned, these orphaned processes must be `kill`ed before `start_command.sh` is tried again. ```shell # example of leftover etcd process entries deephav+ 3943722 3931540 0 Jan15 ? 00:00:00 sudo -i -u etcd -g irisadmin /bin/bash -c echo "etcd: Starting etcd (with config file /etc/etcd/dh/latest/config.yaml, logging to /var/log/dh-etcd.log)..."; GOMAXPROCS=16 /usr/bin/etcd --config-file '/etc/etcd/dh/latest/config.yaml' >/var/log/dh-etcd.log 2>&1; ETCD_EXIT_STATUS=$?; echo "etcd: etcd exited with status $ETCD_EXIT_STATUS! etcd bash shell exiting."; exit $ETCD_EXIT_STATUS; 3713482 3943727 3943722 0 Jan15 ? 00:00:00 /bin/bash -c echo "etcd: Starting etcd (with config file /etc/etcd/dh/latest/config.yaml, logging to /var/log/dh-etcd.log)..."; GOMAXPROCS=16 /usr/bin/etcd --config-file '/etc/etcd/dh/latest/config.yaml' >/var/log/dh-etcd.log 2>&1; ETCD_EXIT_STATUS=$?; echo "etcd: etcd exited with status ! etcd bash shell exiting."; exit ; 3713482 3943757 3943727 0 Jan15 ? 00:02:45 /usr/bin/etcd --config-file /etc/etcd/dh/latest/config.yaml ``` > [!IMPORTANT] > Before killing any suspected orphaned etcd processes, double-check that they are not related to other instances of etcd or Deephaven Podman infra pods running correctly on the host. One item to verify is the ownership of the parent process - `deephav...` in the example above. Also, if there are multiple processes, verify that they are related. In the example above, the first is the parent of the second, and the second is the parent of the third. ### Pauses for a relatively long time (60 seconds plus) at two points during initialization This can be seen when viewing the initialization logs interactively with: ```bash podman logs -f dh-infra ``` as relatively long pauses where nothing happens. Once the script continues after the first long pause, it will generally show a message like: `root_prepare execution time: 208.554s`, where `208.554s` is some amount of time in excess of 60 seconds. These pauses are caused by large numbers of old log files in a mounted log volume. Deleting no-longer needed logs from the log volume will reduce or eliminate the pause. ### System not usable after ungraceful shutdown and restart with `podman pod start` This has been seen intermittently when the host or the podman process has been terminated ungracefully/abruptly. If the system is not usable from the Web UI after such a restart scenario: - Log into each of the containers with: ```bash podman exec -it bash ``` where container name is `dh-infra`, `dh-query`, etc. - In each node, run: ```bash /usr/illumon/latest/bin/dh_monit reload ``` to reinitialze monit itself, and run: ```bash /usr/illumon/latest/bin/dh_monit restart all ``` to ensure that all Deephaven processes are restarted. ### Avoid machine start on Linux On Linux, a separately-spawned VM is not required. If Podman is installed, simply loading an image and starting a system is sufficient. For anyone used to using Podman on MacOS, doing the following cycle may have become second nature. These commands are not required on Linux. ```shell podman machine init podman machine start ``` If run unintentionally on Linux, the following error will be seen: ``` Error: exec: "qemu-system-x86_64": executable file not found in $PATH ``` ### `start_command.sh` invocation does not result in a usable pod To tail the log file of a running (or most recently running) infra container, use: ```bash podman logs -f dh-infra ``` Or for a query container: ```bash podman logs -f dh-query ``` In most cases, the infra container is the more likely place to find causes of problems because the bulk of initial cluster configuration is done through the entry point of the infra container. The `dh-init` process that initializes/installs/upgrades a Deephaven Podman deployment takes a few minutes to run on the Deephaven infra node. Viewing the logs during this time will show it doing things like installing and configuring Python, configuring etcd (`admin-init.sh`), and verifying started status of key services like the `configuration_server` and `authentication_server`. If an error occurs relatively early in the `dh_init` process, the pod will go immediately to a failed state at that point and will not be accessible. The types of issues that cause this kind of failure are usually related to something configured on the host, rather than inside the pod, and will need to be corrected before trying again to start the pod with `start_command.sh`. In rare cases, an early failure may also block `podman logs` from being able to access the logs from the pod. If this is the case, you can create a small batch file to run the start command and get the logs, something like: ```shell ./start_command ... sleep 1 podman logs -f dh-infra ``` Most failures that occur later, including those that cause services to fail on startup, are caught by the `dh_init` process, and enter a 10-minute (600-second) wait period before shutting down the pod. This is to allow a user to shell into the pod and inspect log files and configuration to find the cause of the failure. If a pod has already exceeded its 10 minutes and has been shut down, it can be restarted with `podman pod start `. For an infra pod, the default pod name is `dh_infra_pod`. A container that is running can be accessed using: ``` podman exec -it bash ``` For example: ```bash podman exec -it dh-infra bash ``` Some common failure modes in `dh_init` include: - Fails with "Cluster initialization already started but not complete!" in logs. This is usually caused by not deleting the running configuration data. Ensure `sudo rm -vrf "${VOLUME_BASE_DIR:?}"/deephaven-shared-config/{auth,dh-config,etcd,installation,trust,cluster_status}` has been run. It needs to be run every time before running `start_command.sh` for a cluster that has already been started once. - Fails to complete init, and ends up in 10-minute debug window. Run the following to get into the shell: ``` podman exec -it dh-infra bash ``` Then checking with [`dh_monit`](../ops-guide/startup-shutdown.md#dh_monit) shows problems with the `configuration_server` and a failed `authentication_server`. The `authentication_server` current log (`cat /var/log/deephaven/authentication_server/AuthenticationServer.log.current`) shows: ``` Could not start authentication server: java.lang.RuntimeException: No trust store found at "/deephaven-tls/truststore.p12" ``` This is usually caused by not having the correct issuing certificate in `"${VOLUME_BASE_DIR:?}"/deephaven-tls/ca.crt`. - Fails with `access denied` or `File exists` accessing some file or directory during `dh_init`. This is usually caused by not completely deleting running configuration, or by not applying SELinux `chcon` settings. Ensure `sudo rm -vrf "${VOLUME_BASE_DIR:?}"/deephaven-shared-config/{auth,dh-config,etcd,installation,trust,cluster_status}` and `chcon -vR -u system_u -r object_r -t container_file_t "${VOLUME_BASE_DIR:?}"` have been run before `start_command.sh` is called. - `dh_init` does not complete successfully. Checking during the 10-minute debug window shows most services say "Start Pending" in [`dh_monit summary`](../ops-guide/startup-shutdown.md#dh_monit). The configuration_server process starts and stops in a loop. This can be caused by attaching a [`java_lib volume`](../installation/podman-installation.md#start_commandsh-details) with contents that are not readable to Deephaven processes in the container due to incorrect permissions. - Fails with a message stating `find: 'UNKNOWN' is not the name of a known user`. This is caused by `dh_root_prepare` being unable to recognize the owner or group of a file or directory being provided from outside the container. Typically this is because one of the volume paths includes files or directories not owned by the user running Podman. This can be corrected with: `sudo chown -R : $VOLUME_BASE_DIR`. Note that this usually requires sudo rights (sudo prefix to the command line). In some cases, depending on how and where the directories were created, you may be able to correct ownership without using `sudo`. If you do not have `sudo` rights, try without `sudo`, and see if that helps. If not, contact an administrator to update the permissions for you. [General Deephaven troubleshooting procedures](./support.md) can also help to isolate Deephaven Podman initialization issues. ## Troubleshooting SELinux in Podman When running Podman on Linux with SELinux enabled, you may encounter "Permission Denied" errors when accessing files in mounted volumes. This is often caused by SELinux Multi-Category Security (MCS) labels. ### Understanding MCS labels SELinux assigns MCS labels to containers to prevent unauthorized volume sharing between containers. When a volume is mounted with the `:z` option (which Deephaven uses by default), SELinux applies a category label like `c43,c512` to files in that volume. Only containers with matching labels can access those files. The problem occurs when you delete and recreate a pod: the new pod gets a different MCS category (e.g., `c123,c456`), but files created by the old pod still have the old category. This causes "Permission Denied" errors. ### Copied files retain the source SELinux type If you copy files into a volume using `cp -a`, or copy them from a home directory, the original SELinux type (commonly `user_home_t`) can be preserved. A file with `user_home_t` is not readable inside a container, even if the surrounding directory is `container_file_t`. `chcon -R` on a symlinked directory may also not relabel the actual file it points to. To avoid this: - Use `cp` without `-a` when copying into a volume, so the file inherits the target directory's `container_file_t` type. - If a file already has the wrong type, relabel it explicitly: ```bash chcon -v -u system_u -r object_r -t container_file_t -l s0 /path/to/file ``` ### Diagnosing SELinux issues To check the SELinux context of files: ```bash # View SELinux context for files in a directory ls -lZ ${VOLUME_BASE_DIR}/deephaven-shared-config/ # View detailed context for a specific file stat -c '%n %C' ${VOLUME_BASE_DIR}/deephaven-shared-config/cluster_status ``` Example output showing a container MCS label: ``` cluster_status system_u:object_r:container_file_t:s0:c43,c512 ``` The `c43,c512` portion is the MCS category. If this doesn't match the current container's category, access is denied. ### Fixing MCS label conflicts The recommended approach resets the MCS level to `s0` (no categories) on the affected directories: ```bash chcon -vR -u system_u -r object_r -t container_file_t -l s0 "${VOLUME_BASE_DIR:?}"/deephaven-shared-config/ chcon -vR -u system_u -r object_r -t container_file_t -l s0 "${VOLUME_BASE_DIR:?}"/deephaven-etcd/{etc-etcd,var-lib-etcd} ``` > [!NOTE] > There is no way to selectively remove only container-assigned MCS categories — SELinux does not distinguish between categories assigned by containers and those applied manually. The `-l s0` option resets the MCS level to `s0`, clearing all categories. If you have intentionally applied custom MCS labels, you will need to re-apply them after running this command. After clearing the labels, you can restart the pod with `start_command.sh`. ### When MCS label issues occur MCS label conflicts typically happen when: - Deleting and recreating a pod (the new pod gets a different MCS category) - Reusing a test machine for multiple deployments - Upgrading Deephaven by deleting the old pod and creating a new one For upgrade scenarios, see [Restart and upgrade](../installation/podman-upgrade.md#restart-and-upgrade) for the complete procedure. ### Preventing MCS label issues The `DISABLE_SELINUX_*` environment variables can be used to skip the `:z` mount option for specific volumes. When `:z` is disabled for a volume, Podman does not apply MCS labels, which avoids the conflict on pod recreation — but also means SELinux won't restrict which containers can access that volume. See [start_command.sh details](../installation/podman-installation.md#start_commandsh-details) for available options. ## Related documentation - [Podman Quickstart](../../quickstarts/podman-quickstart.md) - [Restart and update a Podman deployment](../installation/podman-upgrade.md) - [Podman Installation](../installation/podman-installation.md) - [Kubernetes](../kubernetes/kubernetes-install-guide.md)