---
title: Installation troubleshooting
sidebar_label: Installation
---

This guide helps system administrators troubleshoot issues that occur during or immediately after a new Deephaven Enterprise installation. It covers the installation process, post-installation verification, and common failure scenarios.

## Installation log locations

When troubleshooting installation issues, consult these log locations:

| Location                                    | Purpose                                                      |
| ------------------------------------------- | ------------------------------------------------------------ |
| `${DH_LOCAL_DIR}/logs/`                     | Installer logs on the Installation Host                      |
| `${DH_LOCAL_DIR}/logs/$node_hostname/`      | Logs copied from remote hosts to the Installation Host       |
| `/var/log/deephaven/install_configuration/` | Configuration scripts run during installation (on each node) |
| `/var/log/deephaven/previous_install/`      | Logs from the previous installation attempt                  |
| `/var/log/deephaven/<service_name>/`        | Service logs after installation completes                    |

> [!TIP]
> Run the `dh_log_sync.sh` script from the Installation Host to copy all logs from remote machines. This simplifies debugging when installation fails on a remote node.

Log files containing sensitive information are only visible to the `irisadmin` user (or your configured [`DH_ADMIN_USER`](../installation/cluster-config-guide.md#dh_admin_user)).

### Quick log searching

To quickly find errors in installation logs:

```bash
# Find fatal errors in installer logs
grep -r "log_fatal\|FATAL\|ERROR" ${DH_LOCAL_DIR}/logs/

# Find Java-related errors
grep -r "java\|JAVA\|JVM" ${DH_LOCAL_DIR}/logs/ | grep -i "error\|fail\|exception"

# Find permission issues
grep -r "Permission denied\|cannot create\|unable to" ${DH_LOCAL_DIR}/logs/
```

## Pre-installation checklist

Before running the installer, verify these requirements on each cluster node:

### System requirements

1. **Swapping disabled:**

   ```bash
   cat /proc/sys/vm/swappiness
   # Should return 0

   swapon --show
   # Should return no output

   # If any swap devices/files are listed, disable them:
   sudo swapoff -a
   ```

2. **Process limits configured:**

   ```bash
   ulimit -u
   # Should return at least 204800
   ```

3. **Open file limits configured:**

   ```bash
   ulimit -n
   # Should return at least 65535
   ```

4. **Sufficient disk space:**

   ```bash
   df -h /var/log /db /usr/illumon
   # Check that sufficient space is available on each mount
   ```

5. **Network connectivity between nodes:**

   ```bash
   # From each node, verify connectivity to all other nodes
   ping -c 3 <other_node_hostname>
   ```

6. **DNS resolution:**

   ```bash
   # Verify all hostnames resolve correctly
   getent hosts <node_fqdn>
   ```

### SSH and permissions

1. **SSH access from Installation Host:**

   ```bash
   ssh ${DH_SSH_USER}@<target_node> "hostname"
   ```

2. **Sudoers permissions:** Verify that the service user can execute required sudo commands. See [Sudoers configuration](../installation/basic-install.md#appendix-d-sudoers).

## Installation script failures

### `master_install.sh` fails to connect

**Symptoms:**

- SSH connection timeouts.
- "Permission denied" errors.
- "Host key verification failed" errors.

**Resolution:**

1. Verify SSH key configuration:

   ```bash
   ssh -i ${DH_SSH_KEY} ${DH_SSH_USER}@<target_node>
   ```

2. Check that host keys are trusted:

   ```bash
   ssh-keyscan <target_node> >> ~/.ssh/known_hosts
   ```

3. Verify the service user exists on all target nodes:

   ```bash
   ssh ${DH_SSH_USER}@<target_node> "id"
   ```

### `dh_root_prepare.sh` fails

**Symptoms:**

- Errors creating users or groups.
- Permission denied when creating directories.
- Package installation failures.

**Resolution:**

1. Review the log in `/var/log/deephaven/install_configuration/` on the affected node.

2. Verify sudoers configuration allows the required operations:

   ```bash
   sudo -l
   ```

3. Check that required users and groups can be created:

   ```bash
   getent passwd irisadmin
   getent group irisadmin dbmergegrp dbquerygrp
   ```

4. Ensure consistent UIDs/GIDs across all nodes. See [Users and groups](../installation/basic-install.md#appendix-b-users-and-groups).

**Common errors and their causes:**

- **Package installation failure:**

  ```text
  Unable to install $required_package on $(hostname)
  ```

  _Action: Check network connectivity to package repositories; verify proxy settings if applicable._

- **Invalid node configuration:**

  ```text
  Invalid Deephaven Node Role: ${deephaven_node_role}
  ```

  _Action: Verify `cluster.cnf` node definitions are correct for this host._

### `dh_install.sh` fails

**Symptoms:**

- Tar extraction errors.
- "No space left on device" errors.
- Missing Java errors.

**Resolution:**

1. Check disk space:

   ```bash
   df -h /usr/illumon /var/lib/deephaven
   ```

2. Verify Java installation:

   ```bash
   java -version
   # Should match the DH_JAVA_VERSION in cluster.cnf
   ```

3. Check that the installation media was uploaded correctly:

   ```bash
   ls -la ${DH_REMOTE_DIR}/*.tar.gz
   ```

**Common errors and their causes:**

- **Java version mismatch:**

  ```text
  Detected unsupported java version 'X'; install java $DH_JAVA_VERSION or set DH_JAVA=/path/to/correct/java
  ```

  _Action: Run `sudo update-alternatives --config java` or set `DH_JAVA` in `cluster.cnf`._

- **Missing installation media:**

  ```text
  No file $core_plus_archive found; unable to guess DH_COREPLUS_VERSIONS
  ```

  _Action: Verify the tar.gz files exist in `${DH_REMOTE_DIR}` or `/var/lib/deephaven`._

- **Disk space exhausted:**

  ```text
  Not enough disk space available at /usr/illumon to unpack files
  ```

  _Action: Clear at least the amount specified in the error message from the affected mount point._

### `etcd_configure.sh` fails

**Symptoms:**

- etcd cluster fails to form.
- "context deadline exceeded" errors.
- Certificate errors during etcd startup.

**Resolution:**

1. Check etcd service status on each node:

   ```bash
   sudo systemctl status dh-etcd
   sudo journalctl -u dh-etcd --no-pager -n 50
   ```

2. Verify network connectivity between etcd nodes on ports 2379 and 2380:

   ```bash
   nc -zv <etcd_node> 2379
   nc -zv <etcd_node> 2380
   ```

3. Check etcd bootstrap files exist:

   ```bash
   ls -la /etc/sysconfig/illumon.d/etcd/client/
   ```

4. If etcd fails to start due to certificate issues, see [Certificate problems](#certificate-problems).

**Common errors and their causes:**

- **Network timeout:**

  ```text
  retrying of unary invoker failed ... error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
  ```

  _Action: Check firewall rules for ports 2379/2380; verify etcd nodes can reach each other with `nc -zv`._

### `dh_node_finalize.sh` fails

**Symptoms:**

- Errors importing configuration to etcd.
- "etcdserver: mvcc: database space exceeded" errors.
- Partial configuration state.

**Resolution:**

If `dh_node_finalize.sh` does not complete normally, the subsequent `dh_install.sh` script may encounter problems:

1. Disable etcd imports temporarily:

   ```bash
   if [ -d /etc/sysconfig/deephaven/etcd/client ]; then
     sudo -u irisadmin touch /etc/deephaven/SKIP_ETCD_IMPORT
   fi
   ```

2. Re-run the installation. `dh_node_finalize.sh` should now succeed.

3. Re-enable etcd imports:

   ```bash
   sudo -u irisadmin rm /etc/deephaven/SKIP_ETCD_IMPORT
   ```

For etcd space issues, see [Troubleshoot etcd errors](./troubleshooting-etcd.md).

## Post-installation verification

### Verify services are running

On each node, check that all Deephaven services have started:

```bash
/usr/illumon/latest/bin/dh_monit summary
```

All processes should show a green `OK` status. Common statuses indicating problems:

| Status             | Meaning                                               |
| ------------------ | ----------------------------------------------------- |
| `Does not exist`   | Process has not started                               |
| `Initializing`     | Process is starting (may cycle if failing repeatedly) |
| `Execution failed` | Process failed to start                               |
| `Not monitored`    | Process is disabled in monit                          |

> [!TIP]
> For processes that cycle between `Initializing` and `Execution failed`, use interactive debugging to see real-time output. See [Interactive process debugging](#interactive-process-debugging).

If `dh_monit up` times out, check which process has problems:

```bash
/usr/illumon/latest/bin/dh_monit status
```

### Verify etcd cluster health

From a node running the `configuration_server`:

```bash
sudo -u irisadmin /usr/illumon/latest/bin/etcdctl.sh endpoint status -w table
```

All nodes should appear in the table. If any show `context deadline exceeded`, that node is down or unreachable.

### Validate the cluster with `check-deephaven-cluster`

If Python is available, run a comprehensive validation:

```bash
/usr/illumon/latest/bin/check-deephaven-cluster -u <admin_user>
```

This script tests:

- User authentication against all auth servers.
- `WebClientData` query is running.
- Groovy and Python workers can be created on each query server.
- `DbInternal.ProcessEventLog` table is being written and can be read.

> [!NOTE]
> Installations with Envoy enabled use port 8000. For systems without Envoy, use port 8123.

### Verify Web UI access

1. From a browser, connect to `https://<deephaven_host>:8000/` (with Envoy) or `https://<deephaven_host>:8123/` (without Envoy).

2. Log in with administrator credentials.

3. Open the **Query Monitor** panel and verify these Persistent Queries are running:
   - `ImportHelperQuery`
   - `RevertHelperQuery`
   - `WebClientData`

4. Start a new console and run a simple query:

   ```python
   t = db.live_table("DbInternal", "ProcessEventLog").where("Date=today()")
   ```

   ```groovy
   t = db.liveTable("DbInternal", "ProcessEventLog").where("Date=today()")
   ```

## Common installation issues

### Certificate problems

**Symptoms:**

- Services fail with "No subject alternative name matching" errors.
- TLS handshake failures.
- Browser security warnings when accessing the Web UI.

**Example error in logs:**

```text
javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching myhost.example.com found
```

**Resolution:**

1. Verify the Configuration Server host file contains the correct hostname:

   ```bash
   cat /etc/sysconfig/illumon.d/dh-config/clients/single/host
   ```

2. Check that the truststore contains a certificate with a matching SAN:

   ```bash
   openssl storeutl -text -noout /etc/sysconfig/deephaven/trust/truststore-iris.pem | grep -A1 "Subject Alternative Name"
   ```

3. Decode a specific certificate to inspect its SANs:

   ```bash
   cat - | openssl x509 -text -noout
   # Paste the certificate section from the .pem file
   ```

For detailed certificate troubleshooting, see [Troubleshoot certificate errors](./troubleshooting-certificates.md).

### Authentication server connection errors during startup

**Symptoms:**

- Configuration Server logs show authentication failures during startup.
- Services cycle between starting and failing.

**Explanation:**

It is normal to see Authentication Server failures in the Configuration Server log while the Authentication Server is starting up. The Configuration Server must start first, as it is a dependency for the Authentication Server. The Configuration Server attempts to connect to the Authentication Server and retries while waiting for it to become available.

**Resolution:**

Use `dh_monit up --block` to start services in the correct order and wait for all processes to come online:

```bash
/usr/illumon/latest/bin/dh_monit up --block
```

### Services fail to start due to port conflicts

**Symptoms:**

- `java.net.BindException: Address already in use` in logs.
- Services immediately fail after starting.

**Resolution:**

1. Identify what is using the port:

   ```bash
   sudo lsof -i :<port_number>
   # or
   sudo ss -tlnp | grep <port_number>
   ```

2. Stop the conflicting process or reconfigure Deephaven to use a different port.

3. See [Deephaven process ports](../ops-guide/process-ports.md) for the list of ports used by Deephaven services.

### Insufficient system resources

**Symptoms:**

- `java.lang.OutOfMemoryError` errors in logs.
- Services crash intermittently.
- System becomes unresponsive.

**Resolution:**

1. Check available memory:

   ```bash
   free -h
   ```

2. Verify swap is disabled (swapping severely impacts Deephaven performance):

   ```bash
   swapon --show
   # Should return nothing if swap is disabled
   ```

3. Review heap allocations in `/etc/sysconfig/illumon.confs/hostconfig.system` and adjust if necessary in `/etc/sysconfig/illumon.confs/illumon.iris.hostconfig`.

4. Check disk space:

   ```bash
   df -h
   ```

### Process limit errors

**Symptoms:**

- "Cannot allocate memory" errors.
- "Too many open files" errors.
- Workers fail to start.

**Resolution:**

1. Check current limits for the Deephaven user:

   ```bash
   sudo -u irisadmin ulimit -a
   ```

2. Verify `/etc/security/limits.conf` contains the required settings:

   ```
   *          soft    nproc     204800
   *          hard    nproc     204800
   *          soft    nofile     65535
   *          hard    nofile     65535
   ```

3. Log out and log back in (or restart the affected service) for changes to take effect.

### Monit fails to start

**Symptoms:**

- Monit service does not start after reboot.
- `dh_monit` commands fail.

**Resolution:**

After an improper shutdown, a stale PID file may prevent Monit from starting:

1. Remove the stale PID file:

   ```bash
   rm ~irisadmin/.monit.pid
   # or find the home directory dynamically:
   rm $(getent passwd ${DH_ADMIN_USER:-irisadmin} | cut -d: -f6)/.monit.pid
   ```

2. Restart Monit:

   ```bash
   sudo systemctl restart monit
   ```

### etcd cluster issues

**Symptoms:**

- Deephaven services cannot start because they cannot connect to etcd.
- `etcdctl` commands timeout or fail.

**Resolution:**

1. Check etcd service status:

   ```bash
   sudo systemctl status dh-etcd
   ```

2. View etcd logs:

   ```bash
   sudo journalctl -u dh-etcd -n 100
   ```

3. Check etcd cluster health:

   ```bash
   sudo -u irisadmin /usr/illumon/latest/bin/etcdctl.sh endpoint health
   ```

For detailed etcd troubleshooting, see [Troubleshoot etcd errors](./troubleshooting-etcd.md).

## Interactive process debugging

If standard log files do not provide enough information, you can start a process interactively for additional debugging output.

> [!CAUTION]
> This is an advanced troubleshooting step that bypasses M/Monit's automatic management. Only use this when standard log analysis is insufficient.

1. Find the process start command:

   ```bash
   cat /etc/sysconfig/illumon.d/monit/<process_name>.conf
   ```

2. Start the process interactively as the admin user:

   ```bash
   sudo su - irisadmin
   /usr/illumon/latest/bin/iris --debug start <process_name>
   ```

3. Observe the output for errors not captured in standard logs.

## Collecting diagnostic information

### Quick diagnostic checklist

Before contacting support, collect:

- [ ] Output of `dh_monit summary` on all nodes
- [ ] Last 100 lines of failing process log
- [ ] Output of `etcdctl.sh endpoint status -w table`
- [ ] Contents of `/etc/sysconfig/deephaven/cluster.cnf` (redact passwords)
- [ ] Output of `java -version`
- [ ] Output of `df -h` and `free -h`

### Detailed collection steps

When contacting Deephaven support, collect the following information:

1. **Installation logs:**

   ```bash
   /path/to/installer/dh_log_sync.sh
   tar czf install_logs.tar.gz ${DH_LOCAL_DIR}/logs/
   ```

2. **Service status:**

   ```bash
   /usr/illumon/latest/bin/dh_monit summary > monit_summary.txt
   /usr/illumon/latest/bin/dh_monit status >> monit_summary.txt
   ```

3. **etcd cluster status:**

   ```bash
   sudo -u irisadmin /usr/illumon/latest/bin/etcdctl.sh endpoint status -w table > etcd_status.txt
   ```

4. **Recent service logs:**

   ```bash
   tar czf service_logs.tar.gz /var/log/deephaven/*/
   ```

5. **System information:**

   ```bash
   uname -a > system_info.txt
   cat /etc/os-release >> system_info.txt
   java -version 2>> system_info.txt
   free -h >> system_info.txt
   df -h >> system_info.txt
   ```

## Related documentation

- [Basic installation](../installation/basic-install.md)
- [Cluster configuration guide](../installation/cluster-config-guide.md)
- [Process startup troubleshooting](./process-startup-troubleshooting.md)
- [Troubleshoot certificate errors](./troubleshooting-certificates.md)
- [Troubleshoot etcd errors](./troubleshooting-etcd.md)
- [System status overview](./support.md)
- [System logs](./system-logs.md)
- [Deephaven process ports](../ops-guide/process-ports.md)