Installation troubleshooting

This guide helps system administrators troubleshoot issues that occur during or immediately after a new Deephaven Enterprise installation. It covers the installation process, post-installation verification, and common failure scenarios.

Installation log locations

When troubleshooting installation issues, consult these log locations:

Log files containing sensitive information are only visible to the irisadmin user (or your configured DH_ADMIN_USER).

Quick log searching

To quickly find errors in installation logs:

Pre-installation checklist

Before running the installer, verify these requirements on each cluster node:

System requirements

1. Swapping disabled:

2. Process limits configured:

3. Open file limits configured:

4. Sufficient disk space:

5. Network connectivity between nodes:

6. DNS resolution:

SSH and permissions

1. SSH access from Installation Host:

2. Sudoers permissions: Verify that the service user can execute required sudo commands. See Sudoers configuration.

Installation script failures

master_install.sh fails to connect

Symptoms:

• SSH connection timeouts.
• "Permission denied" errors.
• "Host key verification failed" errors.

Resolution:

1. Verify SSH key configuration:

2. Check that host keys are trusted:

3. Verify the service user exists on all target nodes:

dh_root_prepare.sh fails

Symptoms:

• Errors creating users or groups.
• Permission denied when creating directories.
• Package installation failures.

Resolution:

1. Review the log in /var/log/deephaven/install_configuration/ on the affected node.

2. Verify sudoers configuration allows the required operations:

3. Check that required users and groups can be created:

4. Ensure consistent UIDs/GIDs across all nodes. See Users and groups.

Common errors and their causes:

• Package installation failure:

  Action: Check network connectivity to package repositories; verify proxy settings if applicable.

• Invalid node configuration:

  Action: Verify cluster.cnf node definitions are correct for this host.

dh_install.sh fails

Symptoms:

• Tar extraction errors.
• "No space left on device" errors.
• Missing Java errors.

Resolution:

1. Check disk space:

2. Verify Java installation:

3. Check that the installation media was uploaded correctly:

Common errors and their causes:

• Java version mismatch:

  Action: Run sudo update-alternatives --config java or set DH_JAVA in cluster.cnf.

• Missing installation media:

  Action: Verify the tar.gz files exist in ${DH_REMOTE_DIR} or /var/lib/deephaven.

• Disk space exhausted:

  Action: Clear at least the amount specified in the error message from the affected mount point.

etcd_configure.sh fails

Symptoms:

• etcd cluster fails to form.
• "context deadline exceeded" errors.
• Certificate errors during etcd startup.

Resolution:

1. Check etcd service status on each node:

2. Verify network connectivity between etcd nodes on ports 2379 and 2380:

3. Check etcd bootstrap files exist:

4. If etcd fails to start due to certificate issues, see Certificate problems.

Common errors and their causes:

• Network timeout:

  Action: Check firewall rules for ports 2379/2380; verify etcd nodes can reach each other with nc -zv.

dh_node_finalize.sh fails

Symptoms:

• Errors importing configuration to etcd.
• "etcdserver: mvcc: database space exceeded" errors.
• Partial configuration state.

Resolution:

If dh_node_finalize.sh does not complete normally, the subsequent dh_install.sh script may encounter problems:

1. Disable etcd imports temporarily:

2. Re-run the installation. dh_node_finalize.sh should now succeed.

3. Re-enable etcd imports:

For etcd space issues, see Troubleshoot etcd errors.

Post-installation verification

Verify services are running

On each node, check that all Deephaven services have started:

All processes should show a green OK status. Common statuses indicating problems:

If dh_monit up times out, check which process has problems:

Verify etcd cluster health

From a node running the configuration_server:

All nodes should appear in the table. If any show context deadline exceeded, that node is down or unreachable.

Validate the cluster with check-deephaven-cluster

If Python is available, run a comprehensive validation:

This script tests:

• User authentication against all auth servers.
• WebClientData query is running.
• Groovy and Python workers can be created on each query server.
• DbInternal.ProcessEventLog table is being written and can be read.

Verify Web UI access

1. From a browser, connect to https://<deephaven_host>:8000/ (with Envoy) or https://<deephaven_host>:8123/ (without Envoy).

2. Log in with administrator credentials.

3. Open the Query Monitor panel and verify these Persistent Queries are running:

   • ImportHelperQuery
   • RevertHelperQuery
   • WebClientData

4. Start a new console and run a simple query:

   GroovyPython

Common installation issues

Certificate problems

Symptoms:

• Services fail with "No subject alternative name matching" errors.
• TLS handshake failures.
• Browser security warnings when accessing the Web UI.

Example error in logs:

Resolution:

1. Verify the Configuration Server host file contains the correct hostname:

2. Check that the truststore contains a certificate with a matching SAN:

3. Decode a specific certificate to inspect its SANs:

For detailed certificate troubleshooting, see Troubleshoot certificate errors.

Authentication server connection errors during startup

Symptoms:

• Configuration Server logs show authentication failures during startup.
• Services cycle between starting and failing.

Explanation:

It is normal to see Authentication Server failures in the Configuration Server log while the Authentication Server is starting up. The Configuration Server must start first, as it is a dependency for the Authentication Server. The Configuration Server attempts to connect to the Authentication Server and retries while waiting for it to become available.

Resolution:

Use dh_monit up --block to start services in the correct order and wait for all processes to come online:

Services fail to start due to port conflicts

Symptoms:

• java.net.BindException: Address already in use in logs.
• Services immediately fail after starting.

Resolution:

1. Identify what is using the port:

2. Stop the conflicting process or reconfigure Deephaven to use a different port.

3. See Deephaven process ports for the list of ports used by Deephaven services.

Insufficient system resources

Symptoms:

• java.lang.OutOfMemoryError errors in logs.
• Services crash intermittently.
• System becomes unresponsive.

Resolution:

1. Check available memory:

2. Verify swap is disabled (swapping severely impacts Deephaven performance):

3. Review heap allocations in /etc/sysconfig/illumon.confs/hostconfig.system and adjust if necessary in /etc/sysconfig/illumon.confs/illumon.iris.hostconfig.

4. Check disk space:

Process limit errors

Symptoms:

• "Cannot allocate memory" errors.
• "Too many open files" errors.
• Workers fail to start.

Resolution:

1. Check current limits for the Deephaven user:

2. Verify /etc/security/limits.conf contains the required settings:

3. Log out and log back in (or restart the affected service) for changes to take effect.

Monit fails to start

Symptoms:

• Monit service does not start after reboot.
• dh_monit commands fail.

Resolution:

After an improper shutdown, a stale PID file may prevent Monit from starting:

1. Remove the stale PID file:

2. Restart Monit:

etcd cluster issues

Symptoms:

• Deephaven services cannot start because they cannot connect to etcd.
• etcdctl commands timeout or fail.

Resolution:

1. Check etcd service status:

2. View etcd logs:

3. Check etcd cluster health:

For detailed etcd troubleshooting, see Troubleshoot etcd errors.

Interactive process debugging

If standard log files do not provide enough information, you can start a process interactively for additional debugging output.

1. Find the process start command:

2. Start the process interactively as the admin user:

3. Observe the output for errors not captured in standard logs.

Collecting diagnostic information

Quick diagnostic checklist

Before contacting support, collect:

• Output of dh_monit summary on all nodes
• Last 100 lines of failing process log
• Output of etcdctl.sh endpoint status -w table
• Contents of /etc/sysconfig/deephaven/cluster.cnf (redact passwords)
• Output of java -version
• Output of df -h and free -h

Detailed collection steps

When contacting Deephaven support, collect the following information:

1. Installation logs:

2. Service status:

3. etcd cluster status:

4. Recent service logs:

5. System information:

Related documentation

• Basic installation
• Cluster configuration guide
• Process startup troubleshooting
• Troubleshoot certificate errors
• Troubleshoot etcd errors
• System status overview
• System logs
• Deephaven process ports