Process startup troubleshooting

All Deephaven processes create log files. If a process does not start correctly, these log files might show the cause.

Log files will be written to /var/log/deephaven/<process_name> or /var/log/deephaven/misc. Processes without write permission to those folders will write logs to /tmp.

All processes create a log file named <MainJavaClass>.log.<datetime> or <proc_name>.log.<datetime>. These log files will periodically roll over, typically every 15 minutes. For convenience, look for a symbolic link to the most current file, <prefix>.log.current.

The process of starting a Deephaven service creates a launch log file named <proc_name>.log.<date>. This file contains the logging that preceeds the start of the Java process. If the Java process fails to start, this will be the only log file. This file contains the actual command line and all options. All launch activity for a date is logged to the same file.

Some of the command line utilities support a --verbose argument for more verbose logging.

Deephaven services

When Deephaven services, managed by M/Monit, processes fail to start, they usually show status in /usr/illumon/latest/bin/dh_monit summary that cycles between Does not exist, Initializing, and Execution failed. When investigating the cause of startup failure, start with the process(es) with the most dependencies (see Process dependencies). For example, if the configuration_server does not start, nothing else can start.

For the particular service being investigated:

  • Start with the process's launch log: /var/log/deephaven/<process_name>/<process_name>.log.<date>
  • If the process starts (as indicated by the launch log), but then dies or restarts, check the process's detailed log: /var/log/deephaven/<process_name>/<class or process_name>.log.current

In the rare case where a process fails to launch and there is little or no information in the launch log, it may be possible to learn more by interactively launching the process.

Process launch commands are stored in .conf files in /etc/sysconfig/illumon.d/monit. For example:

cat /etc/sysconfig/illumon.d/monit/iris_controller.conf

check process iris_controller with pidfile /etc/deephaven/run/iris_controller.pid
    start program = "/usr/illumon/latest/bin/iris start iris_controller"
    stop program  = "/usr/illumon/latest/bin/iris stop iris_controller"

Running "as irisadmin", with sudo su - irisadmin, the iris_controller can be interactively started with:

 /usr/illumon/latest/bin/iris --debug start iris_controller

This will echo to stdout all the details as the iris script attempts to launch the iris_controller process.