Log files overview

Some Deephaven processes record events, errors, and other application messages in their process log files. These are plain text files that can be viewed or searched with grep, tail, less, and other text processing utilities.

Log search term definitions

General

  • FATAL - Any trace logged at log level FATAL should indicate that the process is beginning to shut down due to an error.
  • ERROR - Generally, the rate of ERROR level trace may be interesting, but many traces logged at this level are not critical.

Lifecycle events

  • <process name> INITIALIZING - The process has begun its initialization.
  • <process name> RUNNING - Initialization has completed, and the process is available.
  • <process name> ALIVE - The process continues to be available, logged on a configurable cycle.
  • <process name> SHUTTING_DOWN - The process is begging to shut down.

Process specific

For db_dis:

  • Rejecting and closing channel - This happen rarely, if at all. This message generally points to a configuration issue to which DBAs should be alerted.
  • DataImportStreamProcessor-.+WARN(regex) - If a high rate of warning trace messages are seen from this component (other than during tailer restarts), it may point to a system issue.

Startup script log files

These capture the stdout and stderr streams from Deephaven production processes.

This file appears in the process’s log directory, with name format process_name.log.yyyy-MM-dd, where the date is the process startup date.

This file is not rotated. Be careful with automated log cleanup for long-running processes in order to ensure that this file is not deleted while still in use.

Query Worker log samples

Deephaven workers do not produce the files described in this section. Worker logs are sent to the Process Event Log table directly or via the RemoteQueryDispatcher, depending on the type of log event.

This has implications for monitoring system processes running as workers. Such monitoring must either be done by querying the Process Event Log table, or by consuming binary logs that are written by the LogAggregatorService as an intermediate step.

By default, Query Workers are configured to write logs via the LogAggregatorService and DataImportServer to the DbInternal/ProcessEventLog table in Deephaven.

To enable Query Worker logs to write their logs to plain text files, the following properties can be set in /etc/sysconfig/illumon.d/resources/iris-common.prop:

RemoteQueryDispatcher.writeDatabaseProcessLogs=false
RemoteQueryProcessor.sendLogsToSystemOut=true

The following Query Worker log entries provide a sample of the content found in a query worker's log file:

[2018-02-20T23:04:19.476898-0500] - WORKER_1 - ConnectionMonitor: registering monitored connection job:1744982467/AuthServer_Client/10.128.0.38:42466->10.128.0.38:9030/CommandConnection
[2018-02-20T23:04:19.484982-0500] - WORKER_1 - Successful authentication worker8.internal/10.128.0.38 using delegate token {-1691233174684810865, service
: DelegatedAuthentication, origin: worker8.internal/10.128.0.38, tokenContext: {iris}}
[2018-02-20T23:04:19.486329-0500] - WORKER_1 - Serializing result of type java.lang.Boolean
[2018-02-20T23:04:19.486838-0500] - WORKER_1 - Sending serialized result, length=47 bytes

Note

It's recommended to query the ProcessEventLog rather than reading log files on disk for code running on a query worker. Additionally, avoid having a query worker monitor its own log outputs, as monitoring logs produces more log output.

Garbage collection logs

Deephaven Java processes are configured to record JVM Garbage Collection (GC) statistics. These GC logs can be very useful to understand JVM memory usage or diagnose any memory problems.

Deephaven, by default, uses the G1 garbage collector. Refer to Java's documentation for more information on how G1 garbage collection works.

Log file maintenance

Customers should create a job to archive these logs based on their own internal retention policies. Deephaven recommends at least a few weeks of logs be retained in case troubleshooting is needed.