Log Aggregator Service runbook

The Log Aggregator Service (LAS) is a critical Deephaven data management service that combines binary log entries from multiple processes and writes them into combined binary log files. It prevents write contention by serializing writes to each output stream, ensuring only one writer per log location at a time.

Impact of Log Aggregator Service failure

Level	Impact
Sev 1 - Critical	Any process configured to use the LAS will fail to write logs to the database. This will cause failure of these processes, including the query workers.

Caution

Workers are typically configured to require successful LAS connection before starting. If LAS is down, new workers cannot start.

Log Aggregator Service dependencies

The LAS requires:

Configuration Server — Must be running to access routing and schema configuration.
Authentication Server — Must be running for token validation.
Filesystem access — Must have read/write access to binary log storage paths.
etcd cluster — Must be accessible (via Configuration Server).

The LAS does not depend on Remote Query Dispatcher, allowing it to start independently.

Checking Log Aggregator Service status

Check process is running with monit:

dh_monit status log_aggregator_service

Expected output should show status Running.

Viewing Log Aggregator Service logs

View application log:

cat /var/log/deephaven/las/LogAggregatorService.log.current

Tail the log to follow in real-time:

tail -f /var/log/deephaven/las/LogAggregatorService.log.current

List historical log files:

ls -ltr /var/log/deephaven/las/log_aggregator_service.log.????-??-??

View process stdout/stderr logs:

cat /var/log/deephaven/las/log_aggregator_service.log.$(date +%Y-%m-%d)

Restart procedure

Restart the LAS:

dh_monit restart log_aggregator_service

Warning

Restarting LAS temporarily interrupts log writing from all workers. New workers cannot start until LAS is back online. Running workers may terminate if they cannot buffer log data during the restart.

Verify the restart was successful:

dh_monit status log_aggregator_service

Monitor the log during startup:

tail -f /var/log/deephaven/las/LogAggregatorService.log.current

Configuring the Log Aggregator Service

The LAS server configuration is managed through the logAggregatorServers section of the routing YAML. Client processes enable LAS logging via the useLogAggregatorService property. See Metrics and monitoring for client configuration and Data Routing Service YAML format for server configuration.

Log Aggregator Service deployment

By default, all nodes in a Deephaven cluster run a LAS. Each LAS writes to its own set of binary log output streams. The data routing configuration determines which LAS handles which output streams. For user table data, all sources are directed to the same LAS instance to maintain ordering guarantees.

Configuration files and locations

monit configuration: /etc/sysconfig/illumon.d/monit/log_aggregator_service.conf

Property files:

/etc/sysconfig/illumon.d/resources/iris-common.prop
/etc/sysconfig/illumon.d/resources/log_aggregator_service.prop

Binary log output directory: Defaults to /var/log/deephaven/binlogs/ (configurable via binaryLogDir property; see Binary log files)

Process log directory: /var/log/deephaven/las/