Internal tables

Audit Event Log

Configuration

• Any process that can write to the Audit Event Log or Process Event Log can override several configuration items. All configuration overrides should be based on the process name or main class name.
  • <process name>.useLas - if true, then the events will be written through the LAS; if false, then events will be written directly to binary log files.
  • <process name>.useMainClassNameForLogs - whether to use the class name for log entries; if false, then the retrieved value from the process.name property will be used instead of the class name.
• RemoteQueryProcessor.logCommands - defines whether or not all received commands are logged. The default value is false.
• LocalInternalPartition - if specified, defines the internal partition to which data will be written; if not defined, then a local partition based on the host name will be used.

Auditable events by process

Worker audit events (RemoteQueryProcessor)

• ACL details - full details on the ACLs for a user connected to the worker
• Client async query - a client sent an asynchronous query request
• Client command - a command was received from a client; only logged if the RemoteQueryProcessor.logCommands property is true.
• Client disconnection - a client disconnected from the worker
• Client sync query - a client sent a synchronous query request
• Command cancel - a worker command was cancelled
• Disconnect - a client disconnected from the worker
• Primary client - the worker's primary client connected
• Secondary client - a secondary client requested an action from the worker; the details contain further information
• Script source (controller) - a script was loaded by requesting it from the persistent query controller; if an exception occurred, it will be shown in the details
• Script source (remote) - a script was loaded from a remote source, which may be a console
• Table Access - a user attempted to access a table

Process Event Log

Configuration

The following configuration parameters define the process event log configuration.

RemoteQueryProcessor.sendLogsToSystemOut - if defined and set to true, tells the query workers to send their logs to standard system output. This cannot be used when writing to the process event log.

Any process that can write to the process event log can override several configuration items. All configuration overrides should be based on the process name or main class name.

• <process name>.writeDatabaseProcessLogs - if true, then process event logs will be created; if false, then process event logs will not be created.
• <process name>.useLas - if true, then audit and process events will be written through the LAS; if false, then events will be written directly to binary log files.
• <process name>.useMainClassNameForLogs - whether to use the class name for log entries; if false, then the retrieved value from the process.name property will be used instead of the class name.
• <process name>.logLevel - the minimum log level event which will be written. The default value is INFO. Allowed values are:
  • FATAL
  • EMAIL
  • STDERR
  • ERROR
  • WARN
  • STDOUT
  • INFO
  • DEBUG
  • TRACE
• <process name>.captureLog4j - if true, any output sent to the Log4J logger is written into the process event log.
• <process name>.captureSysout - if true, any system output is written into the process event log.
• <process name>.captureSyserr - if true, any error output is written into the process event log.
• <process name>.aliveMessageSeconds - if non-zero, a message is periodically written to the process event log indicating that the process is still alive.
• LocalInternalPartition - if specified, defines the internal partition to which data will be written; if not defined, then a local partition based on the host name will be used.

Write entries to CSV files

It is possible to write ProcessEventLog entries to CSV files. To turn this on, specify the following property for the dispatchers and workers:

ProcessEventLog.interceptor=com.illumon.iris.db.util.logging.ProcessEventLogInterceptorCsv

Also specify a full path name to a directory where the CSV files will be written with the property. This directory must be writable by all the processes that will generate these files; typically making it group-writable by dbmergegrp will be adequate:

ProcessEventLog.interceptor.csv.directory=/path/to/directory

CSV file names will consist of the following pattern:

<PQ name if available>-<process name>-<host name>-<optional GUID>.date/timestamp

Some messages (during initial worker startup and shutdown) will be logged in the dispatcher’s log instead of the workers' logs.

The following properties define the behavior:

• ProcessEventLog.interceptor.csv.format - an optional CSV format, from org.apache.commons.csv.CSVFormat#Predefined. If none is specified, the default is Excel.
• ProcessEventLog.interceptor.csv.delimiter - an optional delimiter. If none is specified (the property is non-existent or commented out), the default is taken from the CSV format. Delimiters must be one character.
• ProcessEventLog.interceptor.csv.queueCapacity - to ensure that CSV writes do not affect performance, all CSV operations are submitted to a queue and performed off-thread. This specifies the queue’s capacity. If the capacity is exceeded (because the writer thread can’t keep up), further writes will hold up the process until there is available queue capacity. The default queue capacity is 1,000.
• ProcessEventLog.interceptor.csv.rolloverDaily - if this is specified, the CSV files will roll over daily. The default is true. If the files are rolling over daily (or not at all), the date/timestamp will be in the format yyyy-MM-dd, such as 2021-04-03.
• ProcessEventLog.interceptor.csv.rolloverHourly - if this is specified, the CSV files will roll over hourly. This takes precedence over daily rollover. The default is false. If the files are rolling over hourly, the date/timestamp will include a time and offset, such as 2021-04-29.150000.000-0400.
• ProcessEventLog.interceptor.csv.timeZone - the time zone to be used for filenames and timestamps in the CSV files. The default is the system default time zone. This is the text name of the time zone, such as America/New_York.
• ProcessEventLog.interceptor.csv.flushMessages - how frequently to flush the queue to disk (it will always flush when the queue is emptied). The default value is 100.

If you are not seeing CSV files being created, we recommend the following steps:

1. Check the most recent startup log or the process event log for the worker. If there is a configuration error in the interceptor properties, this is where it will most likely show up. It is designed to not prevent process and worker startup if it is misconfigured.
2. Check the permissions on the directory to which CSV files are being written. It will need to be writable by all the processes, typically dbmergegrp.
3. Since the PQ name is part of the filename, special Linux file path characters can cause issues. For example, a forward-slash / will be interpreted as a directory separator. For this case, appropriate subdirectories will need to be created to hold the CSV files.

Process Info

The Process Info table (ProcessInfo) captures the system properties, JVM arguments, memory and memory pool info, and other initial conditions of processes on startup. Its information is intended primarily for debugging purposes.

To disable the ProcessInfo table, set the following property to false:

IrisLogDefaults.writeDatabaseProcessInfo=false

Columns

Process Metrics

The Process Metrics table (ProcessMetrics) captures internal metrics that were previously written to the stats.log CSV. The table is disabled by default.

To enable the ProcessMetrics table, set the following property to true:

IrisLogDefaults.writeDatabaseProcessMetrics=true

Columns

Process Telemetry

The DbInternal.ProcessTelemetry table enables monitoring UI performance of the Swing front-end. For each user-initiated action, the console logs the duration between the start of the operation and when the table is ready for use. By aggregating this telemetry, an overall picture of the system's health - as perceived by users - is available. Detailed information can then be used to investigate potential performance problems.

To write these events, the Swing console internally buffers the data and then sends it to the new TelemetryHelperQuery query. If the TelemetryHelperQuery is not running, data is buffered up to a configurable limit, at which point the oldest telemetry data is discarded.

Permissions

For new installations, an ACL is automatically applied to the Process Telemetry table. For an upgrade, an ACL editor must create a new "Table ACL" for the raw DbInternal.ProcessTelemetry data to be seen by unprivileged users. The ACL should be similar to the "allusers/DbInternal.ProcessEventLog" ACL, but for the ProcessTelemetry table:

allusers | DbInternal | ProcessTelemetry | new UsernameFilterGenerator("EffectiveUser")

Configuration

Several new startup parameters define the Telemetry behavior for the console under the [service.name=iris_console] stanza:

Columns

telemetrySummary()

The telemetrySummary() function, which accepts an optional "Date" parameter, is available in the default Groovy session. The method will provide summary tables derived from the raw Telemetry data that aggregate the data in meaningful ways.

Worker Performance Tables

Legacy workers write three performance related tables, which can be analyzed with performanceOverview.

Query Operation Performance Log

The Query Operation Performance Log (QueryOperationPerformanceLog) contains performance details on Deephaven query operations. Each query is broken up into its component parts for this log, allowing in-depth understanding of the performance impacts of each individual operation for a query.

Columns

Query Performance Log

The Query Performance Log (QueryPerformanceLog) contains details on query-level performance for each worker. A given worker may be running multiple queries; each will have its own set of query performance log entries.

Configuration

RemoteQueryDispatcher.logQueryPerformance - specifies whether or not to log query performance.

Columns

Update Performance Log

The Update Performance Log (UpdatePerformanceLog) contains aggregated performance details on incremental update operations performed in the LiveTableMonitor loop.

Update performance logging allows three types of logging.

1. Database - for workers only, this writes update performance data to the UpdatePerformanceLog table's intraday data
2. Log - the update performance data is printed to standard out
3. Listener - the update performance data is supplied to registered UpdatePerformanceTracker.Listeners; this can be used to programmatically handle the performance logs in-process.

The mode is driven by the UpdatePerformanceTracker.reportingMode property, which can be set to the following values:

• NONE
• LOG_ONLY
• DB_ONLY
• LISTENER_ONLY
• LOG_AND_LISTENER
• DB_AND_LISTENER
• LOG_AND_DB
• ALL

Columns

Log Index Tables