Binary log files

Several Deephaven processes write binary log files, which contain data that will be written to tables in the DbInternal namespace. The locations of these log files are driven by properties similar to the application log files, including the logroot property.

Binary log locations

When writing a binary log file, the application will determine the location using a three-tiered property search, using the namespace and table name; the most specific result found is used as the location.

binaryLogDir.<namespace>.<table name>
binaryLogDir.<namespace>
binaryLogDir

If no location is defined for the table or namespace, the location defined by the binaryLogDir property will be used.

The default location for binary logs is /var/log/deephaven/binlogs. This is defined in Deephaven's property files:

binaryLogDir=<logroot>/binlogs

The default location for binary logs uses the same log root as the application logs (/var/log/deephaven), but defines a different subdirectory:

binaryLogDir=<logroot>/binlogs

Process Event logs have their own subdirectory:

binaryLogDir.DbInternal.ProcessEventLog=<logroot>/binlogs/pel

Performance logs have a different subdirectory:

binaryLogDir.DbInternal.QueryOperationPerformanceLog=<logroot>/binlogs/perflogs
binaryLogDir.DbInternal.QueryPerformanceLog=<logroot>/binlogs/perflogs
binaryLogDir.DbInternal.UpdatePerformanceLog=<logroot>/binlogs/perflogs
binaryLogDir.DbInternal.QueryOperationPerformanceLogCoreV2=<logroot>/binlogs/perflogs
binaryLogDir.DbInternal.QueryPerformanceLogCoreV2=<logroot>/binlogs/perflogs
binaryLogDir.DbInternal.UpdatePerformanceLogCoreV2=<logroot>/binlogs/perflogs
binaryLogDir.DbInternal.ServerStateLogCoreV2=<logroot>/binlogs/perflogs

View binary log files with iriscat and iristail

Because Deephaven log files on disk are in binary format, they can't be viewed as text. Instead use the iriscat utility. This can come in handy when there are system errors preventing you from querying the database using the Deephaven Console GUI client.

You can run iriscat using this command:

/usr/illumon/latest/bin/iriscat <path to binary file>

For example:

/usr/illumon/latest/bin/iriscat
/var/log/deephaven/binlogs/test.file.bin.2018-02-02.141436.113-0800

This will display the contents of the file in CSV format. To view them in the same file as formatted text, use the -l flag. For example:

/usr/illumon/latest/bin/iriscat -l <path to binary file>

You can get the full list of options with --help:

$ /usr/illumon/latest/bin/iriscat --help
usage: IrisCat [-C <columns>] [-d] [-e <endRow>] [-F <format>] [-l] [-s <startRow>] [-S <scale>] [-T <timeZone>]
 -C <columns>    Names of columns to display, comma-separated
 -d              Show debugging information (row flags, column metadata)
 -e <endRow>     Index of the last row displayed
 -F <format>     Format date time columns, comma-separated (e.g. "Column=HH:mm:ss.SSS", or "Column" for ISO8601 microsecond)
 -l              Format output as if it were a log file (useful for ProcessEventLog)
 -s <startRow>   Index of the first row displayed
 -S <scale>      Scale to use when interpreting date time column values (SECOND|MILLI|MICRO|NANO). If not specified, will attempt to infer from value magnitudes.
 -T <timeZone>   Time zone used to interpret and display time columns (e.g. DST or CST)

Additionally, the iristail command functions similarly, but looks at the end of a binary log, with the additional ability to tail the log file:

/usr/illumon/latest/bin/iristail -l <path to binary file>

You can get the full list of options with --help:

/usr/illumon/latest/bin/iristail --help
usage: IrisTail [-C <columns>] [-d] [-f] [-F <format>] [-l] [-n <number>]
       [-S <scale>] [-T <timeZone>]
 -C <columns>    Names of columns to display, comma-separated
 -d              Show debugging information (row flags, column metadata)
 -f              output appended data as the file grows
 -F <format>     Format date time columns, comma-separated (e.g.
                 "Column=HH:mm:ss.SSS", or "Column" for ISO8601
                 microsecond)
 -l              Format output as if it were a log file (useful for
                 ProcessEventLog)
 -n <number>     number of rows to display
 -S <scale>      Scale to use when interpreting date time column values
                 (SECOND|MILLI|MICRO|NANO). If not specified, will attempt
                 to infer from value magnitudes.
 -T <timeZone>   Time zone used to interpret and display time columns
                 (e.g. DST or CST)