Working with Binary Log Files

Binary Log File Partitions

Real-time Deephaven data ingestion is typically performed by using an application to log data to Deephaven binary log files, and configuring a tailer process to send these logs to one or more Data Import Server processes. Each binary log file is specific to a given namespace, table, internal partition and column partition.

The filename of each binary log is always used to determine the column partition value (usually the date) for the data within that file, and this must be created correctly by the application. It is recommended that the log filenames be in a standard format of:

Here's an example log generated by the Persistent Query (PQ) controller:

This consists of:

• DbInternal - The namespace.
• PersistentQueryStateLog - The table name.
• System - The table's type.
• PersistentQueryController_vm1 - The internal partition name as generated by the application; in this case, it is the process name followed by a host name.
• 2023-03-21 - The column partition value, indicating the partition into which this log file's data will be loaded.
• 2023-03-21.151353.472+0000 - Further details used to order the log files; in this case, a millisecond-level timestamp and a timezone offset.

Obviously, it is critical that the file name for these binary log files be generated correctly. In particular, the column partition value and following details are dynamic.

Binary log transactions

Binary log transactions allow multiple rows to be grouped together as an atomic unit. When a transaction is written to a binary log, all rows in that transaction are either committed together or rolled back together, maintaining data consistency across the rows.

Transaction flags

Binary log rows can be marked with transaction flags that indicate their role within a transaction. These flags are defined in the Row.Flags enumeration and include:

• StartTransaction - Marks the first row of a transaction.
• EndTransaction - Marks the last row of a transaction.
• SingleRow - Indicates a standalone row that is both the start and end of a single-row transaction.
• None - Default flag for rows that are part of a transaction but are neither the start nor end.

Rows between StartTransaction and EndTransaction flags form atomic units that are either all committed or all rolled back together.

Cross-table limitations

Binary log transactions operate on a single table only. If your application requires coordinated updates across multiple tables, you cannot rely on binary log transactions alone. Instead, you must use query engine tools like SyncTableFilter or LeaderTableFilter to ensure that related data across tables remains consistent. See Synchronizing tables for more information.

Impact on Log Aggregator Service

Large transactions can significantly impact Log Aggregator Service (LAS) memory usage. The LAS must buffer entire transactions because:

1. The LAS may receive data from multiple input streams.
2. Transactions cannot interleave with other rows — they must be processed atomically.
3. All transaction rows must be held in memory until the transaction completes.

For example, a 4-million-row transaction will be buffered entirely in memory until the EndTransaction flag is encountered. Very large transactions can exhaust LAS memory and cause service failures.

To mitigate this risk, Deephaven enforces transaction size limits. When designing applications that use binary log transactions with LAS in the data path, carefully consider transaction sizes and available memory.

Impact on Data Import Server

Unlike the LAS, the Data Import Server (DIS) is not significantly impacted by large transactions. The DIS:

1. Requires a single connection from a tailer.
2. Writes uncommitted transaction data directly to column files as it arrives.
3. Does not include uncommitted data in checkpoint records.
4. Does not publish uncommitted data to consumers until the transaction completes.
5. Can roll back transactions if necessary, even after writing pending rows to disk.

Because the DIS writes transaction data to disk incrementally rather than buffering it in memory, transaction size does not create memory pressure. A 4-million-row transaction has approximately the same memory footprint as individual rows.

Java Loggers

Creating a logger in Java always consists of two steps:

1. The logger instance (of the class generated from the schema) is created.
2. The logger instance is initialized, indicating how it is to generate binary log file names.

After initialization is complete, the logger can be used to write data.

IntradayLoggerBuilder

The IntradayLoggerBuilder provides a fluent API for creating and initializing intraday loggers. It handles file naming, partitioning, and logger configuration automatically based on the provided settings.

Builder options

The following table describes all available builder methods:

Column partition determination

When makeLogger is called, column partitioning is determined in the following order of precedence:

1. columnPartitionValue — If provided, used for all rows regardless of other settings.
2. Dynamic partitioning — If enabled via properties (loggerClass.useDynamicPartitions=true, processName.useDynamicPartitions=true, or loggerClass.processName.useDynamicPartitions=true), the partition is determined from schema and data.
3. Time zone-based — Uses the specified time zone (or system default) to determine the partition from the write timestamp.

Logging destination

Rows are written directly to binary log files unless:

• The properties specify LAS (processName.useLogAggregatorService=true or loggerClass.processName.useLogAggregatorService=true), or
• setForceUseLas(true) is called.

Internal partition naming

The internal partition is constructed as follows:

1. Base partition from getInternalPartitionFromConfig or getNodeIdentifier.
2. Optional suffix from setInternalPartitionSuffix (e.g., ABC-DEF).
3. Optional log format suffix from setSuffixInternalPartitionWithLogFormat (e.g., ABC-DEF-4).

Example with all suffixes:

To use zero-padded log format suffixes for proper lexicographic ordering:

Static utility methods

Factory classes

Several Deephaven Java factories are available to assist in the creation of binary log file names. These factories all assume the creation of a standard Deephaven logger class from a Deephaven schema. These factories are used during the initialization of the generated logger classes and automatically roll over filenames.

All code examples assume that the following static variables are defined:

• private static final int BINARYSTORE_WRITER_VERSION = 2;
  • BINARYSTORE_WRITER_VERSION defines the version of the Deephaven binary store writer, and should always be 2.
• private static final int BINARYSTORE_QUEUE_SIZE = 10000;
  • BINARYSTORE_QUEUE_SIZE is a default value to be used for the queueSize parameter in logger initialization.

They also assume a Deephaven Configuration instance is available, using a statement such as the following:

Factory Usage

To use factories, create a logger instance and call the initialization method on the created logger, passing in the factory instance. The initialization method is defined as follows:

• tableWriterBuilder - The factory to be used to build the files.
• queueSize - Defines the number of entries created during logger initialization; if this number is exceeded, the application may pause while waiting for entries to be freed.

LongLivedProcessBinaryStoreWriterFactory

LongLivedProcessBinaryStoreWriterFactory can be used to create filenames that automatically roll over from hour to hour, and from day to day. It always uses the time zone of the system on which the logger is running.

Every hour the factory rolls the binary log files over, creating a new log file with its name based on the server's current date and time. The rollover time and new filename are based on the time at which the data is written to the log. There is currently no logic to determine the column partition value (date) from the logged data.

If a process is logging data continuously, it may log data without a break as the date changes (midnight). In this case, data at the end of the day may be written slightly after the date change, in which case it will go into the next day's partition.

Ideally, to avoid the rollover issue, the factory will be used with an appropriate time zone where the application is not logging data around midnight.

The LongLivedProcessBinaryStoreWriterFactory constructor is defined as follows:

• path - The complete path to which the logs will be written. This includes the directory, and the filename up to .bin.
• version - The writer version to be used, usually 2.
• log - A Deephaven Logger instance to which errors will be logged.

The following example initializes a logger for the PersistentQueryStateLog shown earlier. It uses the configuration.getLogPath() method to determine the default log directory location:

BinaryStoreWriterFactory

The BinaryStoreWriterFactory can be used to create files that roll over from hour-to-hour, but not from day-to-day. It allows definition of the time zone to be used to determine the filenames, and even the filename format, although changing the filename format is not recommended. (The recommended format is yyyy-MM-dd.HHmmss.SSSZ, as this will be handled correctly by the tailer's default configuration.)

The BinaryStoreWriterFactory does not roll over days: the date is always set to the date when the log was initialized.

The BinaryStoreWriterFactory constructor is defined as follows:

• path - The complete path to which the logs will be written. This includes the directory, and the filename up to .bin.
• rollFormat - A DateFormat that specifies the format in which the binary log files will be written.
• version - The writer version to be used, usually 2.
• log - A Deephaven Logger instance to which errors will be logged.

The following example initializes a logger for the PersistentQueryStateLog shown earlier. It uses the configuration.getLogPath() method to determine the default log directory location, and writes logs using the date value for the Asia/Tokyo time zone:

Manual file creation

Another option for creating binary log file names is for the application to manage them itself. In this case, when the logger is initialized, the filename is specified including the date partition. A BinaryStoreWriterFactory will be created for the user, and it will roll over based on the system's current time. This should be used with caution as when the time ticks past midnight, the filename timestamps will also reset.

To use this method, a logger is created and initialized directly. The logger's initialization signature is defined as follows:

• path - The full path for this log.
• queueSize - Defines the number of entries created during logger initialization; if this number is exceeded, the application may pause while waiting for entries to be freed.

For example:

C++ Loggers

The Deephaven C++ logger is exposed in the BinaryStoreWriter.hpp as BinaryStoreWriter. When constructing the log file name, you may pass in a simple file name, which will be used as the file name of your log file and no further rotation or changes are made to the binary log file name.

You may alternatively pass in a rotation interval, expressed in seconds. The file will be rotated on the rotation interval, rounding to the specified value. For example, if you pass in an interval of 900 seconds, then the file is rotated every fifteen minutes on the hour, and at 15, 30, and 45 past the hour. To generate the log file name, the logger concatenates the base file name passed in, a dot ("."), and the time the file was created. For the first file, the actual time is used. For subsequent files, the time is rounded to the beginning of the rotation interval. This provides predictable log file names.

The file dates are formatted using the following strftime format, %F-%H%M%S%z, producing dates of the format YYYY-MM-DD-HHMMSS-TZ. For example, with a base file name of "trades-out.bin" the logger would produce a file name of the form "trades-out.bin.2018-05-31-082542-0400". Presently, the time zone of the date format will be the time zone of the machine producing the logs. To use alternative time zones for the column partition, the application must format the column partition value internally, and the Deephaven administrator must update the tailerConfig.xml to appropriately parse the file name format containing both the column partition and the local date.

Related documentation

• Batch binary log files
• Binary log format