---
title: Intraday Binary Log format configuration (deprecated)
sidebar_label: Intraday Binary Log format configuration (deprecated)
---

For a general overview of streaming data, see [streaming introduction](./streaming-intro.md).

> [!WARNING]
> The `LoggerListener` and `Listener` XML elements are deprecated and will be removed in a future release.
> See [binary log format configuration](./binary-log-format-configuration.md) for the replacement.

### The Logger Interfaces

The customer application might have "rows" of data to stream in two main formats:

1. sets of values using basic data types like double or String, and
2. complex objects that are instances of custom classes.

The generated logger class will provide a log method that will be called each time the application has data to send to Deephaven, with its arguments based on the schema. To create the format for the log method, the logger class will always implement a Java interface. Deephaven provides several generic interfaces based on the number of arguments needed. For instance, if three arguments are needed, by default the Deephaven generic ThreeArgLogger is used. These arguments might be basic types such as double or String, custom class types, or a combination of both.

Deephaven provides generic logger interfaces for up to eight arguments, plus a special MultiArgLogger interface for loggers with more than eight arguments. The MultiArgLogger is more generic than the other interfaces in that the other interfaces will have their arguments typed when the logger code is generated, while the MultiArgLogger will simply take an arbitrarily long list of objects as its arguments. One known limitation of the MultiArgLogger is that it cannot accept generic objects among its arguments. In this case "generic objects" refers to objects other than String or boxed primitive types. The logger interfaces for fixed numbers of arguments do not have this limitation.

In many cases the events to be logged will have a large number of properties. Rather than use the MultiArgLogger there are two other approaches that are preferred: either create a custom logger interface or pass the event as a custom object.

In most cases the custom object is the easier solution, since such an object probably already exists in the API of the data source from which the custom application is receiving data. For example, if the custom application is receiving Twitter tweets as tweet objects, this object type could be added to the schema as a SystemInput, and Tweet properties could be used in the intradaySetters:

```xml
<SystemInput name="tweet" type="com.twitter.event.Tweet" />

...

<Column name="text" dataType="String" intradaySetter="tweet.getText()" />

<Column name="deleted" dataType="Boolean" intradaySetter="tweet.getDeleted()" />
```

This way, the customer application only needs to pass a Tweet object to the log() method instead of having to pass each of the properties of the object.

#### Custom Logger Interfaces

The other option for logging more than eight properties is to define a custom logger interface. A custom logger interface extends `IntradayLogger` and specifies the exact names and types of arguments to be passed to the `log()` method:

```java
public interface TickDataLogFormat1Interface extends IntradayLogger {
   void log(Row.Flags flags,
      long timestamp,
      String name,
      float price,
      int qty);

default void log(long timestamp,
      String name,
      float price,
       int qty)
   {
    log(DEFAULT_INTRADAY_LOGGER_FLAGS, timestamp, name, price, qty);
   }
}
```

Note the `timestamp` argument. This column is often included in Deephaven tables to track when a logger first "saw" a row of data. By default, this is "now" in epoch milliseconds at the time the event was received. If needed, a custom intradaySetter and dbSetter can be specified to use other formats or precisions for this value.

A custom logger interface should specify any exceptions the `log()` method will throw. For instance, a logger that handles BLOB arguments will need to include throws IOException as part of its `log(...)` method declarations:

```java
void log(<parameters as above>) throws IOException { ... }
```

In this case, the "1" in the name (TickDataLogFormat1Interface) is to denote this is the first version of this interface, which is helpful if we later need to revise it. This is convention and recommended, rather than an enforced requirement in naming logger interfaces.

### Log Formats

Each logger and listener corresponds to a distinct version number, which is specified in a `logFormat` attribute when
defining a logger or listener in the schema. A listener will only accept data that has been logged with a matching
column set and version number. Log formats are helpful when making schema modifications: if columns are added or removed,
additional logger/listener elements can be added to support the new set of columns as a new log format, without losing
support for binary logs generated using earlier versions of the schema. The existing logger/listener elements can be
updated to ignore the new columns — for example, a new column can be excluded from loggers for an older log format by
setting `intradayType="None"`, and default values for the new columns be configured within the listener by setting the
`dbSetter` attribute (such as `dbSetter=NULL_LONG`).

If a log format version is not specified, it will default to "0".

### Loggers in Deephaven Schemas

Loggers are defined in a `<Logger>` element, only required when a Java or C# logger is needed. A logger element has the
following attributes:

| Element Name             | Optional/Required | Default                                     | Description                                                                                                                                                                                                                                                                                 |
| ------------------------ | ----------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `logFormat`              | Optional          | `0`                                         | Specifies the logger's version number.                                                                                                                                                                                                                                                      |
| `loggerPackage`          | Required          | N/A                                         | Specifies the Java package for the generated code.                                                                                                                                                                                                                                          |
| `loggerClass`            | Optional          | Generated from `logFormat` and table `name` | Specifies the Java class name for the generated code; defaults to a value that includes the table name and log format version (if non-zero); e.g., `TestTableLogger`, `TestTableFormat2Logger`. If specified, the value (class name) must include "Logger" somewhere within its definition. |
| `loggerInterface`        | Optional          | Determined from `SystemInput` elements      | Specifies a Java interface that the generated logger class will implement. Defaults to a generic interface based on the number of system input parameters; e.g., `com.illumon.intradaylogger.FourArgLogger`. The class name must include the word "Logger".                                 |
| `loggerInterfaceGeneric` | Optional          | N/A                                         | The use of this attribute is deprecated.                                                                                                                                                                                                                                                    |
| `loggerLanguage`         | Optional          | `JAVA`                                      | Specifies the logger language. If not specified, a default value of `JAVA` will be used. Supported values are:<br />- `JAVA` <br />- `CSHARP` or `C#`                                                                                                                                       |
| `tableLogger`            | Optional          | `false`                                     | If specified as `true` this indicates that a logger should be generated that can write data directly to a Deephaven table.                                                                                                                                                                  |
| `verifyChecksum`         | Optional          | `true`                                      | If specified as `false` then the logger loaded by a logging application will not be checked against the latest logger generated by the schema. This configuration is not recommended.                                                                                                       |
| `generateLogCalls`       | Optional          | `true`                                      | If specified as `true`, the logger's `log` methods should be generated.                                                                                                                                                                                                                     |

#### Logger code generation elements

Logger code generation is supplemented by the following elements:

| Element Name      | Optional/Required                                             | Description                                                                                                                                                                                                                                                                                       |
| ----------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `<LoggerImports>` | Optional                                                      | Specifies a list of extra Java import statements for the generated logger class.                                                                                                                                                                                                                  |
| `<LoggerFields>`  | Optional                                                      | Specifies a free-form block of code used to declare and initialize instance or static members of the generated logger class.                                                                                                                                                                      |
| `<SystemInput>`   | Required unless the `loggerInterface` attribute was specified | Declares an argument that the Logger's `log()` method will take, with two attributes:<br /><br />- `name` - the name of the argument, available as a variable to `intradaySetter` expressions.<br />- `type` - the type for the argument; any Java type including those defined in customer code. |
| `<ExtraMethods>`  | Optional                                                      | If specified, these are extra methods added to the generated logger class.                                                                                                                                                                                                                        |
| `<SetterFields>`  | Optional                                                      | If specified, these are extra fields (variables) added to the setter classes within the generated logger.                                                                                                                                                                                         |

The application should call the `log()` method once for each row of data, passing the data for the row as arguments.

##### Logger interfaces

All generated loggers must implement the `com.illumon.intradaylogger.IntradayLogger` interface, which defines the
base methods required of a Deephaven logger. This does not include the `log()` method that is actually used by
applications for logging data.

Generated loggers will automatically implement an interface that defines a `log()` method that takes as many arguments
as there are `<SystemInput>` elements. There are specific interfaces for up to eight input parameters; beyond that,
the `MultiArgLogger` interface is used, which defines a varargs `log()` method.

| Interface                                   | Signature of `log()` method                                          |
| ------------------------------------------- | -------------------------------------------------------------------- |
| `com.illumon.intradaylogger.OneArgLogger`   | `log(T1 value)  throws IOException`                                  |
| `com.illumon.intradaylogger.FourArgLogger`  | `log(T1 value1, T2 value2, T3 value3, T4 value4) throws IOException` |
| `com.illumon.intradaylogger.MultiArgLogger` | `log(Object... values) throws IOException`                           |

These interfaces can be used to develop applications that must log data, but do not have access to the generated logger
code at compile time.

###### Custom logger interfaces

Custom logger interfaces are helpful because they allow application development with a `log()` method specific to the
intended argument types, without generating copies of the logger on the development workstation (e.g., when not
developing with the [Deephaven Gradle plugin](../../resources/how-to/local-query-development.md)). This allows logger
interfaces with a `log()` method that can take primitive arguments (rather than only Java Objects) and that can have
more than eight method arguments (without using varargs). For some cases of high-throughput logging, using Java boxed
types instead of primitive types (such as `java.lang.Double` instead of `double`) can adversely affect performance. This
is true of varargs calls as well, since Java varargs calls require copying all arguments into a new array.

A custom logger interface must extend the `IntradayLogger` interface and define a `log()` method whose signature matches
the `<SystemInput>` elements. Specifically:

- The `log()` method in the custom interface must have the same number of arguments as there are `<SystemInput>`elements
  for the log format.
- The `type` of each `<SystemInput`> must match the argument types in the custom interface's `log()` method for the same
  position. For example, if the `log()` method's signature is `log(char myFirstVal, Number mySecondVal)`, then the
  `type` of the first `<SystemInput>` must be `byte`, and the `type` of the second `SystemInput` must be a `Number` or a
  subclass of `Number` (such as `Integer`, `Double`, or `BigInteger`).

See [Appendix G](#appendix-c-example-custom-logger-interface) for an example custom logger interface.

#### Logger `<Column>` elements

The `<Column>` elements declare the columns contained in the logger's output. A logger's column set typically matches the destination table's column set, but this is not a requirement, as the listener can convert the data from the logger's format to the columns required for the table. Each column in the table's Column set must exist in the Logger Column elements.

The `<Column>` element for a logger supports the following attributes:

| Attribute Name           | Optional/Required | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| ------------------------ | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `datePartitionInput`     | Optional          | Specifies that if the logger is Java, the column's value can be used to automatically calculate the column partition value for every logged row. The column's time precision must be specified in the attribute. Available values include:<br />- seconds <br />- millis <br />- micros <br />- nanos To use this option the logging program must initialize the logger with an appropriate time zone name. See the [streaming binary logs](../../crash-course/data-in/streaming-binlogs.md) section in the Crash Course.                                                           |
| `directSetter`           | Optional          | A Java expression that uses the logger arguments as inputs to produce the value to be used in this column. This should be set whenever the `intradaySetter` is set. The `directSetter` should produce the same result for a column as the corresponding `dbSetter` in the listener for this log format. To automatically use the `intradaySetter` expression as the `directSetter`, use the special value `"matchIntraday"`.                                                                                                                                                        |
| `intradayType`           | Optional          | The data type for values in the log, which may be different from the `dataType` for the Column. If `intradayType` is not specified, then the column' s `dataType` is used. An `intradayType` of `none` indicates that this column should not be included in the log.                                                                                                                                                                                                                                                                                                                |
| `intradaySetter`         | Required          | A Java expression to produce the value to be used for this column in the binary log file. This can customize how a value from the logging application should be interpreted or converted before writing it into the binary log file. The expression may use the names of any `<SystemInput>` elements as variables and perform valid Java operations on these variables. The expression must return a value of the `intradayType`.                                                                                                                                                  |
| `functionPartitionInput` | Optional          | If set to `true`, this attribute specifies that if the logger is Java, the column's value can be used to calculate the column partition value for every logged row by using a lambda provided in the logger initialization. To use this option the logging program must initialize the logger with an appropriate lambda. See the [streaming binary logs](../../crash-course/data-in/streaming-binlogs.md) section in the Crash Course. Note that when this is set on a primitive type column, the generated function will use the corresponding boxed type.                        |
| `name`                   | Required          | The name of the column in the logger's output. A corresponding `<Column>` element with the same name must exist in the listener that corresponds to the logger's `logFormat` version.                                                                                                                                                                                                                                                                                                                                                                                               |
| `timePrecision`          | Optional          | The time precision assumed when reading from and writing to a DateTime column in a log file. Can be one of `second`, `millis`, `micros`, or `nanos`. If not specified, this attribute defaults to `millis`.                                                                                                                                                                                                                                                                                                                                                                         |
| `partitionSetter`        | Optional          | This attribute is only valid for the partitioning column. It specifies that the value will be specified by a dynamic-partitioning function using a defined system input specified by the `directSetter` attribute. Without this attribute, the value passed to a `functionPartitionInput` is taken from a stored column, but this allows it to be passed to the function without linking it to another column's data. For example, if `DataType` was the partitioning column, a system input with the name `dataType` could be passed to the dynamic partition function as follows: |

```xml
<SystemInput name="dataType" type="String"/>
<Column name="DataType"
        functionPartitionInput="true"
        directSetter="dataType"/>
```

If a logger depends on a customer's classes, those classes must be present in the classpath both for logger generation (as it is required for compilation) and at runtime.

If a logger uses column partition values that are calculated for each row, it will only allow a limited number of open partitions at any time. If this number is exceeded, the least-recently-used partition will be closed before the new one is opened. If a large number of partitions are expected to be written to at any time, the default number of open partitions may not be sufficient, as quick rollover of partitions will cause the creation of a lot of new files or network connections. These values can be configured with the following properties:

- `MultiPartitionFileManager.maxOpenColumnPartitions` - this is for loggers writing directly to binary log files. The default value is 10.
- `logAggregatorService.maxOpenColumnPartitions` - this is for loggers writing through the Log Aggregator Service. The default value is 10.

The following is an example logger for the
[basic example table](../../data-guide/tables-and-schemas.md#basic-table-attributes) in the
[Table and Schemas](../../data-guide/tables-and-schemas.md) documentation.

```xml
<Logger logFormat="1" loggerPackage="io.deephaven.example.gen">
    <SystemInput name="Alpha" type="String"/>
    <SystemInput name="BravoInt" type="int"/>
    <SystemInput name="Charlie" type="int"/>
    <SystemInput name="DeltaString" type="String"/>

    <Column name="Alpha"/>
    <Column name="Bravo" intradaySetter="(byte) BravoInt" directSetter="matchIntraday"/>
    <Column name="Charlie" intradaySetter="Charlie + 1" directSetter="matchIntraday"/>
    <Column name="Delta" intradaySetter="Double.parseDouble(DeltaString)" directSetter="matchIntraday"/>
    <!-- This column exists in the table schema, but not in the V1 log.  Missing columns are not allowed; therefore it must have an intradayType of none. -->
    <Column name="Echo" intradayType="none"/>
</Logger>
```

Note the use of `intradaySetter` expressions to modify the logged data. These transformations are applied in the logging
application when the client code calls the `log()`. When an `intradaySetter` is specified, the output data in the binary
log file will reflect the data transformed via the setter expression, rather than the raw input passed to the `log()`
method.

This example logger also excludes the `Echo` column; a default value will have to be populated via the `dbSetter` in the
listener. See [object serialization with autoblob](#object-serialization-with-autoblob) for an example of logging
arbitrary Java objects (such as `java.util.List`).

### Listeners in Deephaven Schemas

Listeners are defined in a `<Listener>` element.

A listener element has the following attributes:

| Element Name      | Optional/Required | Default                                                   | Description                                                                                                                                                                                                                                          |
| ----------------- | ----------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `logFormat`       | Optional          | `0`                                                       | Specifies the logger's version number.                                                                                                                                                                                                               |
| `listenerPackage` | Optional          | `<SchemaConfig.defaultListenerPackagePrefix>.<namespace>` | Specifies the Java package for the generated code. Defaults to the value of the `SchemaConfig.defaultListenerPackagePrefix` configuration property with the namespace converted to lowercase and appended (e.g., `com.mycompany.testnamespace`).     |
| `listenerClass`   | Optional          | Generated from `logFormat` and table `name`               | Specifies the Java class name for the generated code. Defaults to a value that includes the table name and log format version (if non-zero); e.g., `TestTableListener`, `TestTableFormat2Listener`. The class name must include the word "Listener". |

#### Listener code generation elements

Listener code generation is supplemented by three elements:

- `<ImportState>` - Optional – Specifies a state object used for producing validation inputs, with three attributes:
  - `importStateType` - Required - a full class name, must implement the `com.illumon.iris.db.tables.dataimport.logtailer.ImportStateinterface`.
  - `importStateConstructorArgs` - Optional - a comma-separated list of arguments to pass to the constructor of the `importStateType` class. If not specified, the default constructor will be used.
  - `stateUpdateCall` - required - a code fragment used to update the import state per row which may reference column names or fields; e.g., `newRow(Bravo)`.

| Element Name        | Optional/Required | Description                                                                                                                                                                                                                    |
| ------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `<ListenerImports>` | Optional          | Specifies a list of extra Java import statements for the generated listener class.                                                                                                                                             |
| `<ListenerFields>`  | Optional          | Specifies a free-form block of code used to declare and initialize instance or static members of the generated listener class.                                                                                                 |
| `<ImportState>`     | Optional          | Specifies a state object used for tracking additional state associated with the import (such as the position of the latest ingested row for each value in a key column). See [import states](#import-states) for more details. |

#### Listener `<Column>` elements

Each `<Listener>` element contains an ordered list of `<Column>` elements. The `<Column>` elements declare both the columns expected in the data from the logger and the columns to write to the table. The `<Column>` elements for a listener support the following attributes:

- `name` - Required – The name of the column. A column of this name does not necessarily need to exist in the table itself.
- `dbSetter` - Optional, unless `intradayType` is `none` – A Java expression to produce the value to be used in this column. This can customize how a raw value from the binary log file is interpreted when writing into a Deephaven column. The expression may use any fields or column names as variables, except columns for which `intradayType` is `none`.
- `intradayType` - Optional (defaults to `dataType`) – The data type of this column as written by the logger. Use none if a column is present in the table but not in the logger's output - if this is the case, a `dbSetter` attribute is required. This attribute is only required when the logger uses a different data type than the table itself.

#### DBDateTime

One special case is the DateTime or DBDateTime data type. (DateTime is an alias for Deephaven's DBDateTime type, which
is a datetime with nanosecond precision.) It is stored internally as nanoseconds from epoch. However, by default,
listener code will assume that a DBDateTime is logged as a long value in milliseconds from epoch (such as from
Java's `System.currentTimeMillis()` method). If the value provided by the logger is something other than milliseconds
from epoch, a custom setter must be specified in the dbSetter attribute.

For example:

`dbSetter="com.illumon.iris.db.tables.utils.DBTimeUtils.nanosToTime(LoggedTimeNanos)"`

In this case, the logging application is providing a long value of nanoseconds from epoch using the column name of LoggedTimeNanos.

The following is an example listener for the
[basic example table](../../data-guide/tables-and-schemas.md#basic-table-attributes) in the
[Table and Schemas](../../data-guide/tables-and-schemas.md) documentation:

```xml
<Listener logFormat="1" listenerPackage="io.deephaven.example.gen">
    <Column name="Alpha"/>
    <Column name="Bravo"/>
    <Column name="Charlie"/>
    <Column name="Delta"/>
    <Column name="Echo" intradayType="none" dbSetter="java.util.Collections.emptyList()"/>
</Listener>
```

### Combined Definition of Loggers and Listeners

It is possible to declare a Logger and Listener simultaneously in a `<LoggerListener>` element. A `<LoggerListener>` element requires both a `listenerPackage` attribute (assuming the default listener package name is not to be used) and a `loggerPackage` attribute. The `<Column>` elements declared under the `<LoggerListener>` element will be used for both the logger and the listener. This is useful as it avoids repetition of the `<Column>` elements.

An example of a `<LoggerListener>` declaration for the example table is provided below.

```xml
<LoggerListener logFormat="1"
                loggerPackage="io.deephaven.example.gen"
                listenerPackage="io.deephaven.example.gen">
    <SystemInput name="Alpha" type="String"/>
    <SystemInput name="BravoInt" type="byte"/>
    <SystemInput name="Charlie" type="int"/>
    <SystemInput name="DeltaString" type="String"/>

    <Column name="Alpha" />
    <Column name="Bravo" intradaySetter="(byte) BravoInt" directSetter="matchIntraday"/>
    <Column name="Charlie" intradaySetter="Charlie + 1" directSetter="matchIntraday" />
    <Column name="Delta" intradaySetter="Double.parseDouble(DeltaString)" directSetter="matchIntraday"/>
    <!-- This column exists in the table schema, but not in the V1 log.  Missing columns are not allowed; therefore it must have an intradayType of none. -->
    <Column name="Echo" intradayType="none" dbSetter="java.util.Collections.emptyList()" />
</LoggerListener>
```

### Combined Definitions of Table, Loggers, and Listeners

In past Deephaven releases, some schemas controlled code generation from the `<Column>` elements declared for the `<Table>`, without explicitly declaring the columns under a `<Logger>`, `<Listener>`, or `<LoggerListener>` attribute. The attributes normally defined on a `<Column>` element within `<Logger>`, `<Listener>`, or `<LoggerListener>` block were placed on the same `<Column>` elements used to define the table itself. This type of schema has been deprecated, as it does not allow for multiple logger or listener versions, or for maintaining backward compatibility during schema modifications.

### Import States

Import states are extensions to the data ingestion infrastructure that allow additional state to be recorded as the
data is ingested.

See: [Configuring import-driven lastBy queries](../../legacy/importing-data/fast-lastby.md) and [Indexing table schema](../../sys-admin/table-storage/partition-indexing.md#index-table-schema).

| Attribute         | Description                                                                                                                                                                                                                                  |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `importStateType` | The [`ImportState`](https://docs.deephaven.io/javadoc/2026.01/com/illumon/iris/db/tables/dataimport/logtailer/ImportState.html) implementation to create.                                                                                    |
| `stateUpdateCall` | The code to call for each new row that is ingested. Any columns available in the log file (i.e., columns whose `intradayType` is not `"none"`) may be used as variables in this expression (as with the `dbSetter` attribute for listeners). |

```xml
<Listener logFormat="1" listenerPackage="io.deephaven.example.gen" listenerClass="ImportStateExampleTableFormat1Listener">
    <ImportState importStateType="com.illumon.iris.db.tables.dataimport.logtailer.ImportStateRowCounter"
                 stateUpdateCall="newRow()"/>
    <Column name="Alpha"/>
    <Column name="Bravo"/>
    <Column name="Charlie"/>
    <Column name="Delta"/>
    <Column name="Echo" intradayType="none" dbSetter="java.util.Collections.emptyList()"/>
</Listener>
```

### Generate loggers

#### Introduction

Streaming data ingestion by Deephaven requires a table structure to receive the data, generated logger classes to format data and write it to binary log files, generated listeners used by Deephaven for ingestion, and processes to receive and translate the data. The main components are:

- **Schemas** — XML used to define table structure for various types of tables. These are described in detail in Schemas.
- **Loggers** — Generated class code that will be used by the application to write the data to a binary log, or to the Log Aggregator Service.
- **Log Aggregator Service (LAS)** — A Deephaven process that combines binary log entries from several processes and writes them into binary log files. The loggers may instead write directly to log files without using the LAS.
- **Tailer** — A Deephaven process that reads the binary log files as they are written and streams the data to a Data Import Server (DIS).
- [**Data Import Server (DIS)**](../dis.md) — A Deephaven process that efficiently writes streaming data to disk while distributing it over the network to downstream clients (such as queries).
- **Listeners** — Generated Java classes used in the DIS to convert the binary entries sent by the tailer into the appropriate columnar format for the target intraday table.

The logger and the listener work together, and both are generated based on the table's schema. The logger converts
elements of discrete data into the Deephaven row-oriented binary log format, while the listener converts the data from
the binary log format to Deephaven's column-oriented data store.

Although the typical arrangement is to stream events through the logger, tailer, and listener into the intraday tables as they arrive, there are also applications where only the logger is used, and the binary log files are manually imported when needed. The corresponding listener is then used when that later import is run.

Customers can generate custom loggers based on the definitions contained in schemas, and this will be
required to use the streaming data ingestion described above. Logger generation is normally done through the use of
the `generate_loggers` script, provided as part of the software installation. Listeners are generated dynamically when
needed.

When Deephaven loads a logger class, it will first verify that the class matches the current schema for the table in
question. An error will occur if the logger class is not compatible with the current schema — for example, if it was
generated based on an earlier version of the schema. Accordingly, whenever a table schema is modified and redeployed
(e.g., with the `dhconfig schema import` command or the [schema editor](../../legacy/importing-data/schemas/schema-editor.md)), any related
loggers must also be recreated.

#### `generate_loggers` Script

Once the schemas are defined, the `generate_loggers` script will normally be used to generate logger classes. It finds schemas, generates and compiles Java files based on the definitions in these schemas, and packages the compiled .class files and Java source files into two separate JAR files which can be used by the application and Deephaven processes, or by the customer for application development. The `IntradayLoggerFactory` class is called to perform the actual code generation, and it uses the properties described above when generating the logger and listener code.

To use it with default behavior, simply call the script without any parameters, and it will generate the loggers for any customer schemas it finds through the schema service. It will also generate listener classes and compile them to ensure that schemas are valid, but it won't save the generated listener code.

The simplest way to call this script is:

```bash
sudo /usr/illumon/latest/bin/iris generate_loggers
```

This call will generate all loggers that are not internal Deephaven ones, and place a default jar file in a location where it will be accessible to the application.

The script will use several default options based on environment variables in the host configuration file's `generate_loggers` entry.

- `ILLUMON_JAVA_GENERATION_DIR` - the directory into which the generated Java files will be placed. If this is not supplied, then the directory `$WORKSPACE/generated_java` will be used. This can also be overridden with the `javaDir` parameter as explained below. Two directories will be created under this directory:
  - `build_generated` - used to generate the compiled Java class files.
  - `java` - used to hold the generated Java code.

A typical value for this is: `export ILLUMON_JAVA_GENERATION_DIR=/etc/sysconfig/illumon.d/resources`:

- `ILLUMON_CONFIG_ROOT` indicates the customer configuration root directory. If defined, the script copies the generated logger/listener JAR file to the java_lib directory under this. A typical value for this is: `export ILLUMON_CONFIG_ROOT=/etc/sysconfig/illumon.d`.
- `ILLUMON_JAR_DIR` - the directory in which the generated JAR files will be created. If it is not defined, then the workspace directory will be used. This is not the final location of the generated JAR file that contains the compiled .class files, as it is copied based on the `ILLUMON_CONFIG_ROOT` environment variable. The JAR file that contains the logger Java sources is not copied anywhere.

Several options are available to provide flexibility in logger/listener generation. For example, a user could generate loggers and listeners from non-deployed schema files for use in application development:

- `outputJar` - specifies the filename of the JAR file generated by the logger/listener script. If the parameter is not provided, the default JAR file name is `IllumonCustomerGeneratedCode.jar`.
- `packages` - a comma-delimited list which restricts which packages will be generated. If a logger or listener package doesn't start with one of the specified names, generation will be skipped for that logger or listener. If the parameter is not provided, all loggers and listeners for found schema files will be generated. Customer logger and listener packages should never start with `com.illumon.iris.controller`, `com.illumon.iris.db`, or `io.deephaven.iris.db`, as these are reserved for internal use.
- `javaDir` - specifies the directory which will be used to write generated Java files. A logs directory must be available one level up from the specified `javaDir` directory. If the parameter is not provided, the directory `generated_java` under the workspace will be used. In either case, under this directory a subdirectory `build_generated` will be used to contain compiled .class files, and this subdirectory will be created if it does not exist.
- `schemaDir` - specifies a single directory to search for schema files. The specified location is searched for schema files. If this parameter is provided, the schema service is not used to retrieve schemas; instead, only the location specified by this parameter is used. Especially combined with `jarDir`, this may be useful to test logger and listener generation before deploying a schema and the associated loggers and listeners. Note: This directory must be readable by the user that will run the command, usually `irisadmin`.
- `jarDir` - specifies a directory to hold the generated JAR file. If the parameter is not provided, then the workspace directory (as defined in the host configuration file) will be used. The generated JAR file will always be copied to a location specified by `$ILLUMON_CONFIG_ROOT/java_lib`, which defaults to a location where the Deephaven application will find it (currently `/etc/sysconfig/illumon.d/java_lib)`.
- `javaJar` - specifies a JAR file in which the generated logger source (java) files will be placed. This will be placed in the same directory as the generated JAR file. If the parameter is not specified, then a JAR file with the name "IllumonCustomerGeneratedCodeSources.jar" will be created.
- `listenerCreationOptions` - by default, the `generate_loggers` script generates and compiles listener classes to verify that they can be created from the schemas, but does not write the class code. This behavior can be overridden.

  - `listenerCreationOptions=COMPILE_ONLY` - creates the listener classes and try to compiles them but does not write Java files (this is the default).

  - `listenerCreationOptions=CREATE_JAVA_FILES` - creates the listener java files.

  - `listenerCreationOptions=SKIP` - does not attempt to create or compile listener classes.

For example, the following command will generate a logger/listener jar using several custom options:

```bash
export WORKSPACE=/home/username/workspace
export DH_JAVA=/usr/bin/java (or any other path to a valid Java executable)
export JAVA_HOME=/usr/java/jdk[version] (or wherever bin/java and bin/javac can be found)
/usr/illumon/latest/bin/generate_loggers \
-d /usr/illumon/latest/ \
-f iris-common.prop \
-j -Dlogroot=/home/username/logs \
outputJar=test.jar \
packages=com.customer.gen \
javaDir=/home/username/java_gen_dir \
schemaDir=/home/username/schema \
jarDir=/home/username/jars
```

- The `WORKSPACE` environment variable will be used to store temporary files (in this case, `/home/username/workspace`, which must already exist).
- The `DH_JAVA` environment variable tells it where to find Java.
- The `JAVA_HOME` environment variable tells it where to find the JDK.
- `-d /usr/illumon/latest/` indicates where Deephaven is installed.
- `-f iris-common.prop` indicates the property file to use (this is a common property file that will exist on most installations).
- `-j -Dlogroot=/home/username/logs` indicates the root logging directory; logs will be placed in subdirectories under this directory, which should already exist.
- The `schemaDir` parameter tells it to look for schema files in the directory `/home/username/schema`, instead of using the schema service.
- The `outputJar` parameter tells it to generate a JAR with the name "test.jar".
- The `packages` parameter tells it to only generate classes that start with packages `com.customer.gen`.
- Because it is only operating out of the user's directories it can be run under the user's account.

Note: in this example we are calling `generate_loggers` directly, rather than through `/usr/illumon/latest/bin/iris`.

If the generation process completes correctly, the script will show which Java files it generated and indicate what was added to the JAR files. The final output should be similar to the following:

```bash
**********************************************************************
Jar file generated: /db/TempFiles/irisadmin/IllumonCustomerGeneratedCode.jar
Jar file not copied to a Deephaven-available location
Jar file with logger sources created as /db/TempFiles/irisadmin/IllumonCustomerGeneratedCodeSources.jar
********************
```

#### Generate loggers for local development

When developing locally (i.e., building or running on your desktop/laptop), Deephaven's Gradle plugin can automatically
generate logger classes based on locally-available schema files. This allows code to be written and compiled based on
the columns in local versions of the schema files, rather than the schemas deployed to the server. Typically, the schema
files are stored in git alongside the code that depends on them. The locally-generated loggers are used during
compilation and local execution, but are not packaged into `.jar` files that are deployed on the server, since
the server will already have versions of the loggers generated from the `generate_loggers` script.

Please see the section on [local query development](../../resources/how-to/local-query-development.md) for more details.

### Column codecs in loggers and listeners

If a [column codec](../../data-guide/tables-and-schemas.md#column-codecs) is used when reading a column,
the codec generally should be integrated into the logger and listener as well. This requires three changes to both
the logger and the listener:

1. Importing the codec class into the logger and listener in the [`LoggerImports`](#logger-code-generation-elements)
   and [`ListenerImports`](#listener-code-generation-elements) elements.
2. Instantiating the codec class in the `LoggerFields` and `ListenerFields` elements. Note that codecs may take
   constructor arguments that influence their behavior (such as configuring caching or buffer sizes).
3. Updating the `intradaySetter`/`directSetter` for the column in the logger, and the `dbSetter` in the listener, to
   call the codec's [`encode()`](https://docs.deephaven.io/javadoc/20231218/com/illumon/util/codec/ObjectCodec.html#encode(TYPE))/[`decode()`](https://docs.deephaven.io/javadoc/20231218/com/illumon/util/codec/ObjectDecoder.html#decode(byte%5B%5D,int,int))
   methods when writing/reading the data.

Additionally, the logger's `intradayType` for the column must be set to "Blob" to set the column's expected data type in
the [binary log file](../binary-log-format.md#column-definition-record).

For example, the following schema uses codecs to efficiently store byte array, `BigDecimal`, and `BigInteger` objects
(the `Foxtrot`, `Echo`, and `Golf` columns).

```xml
<Table namespace="ExampleNamespace" name="CodecsTestTable" storageType="NestedPartitionedOnDisk">
    <Partitions keyFormula="__PARTITION_AUTOBALANCE_SINGLE__"/>
    <Column name="Date" dataType="String" columnType="Partitioning"/>
    <Column name="Alpha" dataType="String"/>
    <Column name="Bravo" dataType="byte"/>
    <Column name="Charlie" dataType="int"/>
    <Column name="Delta" dataType="double" />
    <Column name="Echo" dataType="java.util.List" />
    <Column name="Foxtrot" dataType="java.math.BigDecimal" objectCodec="com.illumon.util.codec.BigDecimalCodec" objectCodecArguments="13,7"/>
    <Column name="Golf" dataType="java.math.BigInteger" objectCodec="com.illumon.util.codec.BigIntegerCodec" objectCodecArguments="13"/>
    <Column name="Hotel" dataType="byte[]" objectCode="com.illumon.util.codec.ByteArrayCodec" objectCodecArguments="32,notnull"/>

    <LoggerListener logFormat="1"
                    loggerPackage="io.deephaven.example.gen" loggerClass="CodecsTestTableLogger"
                    listenerPackage="io.deephaven.example.gen" listenerClass="CodecsTestTableListener">
        <SystemInput name="Alpha" type="String"/>
        <SystemInput name="Bravo" type="byte"/>
        <SystemInput name="Charlie" type="int"/>
        <SystemInput name="Delta" type="double"/>
        <SystemInput name="Foxtrot" type="java.math.BigDecimal"/>
        <SystemInput name="Golf" type="java.math.BigInteger"/>
        <SystemInput name="Hotel" type="byte[]"/>

        <LoggerImports>
            import java.nio.ByteBuffer;
            import com.illumon.util.codec.BigDecimalCodec;
            import com.illumon.util.codec.BigIntegerCodec;
            import com.illumon.util.codec.ByteArrayCodec;
        </LoggerImports>
        <LoggerFields>
            BigDecimalCodec bigDecimalCodec = new BigDecimalCodec("13, 7");
            BigIntegerCodec bigIntegerCodec = new BigIntegerCodec("13");
            ByteArrayCodec byteArrayCodec = new ByteArrayCodec("32, notnull");
        </LoggerFields>

        <ListenerImports>
            import com.illumon.util.codec.BigDecimalCodec;
            import com.illumon.util.codec.BigIntegerCodec;
            import com.illumon.util.codec.ByteArrayCodec;
        </ListenerImports>
        <ListenerFields>
            BigDecimalCodec bigDecimalCodec = new BigDecimalCodec("13, 7");
            BigIntegerCodec bigIntegerCodec = new BigIntegerCodec("13");
            ByteArrayCodec byteArrayCodec = new ByteArrayCodec("32, notnull");
        </ListenerFields>

        <Column name="Alpha"/>
        <Column name="Bravo"/>
        <Column name="Charlie"/>
        <Column name="Delta" />
        <Column name="Echo" intradayType="none" dbSetter="null" />
        <Column name="Foxtrot" intradayType="Blob" intradaySetter="ByteBuffer.wrap(bigDecimalCodec.encode(Foxtrot))" directSetter="Foxtrot" dbSetter="bigDecimalCodec.decode(Foxtrot)" />
        <Column name="Golf" intradayType="Blob" intradaySetter="ByteBuffer.wrap(bigIntegerCodec.encode(Golf))" directSetter="Golf" dbSetter="bigIntegerCodec.decode(Golf)" />
        <Column name="Hotel" intradayType="Blob" intradaySetter="ByteBuffer.wrap(byteArrayCodec.encode(Hotel))" directSetter="Hotel" dbSetter="byteArrayCodec.decode(Hotel)" />
    </LoggerListener>
</Table>
```

### Object serialization with autoblob

In addition to custom column codecs, Deephaven loggers/listeners can handle any serializable Java object by enabling
**autoblob**. Autoblob is convenient for Java developers, but Java serialization can be an expensive operation, so autoblob is inappropriate for high-throughput logging. Additionally, your listener is limited to ingesting from Java applications.

Autoblob serialization is controlled by setting the following properties:

| Attribute Name     | Optional/Required | Default         | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| ------------------ | ----------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `autoBlobInitSize` | Optional          | `-1` (disabled) | The initial size to use for the serialization buffer. Setting this property enables `autoblob`. This should be large enough to fit the expected serialized size of the data. Setting this value too low will impact serialization performance, as data must be copied when the buffer is increased to a larger size. Setting this value too high may unnecessarily increase memory usage and the allocation rate, which may increase the JVM's garbage collector (GC) activity. |
| `autoBlobMaxSize`  | Optional          |                 | The maximum size the serialization buffer can grow to. If the serialized form of an object exceeds `autoBlobMaxSize` bytes, the logger will throw an exception.                                                                                                                                                                                                                                                                                                                 |

```xml
<Table namespace="ExampleNamespace" name="AutoblobTestTable" storageType="NestedPartitionedOnDisk">
    <Partitions keyFormula="__PARTITION_AUTOBALANCE_SINGLE__"/>
    <Column name="Date" dataType="String" columnType="Partitioning"/>
    <Column name="Alpha" dataType="String"/>
    <Column name="Bravo" dataType="byte"/>
    <Column name="Charlie" dataType="int"/>
    <Column name="Delta" dataType="double"/>
    <Column name="Echo" dataType="java.util.List"/>

    <LoggerListener logFormat="1"
                    loggerPackage="io.deephaven.example.gen" loggerClass="AutoblobTestTableLogger"
                    listenerPackage="io.deephaven.example.gen" listenerClass="AutoblobTestTableListener" >
        <SystemInput name="Alpha" type="String"/>
        <SystemInput name="Bravo" type="byte"/>
        <SystemInput name="Charlie" type="int"/>
        <SystemInput name="Delta" type="double"/>
        <SystemInput name="Echo" type="java.util.List"/>

        <Column name="Alpha"/>
        <Column name="Bravo"/>
        <Column name="Charlie"/>
        <Column name="Delta"/>
        <Column name="Echo" intradayType="Blob" autoBlob="true" autoBlobInitSize="256" autoBlobMaxSize="16384"/>
    </LoggerListener>
</Table>
```

## Example

See the [streaming binary logs crash course](../../crash-course/data-in/streaming-binlogs.md) for an example.

## Appendix

### Appendix A: Column rename example

This section provides an example on how to rename a column between the application log file and the table schema.
The table below has three columns (`Date`, `Destination` and `SameName`) while The logger has two columns (`SameName` and `Source`).

- `Date` column is not present in the log file, but rather determined by the logging process.
- `SameName` column is in both the log file and table schema, and does not need to be transformed.
- `Source` column in the logger is renamed as `Destination` in the table.

To rename `Source` column as `Destination`, the `Listener` class should include both `Source` and `Destination` columns and their attributes should be:

- Source: A value of `none` for `dBSetter` attribute. This indicates that the column is not present in the table. Additionally, the attribute `intradayType` should be set to the appropriate dataType.
- Destination: A value of `Source` for `dBSetter` to identify its input source. A value `none` for `intradayType` means it is not present in the log file, and cannot be used as part of a `dbSetter`.

An example schema with only a `Listener` class defined is below:

```xml
<Table namespace="ExampleNamespace" name="RenameColumn" storageType="NestedPartitionedOnDisk" >
  <Partitions keyFormula="${autobalance_single}"/>
  <Column name="Date" dataType="String" columnType="Partitioning" />
  <Column name="Destination" dataType="int" columnType="Normal" />
  <Column name="SameName" dataType="int" columnType="Normal" />

  <Listener logFormat="1" listenerPackage="com.illumon.iris.test.gen">
    <Column name="Destination" intradayType="none" dbSetter="Source" />
    <Column name="SameName" dataType="int" />
    <Column name="Source" intradayType="int" dbSetter="none" />
  </Listener>

</Table>
```

For an example schema that includes `Listener` and `Logger` classes, see [Appendix B](#appendix-b-example-schema-to-rename-column)

### Appendix B: Example schema to rename column

```xml
<Table namespace="ExampleNamespace" name="RenameColumn" storageType="NestedPartitionedOnDisk">
    <Partitions keyFormula="${autobalance_single}"/>
    <Column name="Date" dataType="String" columnType="Partitioning"/>
    <Column name="Destination" dataType="int" columnType="Normal"/>
    <Column name="SameName" dataType="int" columnType="Normal"/>

    <Logger logFormat="1" loggerPackage="io.deephaven.example.gen" loggerClass="TestLoggerFormat1RenameColumn">
        <SystemInput name="SameName" type="int"/>
        <SystemInput name="Source" type="int"/>

        <Column name="SameName"/>
        <Column name="Destination" intradayType="none"/>
        <Column name="Source"/>
    </Logger>

    <Listener logFormat="1" listenerPackage="io.deephaven.example.gen">
        <Column name="Destination" intradayType="none" dbSetter="Source"/>
        <Column name="SameName"/>
        <Column name="Source" intradayType="int" dbSetter="none"/>
    </Listener>
</Table>
```

### Appendix C: Example custom logger interface

Below is an example [logger interface](#logger-interfaces):

```java
package io.deephaven.example;

import java.io.IOException;

public class ExampleLoggerInterface {

    void log(String alpha, int bravo, String charlie) throws IOException;
}
```

## Related documentation

- [Binary log format configuration](./binary-log-format-configuration.md)
- [Binary log format](../binary-log-format.md)