How to use IntradayLoggerFactory in a development environment

The IntradayLoggerFactory class is a Deephaven component used internally by the generate_loggers script. It is responsible for parsing table schemas to generate logger code for Java and C# loggers, and Java listener code for schemata that have Logger, Listener, or LoggerListener sections defined.

These classes are used to write and read data in Deephaven binary log files, predominantly as part of data streaming. Deephaven data streaming in general is covered in the Streaming Data overview.

Using IntradayLoggerFactory in a development environment, typically with an IDE such as IntelliJ Idea, makes it easier to develop and test the schema and logging process for a new data source. The other alternative is to deploy the schema to a Deephaven server, along with any dependency JARs, then run generate_loggers, and then copy the resultant JAR or classes back to the development system. Clearly, generating logger and listener classes locally on the development system is easier and faster and allows a more seamless integration with the development process.

Note

The example is configured for the Deephaven release 1.20190816.

The example project

ILF-Gradle is an IntelliJ Idea project containing example pieces and tasks to demonstrate Java logger class and application creation.

Note

Download Project archive

img

The project includes the following components:

  • SampleLogger.TickData.schema - This is a schema file for a simple table with a LoggerListener section.
  • TickDataLogFormat1Interface - This is a Java interface class that defines the properties to be logged by the TickData schema.
  • com.samplelogger.TestLog - This is a logger application that uses the logger class generated from the schema to write some sample data to a binary log file
  • dependencies.gradle - a gradle file that defines the base Deephaven DB module dependency to be downloaded from artifactory.
  • Gradle tasks (under the deephaven group):
    • compileInterface - This compiles the TickDataLogFormat1Interface class. This is a prerequisite to building or running any other components in the project.
    • intradayLoggerFactory - This executes IntradayLoggerFactory to read the schema and generate the logger and listener classes.
    • runTestLogger - This runs the example logger application, which will use the generated logger to write two records to a binary log file.
  • Run configurations (duplicates of gradle tasks, to show another way the classes can be used):
    • Generate LoggerListener - This executes IntradayLoggerFactory to read the schema file and generate the logger and listener classes.
    • Run Logger - This runs the example logger application, which will use the generated logger to write two records to a binary log file.

img

In addition, the gradle configuration for the project extends clean so that clean will also delete the generated logger and listener classes, so the clean task will restore the project to its initial state.

To use this project, it will likely be necessary to edit the dependencies.gradle file and enter/update credentials for the Deephaven artifactory as well as possibly changing the version of Deephaven to be referenced.

repositories {
   maven {
       credentials {
           username = artifactoryUser ?: System.getProperty('user.name')
           password = artifactoryAPIKey ?: "
       }
       url 'https://illumon.jfrog.io/illumon/libs-customer'
   }
  }

apply plugin: 'java'
apply plugin: 'base'

dependencies {
 version = '1.20190816.+'
 implementation 'iris:DB:' + version
 // Later releases only need DB, but pre-20190607 also requires configs
 implementation 'iris:configs:' + version
}

Use of IntradayLoggerFactory

com.illumon.iris.db.tables.dataimport.IntradayLoggerFactory is executed from its main() method. It takes no program arguments, but its behavior can be modified by setting JVM properties. If no special properties are set, it will expect to read all schemata from a default location and create loggers and listeners for all of them. In the example project, properties are set to limit it to processing a single local path for the schema and generating classes under the SampleLogger/java/ path.

These JVM properties are documented in the build.gradle file:

// iris-defaults.prop and needed jars come from the iris:DB entry in dependencies.gradle
systemProperty 'Configuration.rootFile', 'iris-defaults.prop'
systemProperty 'devroot', "$rootDir"

// use local schema files instead of the schema service (consideration for v1.20200928 and later)
systemProperty 'SchemaService.defaultImpl','com.illumon.iris.db.schema.LocalTransientSchemaService'
systemProperty 'SchemaService.Local.sourceDirectory','SampleLogger/schema/'
systemProperty 'SchemaConfig.resourcePath.irisInternal','SampleLogger/schema/'

// use local prop files instead of the configuration service (consideration for North Arapaho and later)
systemProperty 'com.fishlib.configuration.PropertyInputStreamLoader.override','com.fishlib.configuration.PropertyInputStreamLoaderTraditional'

// tee log messages to stdout
systemProperty 'LoggerFactory.teeOutput','true'

// set the directory name into which logger and listener classes should be generated
systemProperty 'IntradayLoggerFactory.javaGenerationDirectory','SampleLogger'

To use IntradayLoggerFactory for a different schema, the main changes needed would be to change SchemaService.Local.sourceDirectory and SchemaConfig.resourcePath.IrisInternal (similar properties with applicability to newer or older versions of Deephaven, respectively) to "point" to the location of the schema files. Of course, all classes referenced by the schema, including logger interfaces, would need to be available in the classpath and compiled as a prerequisite to running IntradayLoggerFactory.

Use of the examples in the project

Running the deephaven\runTestLogger gradle task will run all required tasks needed to get from a schema and logger application to actually logging data:

  1. Compile the TickDataLogFormat1Interface

  2. Run the intradayLoggerFactory task to generate the com.samplelogger.tickdata.gen.TickDataLogFormat1Logger class (under SampleLogger/java)

  3. Compile the com.samplelogger.tickdata.gen.TickDataLogFormat1Logger class

  4. Build and execute the com.samplelogger.TestLog class, which will then create a new binary log file and write two rows to it:

    Timestamp,Exchange,Security,Underlying,BidSize,BidPrice,AskSize,AskPrice,TradeStatus,TickCondition,ExchangeTimestamp,ExchangeTimeNanos

1576017365892,N,"AAPL_12321","AAPL",10,777.77002,10,888.880005,N,A,1000000000,1000000000000000

1576017365892,A,"GOOG_12321","GOOG",10,777.77002,10,888.880005,F,N,1576017365892,1576017365892000000

    This log file is generated with a file name using the following syntax:

    TickDataLog.bin.yyyy-MM-dd.hhmmss.nnn<GMT offset>

For example:

TickDataLog.bin.2019-12-10.153605.822-0700

In newer releases (v1.20190607 and later) the default path for the file is /var/log/deephaven/misc. In earlier releases, the local SampleLogger/working_directory_for_older_versions/logs directory will be used. The file name is controlled by the call to it in the TestLog.class. For the newer versions, the log path can be changed by setting the JVM property logroot, but the file will still be created under the misc directory under the defined logroot.

The individual gradle task can also be run separately. If run from the Idea run configurations, there is no chaining; so the logger interface would need to be explicitly compiled first, e.g., by right-clicking on it and selecting to Recompile it; IntradayLoggerFactory would have to be run to generate the logger class before the logger application itself could be run.

To run IntradayLoggerFactory from the command line, a set of Deephaven JARs would be needed somewhere where they could be passed to a classpath. Gradle downloads JARs to guid-named cache directories that are not very accessible for non-gradle command line use. The Deephaven Launcher or the Configuration Updater can be used to synchronize needed JARs and other resources from a Deephaven server to a client/development system.

Example

A sample IntradayLoggerFactory execution command line follows:

java -cp ~/Downloads/IllumonLauncher/TPlus/java_lib/*:~/Downloads/IllumonLauncher/TPlus/resources/:./SampleLogger/build/classes/java/main -server -Xmx4096m -DConfiguration.rootFile=iris-defaults.prop -DIntradayLoggerFactory.javaGenerationDirectory=SampleLogger -DSchemaService.defaultImpl=com.illumon.iris.db.schema.LocalTransientSchemaService -Dcom.fishlib.configuration.PropertyInputStreamLoader.override=com.fishlib.configuration.PropertyInputStreamLoaderTraditional -DSchemaService.Local.sourceDirectory=SampleLogger/schema/ -DSchemaConfig.resourcePath.irisInternal=SampleLogger/schema/ -DLoggerFactory.teeOutput=true com.illumon.iris.db.tables.dataimport.IntradayLoggerFactory

The classpath references java_lib and resources from a Deephaven instance (in this case, the instance is called TPlus).

The classpath also references SampleLogger/build/classes/java/main because this is where the class for the logger interface can be obtained.

-D is used to pass other JVM arguments needed to set IntradayLoggerFactory to read a single, local, schema directory, and to direct it to create generated classes under SampleLogger/schema/.