Extend Deephaven with custom plugins

What is a plugin?

Plugins are packages that allow you to extend the functionality of your Deephaven installation. In addition to code, plugins may include Deephaven schema and property files, as well as custom monit processes. For example, the Solace integration is a plugin. This guide will show you how to create and deploy your own plugin.

Configure your plugin with Gradle

Refer to the local query development guide for details about how to use Deephaven jars in your development environment. This example also uses IntelliJ IDEA and the Gradle build tool.

Note

Plugin tools supports Gradle 7.X

This is an example build.gradle for a Deephaven plugin. In the following code, we reference the Deephaven Artifactory repository, but your organization may not permit directly connecting to external repositories or have an internal mirror.

import io.deephaven.gradle.tools.Deps
import io.deephaven.gradle.tools.Repos

buildscript {
    repositories {
        maven {
            credentials {
                username = artifactoryUser ?: System.getProperty('user.name')
                password = artifactoryAPIKey ?: ''
            }
            url new URI('https://illumon.jfrog.io/illumon/libs-plugin-customer')
        }
        mavenLocal()
    }
    dependencies {
        classpath 'io.deephaven.gradle:plugins:1.20221001.004'
    }
}

plugins {
    id 'java-library'
}

//Plugins.initialize project, 'data-gen'
apply plugin: 'io.deephaven.plugins'
apply plugin: 'idea'

repositories {
    mavenLocal()
    mavenCentral()
    maven {
        credentials {
            username = artifactoryUser ?: System.getProperty('user.name')
            password = artifactoryAPIKey ?: ''
        }
        url new URI('https://illumon.jfrog.io/illumon/libs-customer')
    }
}

deephaven {
    pluginName = 'example-plugin'
}

allprojects {
    p -> Repos.irisRepo(p)
}

Repos.irisRepo project

dependencies {
    implementation Deps.iris('DB')
}

The buildScript below must be at the top of the file. plugin-tools provides Gradle tools that enable you to build your plugin:

buildscript {
    repositories {
        maven {
            credentials {
                username = artifactoryUser ?: System.getProperty('user.name')
                password = artifactoryAPIKey ?: ''
            }
            url new URI('https://illumon.jfrog.io/illumon/plugin-tools')
        }
        mavenLocal()
    }
    dependencies {
        classpath 'io.deephaven.gradle:plugins:1.20220404.013'
    }
}

Apply the Gradle plugin io.deephaven.plugins and give your plugin a name:

apply plugin: 'io.deephaven.plugins'
deephaven {
    pluginName = 'example-plugin'
}

Add a Maven buildScript repository that allows the Gradle build to import Deephaven jars:

repositories {
    mavenLocal()
    mavenCentral()
    maven {
        credentials {
            username = artifactoryUser ?: System.getProperty('user.name')
            password = artifactoryAPIKey ?: ''
        }
        url new URI('https://illumon.jfrog.io/illumon/libs-customer')
    }
}

Repository	Description
`libs-customer`	Deephaven jars.
`plugin-tools`	Tools to build Deephaven plugins.

After defining pluginName, add:

import io.deephaven.gradle.tools.Repos

allprojects {
    p -> Repos.irisRepo(p)
}

You must add this after your pluginName and before you define any dependencies.

Edit your gradle.properties to specify the artifactoryUrl, irisVersion, and snapshotSource of your project. There are other optional properties available.

artifactoryUrl=https://illumon.jfrog.io/illumon

pluginVersion=1.0.0
irisVersion=1.20230511.262
snapshotSource=release/20230511

Configuration Property	Description
`pluginVersion`	The deployed version of the plugin.
`irisVersion`	The version of Deephaven to use.
`fishLibVersion`	The version of fishlib to use.
`artifactoryUrl`	The base URL for artifactory.
`artifactoryUser`,`artifactoryAPIKey`	Maven repository credentials. These are the same credentials used to access JFrog.
`libSource`	The full artifactory repo name to use; default is libs-develop. Used for iris artifact downloads.
`snapshotSource`	Used only if libSource is not defined. Functions the same as libSource, except the libs- prefix is supplied .
`libPluginSource`	The name of the artifactory repo for plugin-only artifact downloads. Defaults to libs-snapshot-plugins.

Finally, we define our plugin's dependencies. The Deps.iris closure will use the above configuration to resolve your dependencies. This example will only use the DB module.

import io.deephaven.gradle.tools.Deps

dependencies {
    Deps.iris('DB')
}

Building your plugin

Refresh the Gradle project, and you should see the tasks to package the project as an RPM or a tar file. The PlugInExample directory structure

./gradlew makeExample-pluginRpm
./gradlew makeExample-pluginTar

Deploying your plugin

Extract the RPM:

sudo -u irisadmin rpm -Uvh example-plugin-1.0.0-1.x86_64.rpm

or tar package:

cd /etc/sysconfig/deephaven/plugins
sudo -u irisadmin cp /tmp/Example-plugin-1.0.0-Manual.tgz .
sudo -u irisadmin tar xvf Example-plugin-1.0.0-Manual.tgz

Your files will be placed in /etc/sysconfig/deephaven/plugins/<plugin-name>/.

Plugin layout

A few directories will be created within the directory /etc/sysconfig/deephaven/plugins/<plugin-name>/.

/bin

The scripts for activating your plugin. Run the included /etc/sysconfig/deephaven/plugins/<plugin-name>/bin/activate.sh script to activate your plugin.

sudo -u irisadmin /etc/sysconfig/deephaven/plugins/example-plugin/bin/activate.sh
--- Valid options to pass to /etc/sysconfig/deephaven/plugins/example-plugin/bin/activate.sh are:
--- --CLASSPATH Install all classpath types onto processes run on this node
--- --PROPS Import all props into etcd (only needed on a single node)
--- --SCHEMA Install schemas to etcd (only needed on a single node)
--- --PROCESS Run all plugin processes on this nodes

/global

Every jar and directory in /etc/sysconfig/deephaven/plugins/<name>/global will be added to all Deephaven classpaths, except Core+ workers. You should not put files directly in the root global folder, and if you do put class files here, make sure the root package (such as com) is inside a directory (such as classes), and not placed directly into global.

By default, this location will include:

Any jar produced by the gradle project where you add apply plugin: 'io.deephaven.plugins'.
Any jar produced by any gradle project included in Deephaven { projects = [ ':my', ':plugin', ':projects' ] }.
Any archive added to dependencies { global 'some:artifact:id' } of the above gradle projects.
Everything found in src/main/global.

/local

The local directory has the exact same semantics as the global directory, except that entries here will only appear in plugin processes' classpaths.

By default, this location will include:

Any archive added to dependencies { local 'some:artifact:id@tar.gz' } (@tar.gz is not required).
Everything found in src/main/local (this location is configurable).

Note

By default, the extra jars and files will not be available to Core+ workers.

/schema

Contains all schemas that were included in the plugin at src/main/schema. Use the activate.sh script to deploy these schema to your cluster.

/processes

The processes directory is used like a filesystem map; each directory in processes is treated as the process name, and inside each of those directories, there are four files. By default, all of these files are generated for you, but you can override them by putting exact-named files in src/main/processes:

For a plugin named myplugin:

Configuration Property	Description
`myplugin.conf`	A monit conf file. By default, delegates to `/etc/init.d/iris start myplugin`.
`myplugin.sh`	A script that will be linked into `/usr/illumon/latest/bin/start_myplugin`. Necessary only if using `/etc/init.d/iris` to start your process.
`myplugin.hostconfig`	A hostconfig file. Necessary only if using `/etc/init.d/iris` to start your process.
`type`	A file with a value matching an enum name in ProcessType class. The default type is `LONG_RUNNING`, and is configured in Gradle.

Example plugin

This example calculates the average prices of stocks for a given day and logs these averages to the Deephaven table MarketData.AvgStockPrice.

We place MarketData.AvgStockPrice.schema at src/main/schema so that it will be included in our plugin. Notice that we specify the logger's package and interface.

<Table name="AvgStockPrice" namespace="MarketData" defaultMergeFormat="DeephavenV1" storageType="NestedPartitionedOnDisk">
    <Partitions keyFormula="${autobalance_by_first_grouping_column}" />

    <Column name="Date" dataType="String" columnType="Partitioning" />
    <Column name="symbol" dataType="String" />
    <Column name="price" dataType="Double" />

    <LoggerListener logFormat="0"
    loggerPackage="com.plugin.example.log"
    listenerPackage="com.plugin.example.log"
    loggerInterface="com.plugin.example.log.AvgStockPriceLoggerInterface"
    >

        <SystemInput name="symbol" type="java.lang.String" />
        <SystemInput name="price" type="double" />

        <Column name="symbol" />
        <Column name="price" />
    </LoggerListener>
</Table>

The AvgStockPriceLoggerInterface interface extends IntradayLogger. It must use the same package and class name as specified in the schema.

package com.plugin.example.log;

import com.illumon.intradaylogger.IntradayLogger;
import com.illumon.iris.db.tables.dataimport.TableLogger;
import com.illumon.iris.db.tables.utils.DBDateTime;

import java.io.IOException;

public interface AvgStockPriceLoggerInterface extends IntradayLogger {
    void log(String symbol, double price) throws IOException;
}

Here is the Java code which does the average price calculations and uses the AvgStockPriceLoggerInterface to log the results.

package com.plugin.example;

import com.illumon.iris.db.tables.Table;
import com.illumon.iris.db.tables.databases.Database;
import com.plugin.example.log.AvgStockPriceLoggerInterface;

import java.io.IOException;

public class QueryUtils {
    public static Table getAvgStockPriceTable(final Database db, final String date) {
        return db.t("MarketData", "Trades")
                .where("Date = `" + date + "`")
                .view("USym", "Last")
                .avgBy("USym");
    }

    public static void logAvgStockPriceTable(final Database db, final String date, final AvgStockPriceLoggerInterface avgStockPriceLogger) throws IOException {
        final Table avgStockPriceTable = getAvgStockPriceTable(db, date);
        final String[] uSyms = (String[]) avgStockPriceTable.getColumn("USym").getDirect();
        final double[] avgPrices = (double[]) avgStockPriceTable.getColumn("Last").getDirect();
        for(int i = 0; i < uSyms.length; i++) {
            avgStockPriceLogger.log(uSyms[i], avgPrices[i]);
        }
    }
}

After this plugin is extracted:

sudo rpm -Uvh example-plugin-1.0.0-1.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:example-plugin-0:1.0.0-1         ################################# [ 50%]
| --- Starting afterInstall.sh
| --- AUTO_PLUGIN=y was not passed; this plugin has not activated all of its components
| --- To activate the missing components, please run:
| --- /etc/sysconfig/deephaven/plugins/example-plugin/bin/activate.sh  --classpath  --props  --schema  --process

We activate it to place the Java code in the Deephaven classpath, deploy the schema, and generate the AvgStockPriceLogger.

sudo -u irisadmin /etc/sysconfig/deephaven/plugins/example-plugin/bin/activate.sh --all

The plugin's jar is placed on the classpath inside <plugin-name>/global. Reload the client update service in order to sync a client with the latest jar files.

ls -ltr /etc/sysconfig/illumon.d/plugins/example-plugin/
total 4
-rw-r--r--. 1 irisadmin dbmergegrp 40 Nov  9 15:48 installedSchemas
lrwxrwxrwx. 1 irisadmin dbmergegrp 54 Nov  9 16:10 global -> /etc/sysconfig/deephaven/plugins/example-plugin/global

We can now use the generated logger and our plugin jar in a Deephaven console:

import com.plugin.example.QueryUtils
import com.plugin.example.log.AvgStockPriceLogger
import com.illumon.intradaylogger.LongLivedProcessBinaryStoreWriterFactory

String namespace = "MarketData"
String tableName="AvgStockPrice"
String internalPartition = "vm1"
String date = currentDateNy()
String columnPartition = date

filePath = "/var/log/deephaven/binlogs/${namespace}.${tableName}.System.${internalPartition}.${columnPartition}.bin"

tableLogger = new AvgStockPriceLogger()
tableLogger.init(new LongLivedProcessBinaryStoreWriterFactory(filePath, log), 10000)

QueryUtils.logAvgStockPriceTable(db, date, tableLogger)

Note

Download

Example plugin project files