---
title: Extend Deephaven with custom plugins
---

## What is a plugin?

Plugins are packages that allow you to extend the functionality of your Deephaven installation. In addition to code, plugins may include Deephaven schema and property files, as well as custom monit processes. For example, the [Solace integration](../../data-guide/streaming/solace.md) is a plugin. This guide will show you how to create and deploy your own plugin.

## Configure your plugin with Gradle

Refer to the [local query development guide](../../resources/how-to/local-query-development.md) for details about how to
use Deephaven jars in your development environment. This example also uses IntelliJ IDEA and the Gradle build tool.

> [!NOTE]
> Plugin tools support Gradle 7.X.

This is an example `build.gradle` for a Deephaven plugin. In the following code, we reference the Deephaven Artifactory repository, but your organization may not permit directly connecting to external repositories or have an internal mirror.

```groovy
import io.deephaven.gradle.tools.Deps
import io.deephaven.gradle.tools.Repos

buildscript {
    repositories {
        maven {
            credentials {
                username = findProperty("artifactoryUser") ?: System.getProperty('user.name')
                password = findProperty("artifactoryAPIKey") ?: ""
            }
            url new URI('https://illumon.jfrog.io/illumon/libs-plugin-customer')
        }
        mavenLocal()
    }
    dependencies {
        classpath 'io.deephaven.gradle:plugins:1.20221001.004'
    }
}

plugins {
    id 'java-library'
}

//Plugins.initialize project, 'data-gen'
apply plugin: 'io.deephaven.plugins'
apply plugin: 'idea'

repositories {
    mavenLocal()
    mavenCentral()
    maven {
        credentials {
            username = findProperty("artifactoryUser") ?: System.getProperty('user.name')
            password = findProperty("artifactoryAPIKey") ?: ""
        }
        url new URI('https://illumon.jfrog.io/illumon/libs-customer')
    }
}

deephaven {
    pluginName = 'example-plugin'
}

allprojects {
    p -> Repos.irisRepo(p)
}

Repos.irisRepo project

dependencies {
    implementation Deps.iris('DB')
}
```

The `buildScript` above must be at the top of the file. `plugin-tools` provides Gradle tools that enable you to build your plugin:

```groovy
buildscript {
    repositories {
        maven {
            credentials {
                username = findProperty("artifactoryUser") ?: System.getProperty('user.name')
                password = findProperty("artifactoryAPIKey") ?: ""
            }
            url new URI('https://illumon.jfrog.io/illumon/plugin-tools')
        }
        mavenLocal()
    }
    dependencies {
        classpath 'io.deephaven.gradle:plugins:1.20220404.013'
    }
}
```

Apply the Gradle plugin `io.deephaven.plugins` and give your plugin a name:

```groovy
apply plugin: 'io.deephaven.plugins'
deephaven {
    pluginName = 'example-plugin'
}
```

Add a Maven repository that allows the Gradle build to import Deephaven jars:

```groovy
repositories {
    mavenLocal()
    mavenCentral()
    maven {
        credentials {
            username = findProperty("artifactoryUser") ?: System.getProperty('user.name')
            password = findProperty("artifactoryAPIKey") ?: ""
        }
        url new URI('https://illumon.jfrog.io/illumon/libs-customer')
    }
}
```

| Repository      | Description                       |
| --------------- | --------------------------------- |
| `libs-customer` | Deephaven jars.                   |
| `plugin-tools`  | Tools to build Deephaven plugins. |

After defining `pluginName`, add:

```groovy
import io.deephaven.gradle.tools.Repos

allprojects {
    p -> Repos.irisRepo(p)
}
```

You must add this after your `pluginName` and before you define any dependencies.

Edit your `gradle.properties` to specify the `artifactoryUrl`, `irisVersion`, and `snapshotSource` of your project. There are other optional properties available.

```text
artifactoryUrl=https://illumon.jfrog.io/illumon

pluginVersion=1.0.0
irisVersion=1.20230511.262
snapshotSource=release/20230511
```

| Configuration Property                | Description                                                                                                        |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `pluginVersion`                       | The deployed version of the plugin.                                                                                |
| `irisVersion`                         | The version of Deephaven to use.                                                                                   |
| `fishLibVersion`                      | The version of fishlib to use.                                                                                     |
| `artifactoryUrl`                      | The base URL for artifactory.                                                                                      |
| `artifactoryUser`,`artifactoryAPIKey` | Maven repository credentials. These are the same credentials used to access JFrog.                                 |
| `libSource`                           | The full artifactory repo name to use; default is libs-develop. Used for iris artifact downloads.                  |
| `snapshotSource`                      | Used only if `libSource` is not defined. Functions the same as `libSource`, except the `libs-` prefix is supplied. |
| `libPluginSource`                     | The name of the artifactory repo for plugin-only artifact downloads. Defaults to libs-snapshot-plugins.            |

Finally, we define our plugin's dependencies. The `Deps.iris` closure will use the above configuration to resolve your dependencies. This example will only use the `DB` module.

```groovy
import io.deephaven.gradle.tools.Deps

dependencies {
    implementation Deps.iris('DB')
}
```

## Building your plugin

Refresh the Gradle project, and you should see the tasks to package the project as an RPM or a tar file.
![The `PlugInExample` directory structure](../../assets/plugin.png)

```shell
./gradlew makeExample-pluginRpm
./gradlew makeExample-pluginTar
```

## Deploying your plugin

Extract the RPM:

```shell
sudo -u irisadmin rpm -Uvh example-plugin-1.0.0-1.x86_64.rpm
```

or tar package:

```shell
cd /etc/sysconfig/deephaven/plugins
sudo -u irisadmin cp /tmp/Example-plugin-1.0.0-Manual.tgz .
sudo -u irisadmin tar xvf Example-plugin-1.0.0-Manual.tgz
```

Your files will be placed in `/etc/sysconfig/deephaven/plugins/<plugin-name>/`.

### Plugin layout

A few directories will be created within the directory `/etc/sysconfig/deephaven/plugins/<plugin-name>/`.

#### /bin

The scripts for activating your plugin. Run the included `/etc/sysconfig/deephaven/plugins/<plugin-name>/bin/activate.sh` script to activate your plugin.

```text
sudo -u irisadmin /etc/sysconfig/deephaven/plugins/example-plugin/bin/activate.sh
--- Valid options to pass to /etc/sysconfig/deephaven/plugins/example-plugin/bin/activate.sh are:
--- --CLASSPATH Install all classpath types onto processes run on this node
--- --PROPS Import all props into etcd (only needed on a single node)
--- --SCHEMA Install schemas to etcd (only needed on a single node)
--- --PROCESS Run all plugin processes on this nodes
```

#### /global

Every jar and directory in `/etc/sysconfig/deephaven/plugins/<name>/global` will be added to all Deephaven classpaths, except Core+ workers. You should not put files directly in the root global folder, and if you do put class files here, make sure the root package (such as `com`) is inside a directory (such as `classes`), and not placed directly into `global`.

> [!NOTE]
> Additional directories are available for more targeted classpath inclusion:
>
> - `/auth` — Added to the authentication server classpath.
> - `/worker` — Added to server-side worker classpaths.
> - `/client` — Distributed to desktop clients via the Client Update System (CUS).
>
> For full details, see the [plugin tools README](https://github.com/deephaven-ent/iris/blob/rc/gplus/build-logic/plugins/README.md).

By default, this location will include:

1. Any jar produced by the gradle project where you add `apply plugin: 'io.deephaven.plugins'.`
2. Any jar produced by any gradle project included in `Deephaven { projects = [ ':my', ':plugin', ':projects' ] }.`
3. Any archive added to dependencies `{ global 'some:artifact:id' }` of the above gradle projects.
4. Everything found in `src/main/global`.

#### /local

The local directory has the exact same semantics as the global directory, except that entries here will only appear in plugin processes' classpaths.

By default, this location will include:

1. Any archive added to dependencies `{ local 'some:artifact:id@tar.gz' }` (`@tar.gz` is not required).
2. Everything found in `src/main/local` (this location is configurable).

> [!NOTE]
> By default, the extra jars and files will not be available to Core+ workers.

#### /schema

Contains all schemas that were included in the plugin at `src/main/schema`. Use the `activate.sh` script to deploy these schema to your cluster.

#### /processes

The processes directory is used like a filesystem map; each directory in processes is treated as the process name, and inside each of those directories, there are four files. By default, all of these files are generated for you, but you can override them by putting exact-named files in `src/main/processes`:

For a plugin named `myplugin`:

| Configuration Property | Description                                                                                                                                   |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `myplugin.conf`        | A monit conf file. By default, delegates to `/etc/init.d/iris start myplugin`.                                                                |
| `myplugin.sh`          | A script that will be linked into `/usr/illumon/latest/bin/start_myplugin`. Necessary only if using `/etc/init.d/iris` to start your process. |
| `myplugin.hostconfig`  | A hostconfig file. Necessary only if using `/etc/init.d/iris` to start your process.                                                          |
| `type`                 | A file with a value matching an enum name in ProcessType class. The default type is `LONG_RUNNING`, and is configured in Gradle.              |

### Example plugin

This example calculates the average prices of stocks for a given day and logs these averages to the Deephaven table `MarketData.AvgStockPrice`.

We place `MarketData.AvgStockPrice.schema` at `src/main/schema` so that it will be included in our plugin. Notice that we specify the logger's package and interface.

```xml
<Table name="AvgStockPrice" namespace="MarketData" defaultMergeFormat="DeephavenV1" storageType="NestedPartitionedOnDisk">
    <Partitions keyFormula="${autobalance_by_first_grouping_column}" />

    <Column name="Date" dataType="String" columnType="Partitioning" />
    <Column name="symbol" dataType="String" />
    <Column name="price" dataType="Double" />

    <LoggerListener logFormat="0"
    loggerPackage="com.plugin.example.log"
    listenerPackage="com.plugin.example.log"
    loggerInterface="com.plugin.example.log.AvgStockPriceLoggerInterface"
    >

        <SystemInput name="symbol" type="java.lang.String" />
        <SystemInput name="price" type="double" />

        <Column name="symbol" />
        <Column name="price" />
    </LoggerListener>
</Table>
```

The `AvgStockPriceLoggerInterface` interface extends `IntradayLogger`. It must use the same package and class name as specified in the schema.

```java
package com.plugin.example.log;

import com.illumon.intradaylogger.IntradayLogger;
import com.illumon.iris.db.tables.dataimport.TableLogger;
import com.illumon.iris.db.tables.utils.DBDateTime;

import java.io.IOException;

public interface AvgStockPriceLoggerInterface extends IntradayLogger {
    void log(String symbol, double price) throws IOException;
}
```

Here is the Java code which does the average price calculations and uses the `AvgStockPriceLoggerInterface` to log the results.

```java
package com.plugin.example;

import com.illumon.iris.db.tables.Table;
import com.illumon.iris.db.tables.databases.Database;
import com.plugin.example.log.AvgStockPriceLoggerInterface;

import java.io.IOException;

public class QueryUtils {
    public static Table getAvgStockPriceTable(final Database db, final String date) {
        return db.t("MarketData", "Trades")
                .where("Date = `" + date + "`")
                .view("USym", "Last")
                .avgBy("USym");
    }

    public static void logAvgStockPriceTable(final Database db, final String date, final AvgStockPriceLoggerInterface avgStockPriceLogger) throws IOException {
        final Table avgStockPriceTable = getAvgStockPriceTable(db, date);
        final String[] uSyms = (String[]) avgStockPriceTable.getColumn("USym").getDirect();
        final double[] avgPrices = (double[]) avgStockPriceTable.getColumn("Last").getDirect();
        for(int i = 0; i < uSyms.length; i++) {
            avgStockPriceLogger.log(uSyms[i], avgPrices[i]);
        }
    }
}
```

After this plugin is extracted:

```shell
sudo rpm -Uvh example-plugin-1.0.0-1.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:example-plugin-0:1.0.0-1         ################################# [ 50%]
| --- Starting afterInstall.sh
| --- AUTO_PLUGIN=y was not passed; this plugin has not activated all of its components
| --- To activate the missing components, please run:
| --- /etc/sysconfig/deephaven/plugins/example-plugin/bin/activate.sh  --classpath  --props  --schema  --process
```

We activate it to place the Java code in the Deephaven classpath, deploy the schema, and generate the `AvgStockPriceLogger`.

```shell
sudo -u irisadmin /etc/sysconfig/deephaven/plugins/example-plugin/bin/activate.sh --all
```

The plugin's jar is placed on the classpath inside `<plugin-name>/global`. Reload the client update service in order to sync a client with the latest jar files.

```shell
ls -ltr /etc/sysconfig/illumon.d/plugins/example-plugin/
total 4
-rw-r--r--. 1 irisadmin dbmergegrp 40 Nov  9 15:48 installedSchemas
lrwxrwxrwx. 1 irisadmin dbmergegrp 54 Nov  9 16:10 global -> /etc/sysconfig/deephaven/plugins/example-plugin/global
```

We can now use the generated logger and our plugin jar in a Deephaven console:

```groovy
import com.plugin.example.QueryUtils
import com.plugin.example.log.AvgStockPriceLogger
import com.illumon.intradaylogger.LongLivedProcessBinaryStoreWriterFactory

String namespace = "MarketData"
String tableName="AvgStockPrice"
String internalPartition = "vm1"
String date = currentDateNy()
String columnPartition = date

filePath = "/var/log/deephaven/binlogs/${namespace}.${tableName}.System.${internalPartition}.${columnPartition}.bin"

tableLogger = new AvgStockPriceLogger()
tableLogger.init(new LongLivedProcessBinaryStoreWriterFactory(filePath, log), 10000)

QueryUtils.logAvgStockPriceTable(db, date, tableLogger)
```

> [!NOTE]
> Download
>
> - [Example plugin project files](../../assets/PlugInExample.zip)

## Related documentation

- [Local query development](../../resources/how-to/local-query-development.md)
- [Streaming binary logs](../../data-guide/streaming/streaming-binary-logs.md)
- [Solace integration](../../data-guide/streaming/solace.md)