Create your own plugin

Python plugins are Python packages that extend the server's functionality, the UI, a client API, or a combination of these. They can be as simple as a single Python file or as complex as a multi-file package with dependencies. Plugins are packaged like any other Python package.

This guide walks you through creating a plugin in Deephaven. Specifically, it covers creating a bidirectional plugin that extends the capabilities of Deephaven's Python client API. A bidirectional plugin keeps a message stream open between the client and an object on the server, so the two sides can keep exchanging messages. This guide uses that stream to define custom remote procedure call (RPC) methods that clients call to work with an object on a running server and get objects back.

A bidirectional plugin like the one in this guide has a server-side component, implemented with the deephaven.plugin module, and a client-side component, implemented with the pydeephaven.experimental.plugin_client module.

Not every plugin needs a message stream. A fetch-only plugin sends its object to the client once, and the client can't send messages back. To build one in Python, subclass FetchOnlyObjectType and implement its to_bytes method instead. The Groovy version of this guide walks through a fetch-only plugin in Java or Groovy.

Create the plugin

Project layout

This example creates a plugin called ExampleService. You can call your plugin whatever you'd like.

Note

The directory name does not need to match the plugin name. This guide uses ExampleServicePlugin for the directory name, but the plugin itself is called ExampleService.

The plugin has server- and client-side components. Create the following directory and file structure in your project folder:

Note

This guide assumes the use of the pyproject.toml configuration for packaging. You can also use setup.cfg or setup.py if you prefer.

Server-side plugin

All server-side plugin code lives in the server folder. This folder contains a TOML file for packaging and a subfolder containing the actual plugin implementation.

Server plugin implementation

This example plugin has four parts, all of which go in server/example_plugin_server/__init__.py:

  • The object the plugin manages (ExampleService).
  • The message stream handler, which passes messages between the client and the server (ExampleServiceMessageStream).
  • The object type, which tells the server how to recognize and connect to the object (ExampleServicePlugin).
  • The registration class, which registers the object type with the server (ExampleServicePluginRegistration).

Let's start with the object the plugin manages. In this case, the service has two methods.

Next, define the message stream handler, a subclass of MessageStream that passes messages between the client and the server.

Then define the object type, a subclass of BidirectionalObjectType that ties the object to its message stream. It implements:

  • The name property, which returns the type name, ExampleService.
  • The is_type method, which tells the server whether an object is an ExampleService.
  • The create_client_connection method, which returns a new ExampleServiceMessageStream each time a client connects to an ExampleService.

Finally, add a registration class, a subclass of Registration, to the same file. The registration code is much simpler than the implementation. The server calls its register_into method at startup to register ExampleServicePlugin:

This example handles one class of object, ExampleService, and any number of its instances. To support more classes, do one of the following:

Complete server module

Altogether, the __init__.py should look like this:

init.py

Package the server plugin

Package the server-side code with server/pyproject.toml:

Note

The server-side pyproject.toml file must declare an entry point in the deephaven.plugin group named registration_cls that points to the registration class. The server ignores entry points with any other name.

You can test the server side before you write the client: see Use the plugin.

Client-side plugin

The client-side code and packaging live in the client folder of the project.

Client plugin implementation

As with the server-side code, the client-side code is in a single file, client/example_plugin_client/__init__.py. It defines ExampleServiceProxy, which wraps a PluginClient and gives the client the same hello_string and hello_table methods as the server-side object. Each method writes a request to the plugin client's req_stream and reads the reply from its resp_stream.

Client-side testing shows how to get a PluginClient from Session.plugin_client.

The two sides agree on a small protocol:

  • The client sends JSON with method and data keys. Tables travel separately from the JSON payload, as references. A reference is a handle to an object on the server.
  • The server replies with JSON containing result or error, plus any result tables as references.
  • When the connection opens, the server must send an initial message. This example sends an empty payload, which the proxy reads and discards in its constructor.

Package the client plugin

As with the server-side plugin, you package the client-side plugin with a TOML file, client/pyproject.toml:

Note

The client-side plugin for this example requires no registration.

Use the plugin

Once you have written and packaged the server-side and client-side code, you can test the plugin. Install the server-side package, then follow Server-side testing to start a server that exposes an ExampleService and Client-side testing to call it from a client.

Installation

Install the server-side package along with Deephaven, using the method that matches how you run the server.

Install with pip

If you start the server with pip-installed Deephaven, install Deephaven and the plugin in a virtual environment. The server also needs Java 17 or later, with JAVA_HOME set. See Install and run Deephaven with pip for details. The following commands create a virtual environment and install the necessary packages, including the plugin.

Install with Docker

If you start the server with Docker, use Docker Compose to manage the server and any other services. Save the following Dockerfile in the root folder of the project, ExampleServicePlugin/. It builds an image from Deephaven's server image and installs the plugin:

Important

The Docker Compose file below sets the pre-shared key to YOUR_PASSWORD_HERE for demonstration purposes. The client needs this key when connecting to the server. Change this to a more secure key in production.

Save the following Docker Compose file in the same folder. It runs Deephaven using the image built by the above Dockerfile:

With those two files in hand, docker compose up --build builds the image with the plugin installed and starts the server. Use --build again whenever you change the plugin code.

Server-side testing

To test the server side, create an ExampleService object on the server so a client can reach it. Follow the subsection that matches how you installed the plugin.

Test a pip-installed server

If you installed Deephaven with pip, save the following script in the project root and run it with python while venv-server is active. The script starts a Deephaven server, creates an instance of ExampleService, and keeps the server running until you press Ctrl+C.

The script imports the plugin package only after the server starts. The package imports deephaven modules, and Python can't import those until the server is running.

The -Dauthentication.psk JVM argument sets the pre-shared key that clients use to connect. Outside of testing, replace YOUR_PASSWORD_HERE with a secure key:

Test a Docker server

If you run Deephaven in Docker, create an instance of ExampleService so that a client can interact with it. Open the Deephaven IDE at http://localhost:10000/ide/, log in with the pre-shared key from the Docker Compose file, and run the following code:

Client-side testing

To test the client-side plugin, run the following commands from the root of the project to create a client virtual environment, separate from the server, and install the client package:

Important

Replace YOUR_PASSWORD_HERE with the pre-shared key you set when starting the server, either in the Docker Compose file or in the jvm_args passed to Server.

While the server from Server-side testing is running, save the following script in the project root and run it with python while venv-client is active. The script uses the client-side plugin in three steps:

  1. It looks up the server's example_service variable in the Session object's exportable_objects property. This property maps each exportable variable name to a ServerObject, which holds the object's type and a ticket. A ticket is a client's reference to an object on the server.
  2. It wraps the ServerObject in a plugin client with Session.plugin_client.
  3. It passes the plugin client to ExampleServiceProxy and calls the proxy's two methods.

The script prints Result: Hello client. You said: Hello server!, then a 10-row table with the input column X plus the Client column (Hello server!) and the Server column (Hello client!) that the server added.

Share plugin objects between sessions

Sharing is useful when several client sessions need to work with the same server-side plugin object. Every session can already look up a variable in the server's query scope, such as example_service, by name through exportable_objects. An object with no name, such as a table the server sent to one session as a reference, is visible only to the session that received it.

To share an object like that, the session that has it publishes it, and other sessions fetch it. To keep the code short, the following example publishes example_service.

Publish a plugin object

Fetching a plugin object gives your session an export ticket, which only that session can use. Publishing the export ticket to a shared ticket lets any session fetch the same object.

The following script fetches example_service, publishes the export ticket to a new shared ticket, and prints the shared ticket as a hex string that other sessions can use:

Leave this script running while you run the fetch script below, and copy the hex string it prints.

Important

Shared tickets remain valid only as long as the publishing session keeps the object alive. When the publishing session closes or releases the object, the shared ticket becomes invalid.

Fetch a shared plugin object

Another session can fetch the shared plugin object using the shared ticket. In the following script, replace SHARED_TICKET_HEX with the hex string that the publishing script printed:

The type string must match the server plugin's name property, ExampleService. If you change the type name in the plugin, change it here too.

Note

An earlier version of this example is in the plugin-bidirectional-example repository. Its client imports a pydeephaven module that no longer exists, so use the code on this page.