Customize the basic installation

Install multiple versions of Core+

You can install multiple versions of Core+ to take advantage of newer Core+ Engine improvements. You can do this by extracting the package next to any others in the /usr/illumon/coreplus folder and then adding some additional configuration to the iris-environment.prop file. You may also reuse the same installation directories for multiple WorkerKinds, but with different Python virtual environments or custom libraries for distinct applications.

There are four properties that, together, define a runnable version:

WorkerKind.DeephavenCommunity.class=com.illumon.iris.db.tables.remotequery.DndWorkerKind
WorkerKind.DeephavenCommunity.title=Core+
WorkerKind.DeephavenCommunity.path=/usr/illumon/coreplus/latest
WorkerKind.DeephavenCommunity.venv=/usr/illumon/coreplus/venv/latest

The DeephavenCommunity part of the property defines a class of worker with the same name. You can copy these properties and change DeephavenCommunity to something else, for example CorePlusBeta, then change the path and title properties to define the installation. For example, the following set of properties will define a new worker kind for a hypothetical Community version of 0.30.0dev3:

WorkerKind.CorePlusBeta.class=com.illumon.iris.db.tables.remotequery.DndWorkerKind
WorkerKind.CorePlusBeta.path=/usr/illumon/coreplus/deephaven-coreplus-0.30.0dev3-1.20240517.344
WorkerKind.CorePlusBeta.venv=/usr/illumon/coreplus/venv/0.30.0dev3
WorkerKind.CorePlusBeta.title=Core+ Beta

Note: If you are installing Python, follow the steps above, creating a new venv directory for the Core+ version being installed, if it differs from the existing one.

To modify the configuration, export it using the dhconfig tool, edit it and then import it again, with the dhconfig tool as shown below

/usr/illumon/latest/bin/dhconfig properties export iris-environment.prop -d /tmp
# Edit the file with vim, or your favorite editor
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import /tmp/iris-environment.prop

Important

You must restart the Query Server, Merge Server and Web API Service on all nodes on which you install the package. Also, you must refresh your browser without caching.

Once you refresh your browser instance, you will see the new instances in the Engine dropdown menu:

img

When selecting the engine (Legacy, Core+, or Core+ Beta) in a Code Studio, a property controls the display order for each worker kind. Lower values are displayed first by the Code Studio drop-down. The Legacy engine is hardcoded to 100. Core+ workers default to 200, but in the default v1.20230511 (Vermilion) and later iris-environment.prop, they are set to 50 with the property WorkerKind.DeephavenCommunity.displayOrder. As a result, new installations display Core+ first, but installations initially created with releases before Vermilion display Legacy first.

In the following example, the "Core+ Beta" worker is displayed after a "Core+" worker with display order 50 but before the "Enterprise" worker with display order 100.

WorkerKind.CorePlusBeta.class=com.illumon.iris.db.tables.remotequery.DndWorkerKind
WorkerKind.CorePlusBeta.path=/usr/illumon/coreplus/deephaven-coreplus-0.30.0dev3-1.20240517.344
WorkerKind.CorePlusBeta.venv=/usr/illumon/coreplus/venv/0.30.0dev3
WorkerKind.CorePlusBeta.title=Core+ Beta
WorkerKind.CorePlusBeta.displayOrder=75

Install custom Python virtual environments

It's common to install custom Python virtual environments for the Core+ worker, as the default virtual environment is not suitable for all use cases. You can install custom virtual environments with either pip or conda. For instructions on installing custom Python virtual environments in Core+ workers, see Install custom Python virtual environments.

Custom libraries

There are several ways to provide Core+ workers with custom Java libraries:

  1. The "Extra Classpaths" field from a console or Persistent Query configuration, which can specify arbitrary locations.
  2. A Core+ WorkerKind configuration pointing to a customer-created location, e.g.: 
    WorkerKind.CorePlusBeta.customLib=/usr/illumon/coreplus/custom_lib/0.30.0dev3
    This can be useful to define several different kinds of workers, with distinct custom libraries that depend on the worker's role. By defining several worker kinds, it is easy to switch between them, and for administrators to change the libraries without requiring additional per-worker configuration by the user.
  3. A default directory found in every Core+ installation, e.g. /usr/illumon/coreplus/latest/custom_lib/.

Install JavaScript Plugins

Community JavaScript plugins can be installed in Core+ workers.

  1. Create a Dockerfile containing the following:

    FROM ghcr.io/deephaven/web-plugin-packager:latest as js-plugins

# Package the JS plugins to `/build/js-plugins`
ENTRYPOINT cd build && ../pack-plugins.sh \
   # Plugins to install from `npm`
   @deephaven/js-plugin-matplotlib@0.4.0 \
   @deephaven/js-plugin-plotly@0.3.0 \
   @deephaven/js-plugin-plotly-express@0.3.0 \
   @deephaven/js-plugin-ui@0.1.0 \
   # Local plugins are supported but will require
   # volume mounts when running the container.
   /plugin-a
   /plugin-b

  2. Update the list of package specs passed to ./pack-plugins.sh based on the plugins you want to install. The script uses npm pack under the hood and should support any valid <package-spec>. Typical specs would be a package name + version (e.g., @deephaven/js-plugin-matplotlib@0.4.0) or a local file path to a non-published package. See the npm docs for additional documentation.

Note

Local file paths require volume mounts in order to be available to the Docker container.

Caution

This replaces the default JS plugins that are typically installed, so you must include those if you want to keep them. To see the existing list of installed plugins, you can run cat /usr/illumon/coreplus/latest/js-plugins/manifest.json on the server hosting the DHE installation.

  1. Build the image:

    docker build -t my-builder-image .

  2. Run the container:

    docker run --rm \
-v $PWD/build/:/build \
# Include optional volume mounts for local plugins
-v /some/path/to/plugin-a:/plugin-a \
-v /some/path/to/plugin-b:/plugin-b \
# Image that was built in previous step
my-builder-image

    This should create a build/js-plugins folder on your host machine in the directory containing your Dockerfile.

  3. Copy the build/js-plugins directory to a tmp directory on the server running Enterprise:

    scp -r ./build/js-plugins <VM address>:/tmp

  4. You can now delete the build/js-plugins folder on your host machine, as you will not need it anymore.

  5. On the server, copy the js-plugins directory into the Core+ install:

    sudo -u irisadmin cp -r /tmp/js-plugins /usr/illumon/coreplus/latest/