Core+ installation and configuration

To get started with Deephaven Core+, you must first determine which version to install. Core+ packages are built using a specific Enterprise system version (e.g., 1.20231218.432) and a specific Community version (e.g., 0.33.6). You must select a package that matches the Enterprise version of your system and whatever Community Engine version you require.

For example, if your system is running Vermilion (1.20231218.432) and you want to use the 0.33.6 Community Core engine with jdk17, then you should download the deephaven-coreplus-0.33.6-1.20231218.432-jdk17.tgz package.

You can find the published Core+ packages in Artifactory.

See the version matrix for the full list of tested versions.

Prepare the system

The Deephaven installer can automatically unpack the Core+ archive on each machine in the system. To do so, you must set the DH_DND_VERSIONS variable. You may also manually install Core+ by unpacking the tar file using the following instructions.

Once you have selected the package, you must prepare each system for the installation. The Core+ package must be installed on every node in the cluster.

On each host, create the directory /usr/illumon/dnd, then extract the Core+ package into it. The default configuration included with Deephaven Enterprise is configured to expect the Core+ installation to exist at /usr/illumon/dnd/latest, so you must create a symlink to the tar extracted directory.

dndVersion=deephaven-coreplus-0.33.6-1.20231218.432-jdk17
sudo -u irisadmin mkdir /usr/illumon/dnd
sudo -u irisadmin tar -xf /tmp/$dndVersion.tgz -C /usr/illumon/dnd
sudo -u irisadmin rm -f /usr/illumon/dnd/latest
sudo -u irisadmin ln -s /usr/illumon/dnd/$dndVersion /usr/illumon/dnd/latest

If manually installing a Core+ package, you should also create a Python virtual environment for that version. If you are installing into the "latest" directory as above, then after installing the venv you should create a symbolic link as follows:

venvPath=/usr/illumon/dnd/venv/deephaven-coreplus-0.33.6-1.20231218.432-jdk17
sudo -u irisadmin python3.10 -m venv $venvPath
sudo -u irisadmin /bin/bash -c "source $venvPath/bin/activate && pip3.10 install -r /usr/illumon/dnd/latest/py/resources/requirements.txt"
sudo -u irisadmin rm -f /usr/illumon/dnd/venv/latest
sudo -u irisadmin ln -sf $venvPath /usr/illumon/dnd/venv/latest

At this point, you will be able to start Core+ Python workers.

/usr/illumon/latest/bin/dh_monit restart db_query_server
/usr/illumon/latest/bin/dh_monit restart db_merge_server

The requirements.txt file contains packages that must be installed, but does not explicitly list all dependencies. There are two more files in the same directory as the requirements file that list all the dependencies as determined by pip freeze on a virtual environment with the requirements installed:

• frozen-requirements-1.20231218.432-0.33.6.txt has all the necessary packages for the server-side virtual environment.
• frozen-requirements-client-1.20231218.432.txt has all the necessary packages for the Deephaven Core+ Python client.

These files can also be extracted from the Core+ archive under the io.deephaven.enterprise.dnd-0.33.6-1.20231218.432/py/resources directory.

Customization

You can install multiple versions of Core+ to take advantage of newer Core+ Engine improvements. You can do this by extracting the package next to any others in the /usr/illumon/dnd folder and then adding some additional configuration to the iris-environment.prop file. You may also reuse the same installation directories for multiple WorkerKinds, but with different Python virtual environments or custom libraries for distinct applications.

There are four properties that, together, define a runnable version:

WorkerKind.DeephavenCommunity.class=com.illumon.iris.db.tables.remotequery.DndWorkerKind
WorkerKind.DeephavenCommunity.title=Community
WorkerKind.DeephavenCommunity.path=/usr/illumon/dnd/latest
WorkerKind.DeephavenCommunity.venv=/usr/illumon/dnd/venv/latest

The DeephavenCommunity part of the property defines a class of worker with the same name. You can copy these properties and change DeephavenCommunity to something else, for example CorePlusBeta, then change the path and title properties to define the installation. For example, the following set of properties will define a new worker kind for a hypothetical Community version of 0.30.0dev3:

WorkerKind.CorePlusBeta.class=com.illumon.iris.db.tables.remotequery.DndWorkerKind
WorkerKind.CorePlusBeta.path=/usr/illumon/dnd/deephaven-coreplus-0.30.0dev3-1.20231218.432
WorkerKind.CorePlusBeta.venv=/usr/illumon/dnd/venv/0.30.0dev3
WorkerKind.CorePlusBeta.title=Core+ Beta

Note: If you are installing Python, follow the steps above, creating a new venv directory for the Core+ version being installed, if it differs from the existing one.

To modify the configuration, export it using the dhconfig tool, edit it and then import it again, with the dhconfig tool as shown below

/usr/illumon/latest/bin/dhconfig properties export iris-environment.prop -d /tmp
# Edit the file with vim, or your favorite editor
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import /tmp/iris-environment.prop

Once you refresh your browser instance, you will see the new instances in the Engine dropdown menu:

When selecting the engine (Legacy, Core+, or Core+ Beta) in a Code Studio, a property controls the display order for each worker kind. Lower values are displayed first by the Code Studio drop-down. The Legacy engine is hardcoded to 100. Core+ workers default to 200, but in the default v1.20230511 (Vermilion) iris-environment.prop, they are set to 50. As a result, new Vermilion installations display Core+ first, but upgraded installations display Legacy first.

In the following example, the "Core+ Beta" worker is displayed after a "Core+" worker with display order 50 but before the "Enterprise" worker with display order 100.

WorkerKind.CorePlusBeta.class=com.illumon.iris.db.tables.remotequery.DndWorkerKind
WorkerKind.CorePlusBeta.path=/usr/illumon/dnd/deephaven-coreplus-0.30.0dev3-1.20231218.432
WorkerKind.CorePlusBeta.venv=/usr/illumon/dnd/venv/0.30.0dev3
WorkerKind.CorePlusBeta.title=Core+ Beta
WorkerKind.CorePlusBeta.displayOrder=75

Installing Custom Python Virtual Environments

If DH_DND_PYTHON is set to true in your cluster.cnf file, then a default Python virtual environment is configured for Core+ workers. You can also create alternative Python virtual environments and assign them to your own custom worker kinds.

If you do not already have Python installed for Enterprise, see our installation guide. To install a Python environment for the Core+ worker, you must create a virtual environment and configure it properly for the installed Community version. Deephaven recommends placing them in /usr/illumon/dnd/venv/. In the following example, we create a default virtual environment in the /usr/illumon/dnd/venv/0.33.6-custom directory. The requirements.txt file provides a list of packages and versions that Deephaven has tested with this distribution. You should use it as a starting point for adding, removing, or replacing packages. Deephaven does not test our integrations with package versions other than those listed in requirements.txt.

# Replace the paths below where you wish to a) install the virtual environment
# and b) take the requirements.txt file (which specifies Python dependencies) from.
# While not required, Deephaven recommends that you install the virtual environment
# inside the Core+ package directory for each specific version.
venvPath=/usr/illumon/dnd/venv/0.33.6-custom
sudo -u irisadmin python3.10 -m venv $venvPath
sudo -u irisadmin /bin/bash -c "source $venvPath/bin/activate && pip3.10 install -r /usr/illumon/dnd/latest/py/resources/requirements.txt"

Custom Libraries

There are several ways to provide Core+ workers with custom Java libraries:

1. The "Extra Classpaths" field from a console or Persistent Query configuration, which can specify arbitrary locations.
2. A Core+ WorkerKind configuration pointing to a customer-created location, e.g.:

   WorkerKind.CorePlusBeta.customLib=/usr/illumon/dnd/custom_lib/0.30.0dev3
   

   This can be useful to define several different kinds of workers, with distinct custom libraries that depend on the worker's role. By defining several worker kinds, it is easy to switch between them, and for administrators to change the libraries without requiring additional per-worker configuration by the user.
3. A default directory found in every Core+ installation, e.g. /usr/illumon/dnd/latest/custom_lib/.

Installing JavaScript Plugins

Community JavaScript plugins can be installed in Core+ workers.

1. Create a Dockerfile containing the following:

   FROM ghcr.io/deephaven/web-plugin-packager:latest as js-plugins
   
   # Package the JS plugins to `/build/js-plugins`
   ENTRYPOINT cd build && ../pack-plugins.sh \
      # Plugins to install from `npm`
      @deephaven/js-plugin-matplotlib@0.4.0 \
      @deephaven/js-plugin-plotly@0.3.0 \
      @deephaven/js-plugin-plotly-express@0.3.0 \
      @deephaven/js-plugin-ui@0.1.0 \
      # Local plugins are supported but will require
      # volume mounts when running the container.
      /plugin-a
      /plugin-b
   

2. Update the list of package specs passed to ./pack-plugins.sh based on the plugins you want to install. The script uses npm pack under the hood and should support any valid <package-spec>. Typical specs would be a package name + version (e.g., @deephaven/js-plugin-matplotlib@0.4.0) or a local file path to a non-published package. See the npm docs for additional documentation.

1. Build the image:

   docker build -t my-builder-image .
   

2. Run the container:

   docker run --rm \
   -v $PWD/build/:/build \
   # Include optional volume mounts for local plugins
   -v /some/path/to/plugin-a:/plugin-a \
   -v /some/path/to/plugin-b:/plugin-b \
   # Image that was built in previous step
   my-builder-image
   

   This should create a build/js-plugins folder on your host machine in the directory containing your Dockerfile.

3. Copy the build/js-plugins directory to a tmp directory on the server running Enterprise:

   scp -r ./build/js-plugins <VM address>:/tmp
   

4. You can now delete the build/js-plugins folder on your host machine, as you will not need it anymore.

5. On the server, copy the js-plugins directory into the Core+ install:

   sudo -u irisadmin cp -r /tmp/js-plugins /usr/illumon/dnd/latest/