Git

This guide demonstrates how to integrate Deephaven with Git and use a Git repository as a source for Persistent Queries (PQs). This guide uses GitHub as the Git repository integration, but any Git service will work.

By default, a Persistent Query (PQ) is stored in the controller as an XML file containing the configuration and the script. As demonstrated in this guide, the script for a PQ can also be sourced from a Git repository.

Note

Some of the steps shown below require admin access on the infrastructure host.

Set up a Git repository

Create a Git repository with a directory structure that contains two folders: Groovy and Python. Each folder will contain the scripts for their respective languages.

img

All the following commands need to be done by the irisadmin user on the infrastructure box.

To prepare your Deephaven installation to source files from a Git repository, some configuration is needed. This requires the following steps:

  1. Create an SSH folder if one does not yet exist.
  2. Add your Git repository URL to the known_hosts file.
  3. Generate a new SSH key pair if one does not already exist.
  4. Copy the public key to your Git repository.
# Step 1 - If ssh has never been used, create the directory.
mkdir ~irisadmin/.ssh

# Step 2 - Add the git url to known_hosts. Make sure to replace `github.com` with the domain where your git repo is hosted.
ssh-keyscan -t rsa github.com >> ~irisadmin/.ssh/known_hosts

# Step 3 - Generate a new SSH key pair if one does not already exist. When prompted, save in the default location with no password.
ssh-keygen -t ed25519 -C "your_email@example.com"

# Step 4 - Copy the public key after it's printed to the console. You will need to add this to your Git repository.
cat ~/.ssh/id_ed25519.pub

After the public key is copied to your clipboard, paste it into your Git repository. In GitHub, this is done by going to your profile settings and navigating to the SSH and GPG keys section. Since this is an SSH key, select the New SSH key option.

img

See the GitHub documentation for more details.

Configure Deephaven

Several properties need to be set in order to integrate Deephaven with Git. These properties are found in etcd, which cannot be modified directly. So, to update properties, use the dhconfig command line utility to export the properties from etcd to the filesystem:

# Export properties from etcd to /tmp
/usr/illumon/latest/bin/dhconfig properties export iris-environment.prop -d /tmp/

Then, edit /tmp/iris-environment.prop to add properties. The properties shown below are:

  • PersistentQueryController.useLocalGit: Whether or not the Git repository is local. If false, the repository is remote.
  • iris.scripts.repos: A comma-separated list of Git repositories to use as a source for Persistent Queries. testDH is given as the example repository name, but this should be set to the name of your repository.
  • iris.scripts.repo.testDH.groups: A comma-separated list of Deephaven user groups that can access the repository. Use \* to allow all groups.
  • iris.scripts.repo.testDH.updateEnabled: Whether or not the repository should be updated on a schedule. true indicates that the latest is pulled once per minute.
  • iris.scripts.repo.testDH.branch: The branch to pull from.
  • iris.scripts.repo.testDH.prefixDisplayPathsWithRepoName: Whether or not the repository name should be prefixed to the display path of the script.
  • iris.scripts.repo.testDH.root: The filesystem directory to which Deephaven will clone the repository.
  • iris.scripts.repo.testDH.paths: A comma-separated list of paths relative to the root of the Git repository that contains scripts.
  • iris.scripts.repo.testDH.uri: The URI of the Git repository.

For a full description of Git configuration properties, see Git configuration.

PersistentQueryController.useLocalGit=false

# Set the repository name below to that of your Git repository
iris.scripts.repos=testDH

iris.scripts.repo.testDH.groups=\*
iris.scripts.repo.testDH.updateEnabled=true
iris.scripts.repo.testDH.branch=main
iris.scripts.repo.testDH.prefixDisplayPathsWithRepoName=false
iris.scripts.repo.testDH.root=git/myRepoFolder
iris.scripts.repo.testDH.paths=scripts/groovy,scripts/python

# Fill in your git domain, username, and repository name here.
iris.scripts.repo.testDH.uri=git@{git-domain.com}:{username}/{repository-name}.git

When done modifying and adding properties, import the properties back into etcd:

# Append the properties listed to /tmp/iris-environment.prop
/usr/illumon/latest/bin/dhconfig properties import -f /tmp/iris-environment.prop

To ensure the configuration changes are correct, run dhconfig pq check-repo; the output should indicate if the configuration implied by the new properties can be loaded.

/usr/illumon/latest/bin/dhconfig pq check-repo

After verifying the configuration is correct, restart the iris_controller:

monit restart iris_controller

Note

It is not recommended to restart the controller during normal operation, as it will disrupt other services.

When the controller starts, it will automatically pull in the repository. If the controller is unable to do so, it will fail to start. Error logs are found in startup logs under /var/log/deephaven/iris_controller/iris_controller.log.<yyyy-mm-dd>, where <yyyy-mm-dd> is the date on which the controller experienced the error.

Deploy a PQ with Git

PQs can now use the Git repository as a source for their scripts. The Query Monitor has an option Use GIT for source that allows you to select a script from the Git repository in a drop-down menu:

img