Installing Deephaven without Docker

Typically, when you start Deephaven, you would go through these steps, which use Docker and Docker Compose to set up a Deephaven server and its dependencies.

However, running inside of Docker is not the only way you can set up Deephaven. Today, I'm going to walk through the steps necessary to get a Deephaven Community Core server running on a fresh install of Linux.

note

We have now simplified deployment via a single application and Deephaven can more easily be run without Docker. If you have any questions, reach out on our Community Slack.

Here's the kernel I just installed on a recently flashed machine:

/$ uname -a
Linux wote 5.11.0-34-generic #36-Ubuntu SMP Thu Aug 26 19:22:09 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Preparing the machine

Pre-Installed dependencies

Although we'll install most dependencies as we go, a few tools we need were already installed on my machine:

Program	Version Installed
git	2.30.2
tar	1.34

Creating a deephaven user

Deephaven's server is very powerful. You probably already know, but users can write and execute custom code within a REPL. You should assume that any user will have the same privileges as the user that owns the JVM process. For a little extra protection, I like to create a new user that has relatively limited access to this system.

note

Add the JVM flag -Ddeephaven.console.disable=true to disable all REPL access.

/$ sudo adduser deephaven

Let's create a root level directory and transfer ownership to our new user:

/$ sudo mkdir /deephaven
/$ sudo chown -R deephaven:deephaven /deephaven

Ensure this directory is writable by the deephaven user group and that new files inherit the group ownership:

/$ sudo chmod g+ws /deephaven

Let's add our user to the deephaven group.

/$ sudo usermod -aG deephaven $USER

Restart your terminal to allow the group change to take effect.

You can verify that your user is now in the deephaven group:

/$ groups $USER
nate : nate adm cdrom sudo dip plugdev lxd deephaven

Build Community Core

The following section can be skipped if you don't intend to build on this machine. Instead, you can upload and install artifacts built from another host if that suits your style better. Nothing in this section is required to run a Deephaven instance.

If you aren't building on this machine, jump to the Nginx section.

Installing JDK 8

Deephaven requires jdk8 to build.

/$ sudo apt install openjdk-8-jdk
[ ... ]

/$ java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)

Installing Docker

Although our end product won't be using Docker, the build system still utilizes Docker to ensure that any of our users can build the same artifacts that we do.

Here I follow the official instructions as listed for the hirsute 21.04 distribution:

1. Enable apt to use a repository over https:

/$ sudo apt install apt-transport-https \
    ca-certificates                     \
    curl                                \
    gnupg                               \
    lsb-release

2. Install Docker's official GPG key:

/$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
    sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

3. Add Docker's stable hirsute repository:

/$ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] \
       https://download.docker.com/linux/ubuntu \
       $(lsb_release -cs) stable" | \
    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

4. Install the latest and greatest version of Docker (see instructions for alternatives):

/$ sudo apt update
/$ sudo apt install docker-ce docker-ce-cli containerd.io

5. Run docker hello-world test (as super user):

/$ sudo docker run hello-world

This is the response I received:

Docker response

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:61bd3cb6014296e214ff4c6407a5a7e7092dfa8eefdbbec539e133e97f63e09f
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:

 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image, which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:

 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Sudo-less Docker

danger

The docker group grants privileges equivalent to the root user. Every account added to the group can elevate privileges to root without a password.

Let's follow Docker's post-install instructions that enable sudo-less use of Docker.

1. Create the docker group:

/$ sudo groupadd docker

2. Add your user to the docker group:

/$ sudo usermod -aG docker $USER

3. Restart your terminal to allow the group change to take effect:

/$ groups $USER
nate : nate adm cdrom sudo dip plugdev lxd deephaven docker

4. Run the docker hello-world test (no longer as a super user!):

/$ docker run hello-world

Building

Let's obtain the source!

/$ cd /deephaven

## check-out the latest version of Deephaven
/deephaven$ git clone https://github.com/deephaven/deephaven-core

/deephaven$ cd deephaven-core

Let's check out the most recently released version of Deephaven (as of this post):

/deephaven$ git checkout v0.4.1

note

If this isn't your first trip through this post, then you may want to clean any artifacts in the target build directory.

/deephaven/deephaven-core$ rm -rf ./server/server/native/build

Finally, we build and package Deephaven Community Core:

/deephaven/deephaven-core$ ./gradlew :server-server-native:build

Nginx

To serve-up the static assets of our web-client-ui, Deephaven's Docker setup uses nginx. We'll do the same.

You can follow Ubuntu's nginx tutorial to learn more about nginx.

Install nginx:

/$ sudo apt install nginx-full

note

Deephaven's notebook editor and layout manager use nginx's webdav module, which is not installed via the nginx package. This is why we install nginx-full here.

Configure Deephaven as an nginx enabled-site. Let's seed our configuration from the existing configuration that Deephaven's docker compose uses:

/$ sudo cp /deephaven/deephaven-core/web/client-ide/nginx/default.conf /etc/nginx/sites-enabled/deephaven

Edit the file to your taste. In particular:

• I'm changing the port from the default HTTP port of 80 to 1818. This helps us avoid using HTTP when we want to HTTPS.
• I'm replacing the location root from /usr/share/nginx/html to /deephaven/html.
• I'm replacing the storage location for notebooks/layouts from /data to /deephaven/storage.
• I'm replacing the server_name from localhost to a DNS entry on my personal domain that resolves to its IP address on my internal network.

My version of this file can be found here.

We also need to configure the notebooks limit:

• Place dav_ext_lock_zone zone=notebooks:10m; near the end of the http section of /etc/nginx/nginx.conf.

I remove the default site that is installed with nginx. It binds to port 80, and I would prefer this machine to only respond to HTTPS requests:

/$ sudo rm /etc/nginx/sites-enabled/default

Let's create the necessary folders to make the configuration valid:

/$ mkdir /deephaven/html
/$ mkdir /deephaven/storage
/$ mkdir /deephaven/storage/notebooks
/$ mkdir /deephaven/storage/layouts

Let's also add the www-data user to our deephaven group, so that it also can read and write to /deephaven:

/$ sudo usermod -aG deephaven www-data

Finally, let's restart the nginx service:

/$ sudo service nginx restart

If you're successful, then hitting port 80 responds with This site cannot be reached, and hitting port 1818 should give you a 404.

Envoy

Deephaven uses Envoy to send static content requests to nginx and gRPC requests to the server process. To install, I'll follow along with the official directions:

/$ sudo apt update
/$ sudo apt install apt-transport-https gnupg2 curl lsb-release
/$ curl -sL 'https://deb.dl.getenvoy.io/public/gpg.8115BA8E629CC074.key' | sudo gpg --dearmor -o /usr/share/keyrings/getenvoy-keyring.gpg
/$ # Verify the keyring - this should yield "OK"
/$ echo a077cb587a1b622e03aa4bf2f3689de14658a9497a9af2c427bba5f4cc3c4723 /usr/share/keyrings/getenvoy-keyring.gpg | sha256sum --check
/$ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/getenvoy-keyring.gpg] https://deb.dl.getenvoy.io/public/deb/ubuntu $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/getenvoy.list
/$ sudo apt update
/$ sudo apt install -y getenvoy-envoy

Let's grab the Envoy configuration that the Docker setup uses as a starting point:

/$ mkdir /deephaven/config
/$ cp /deephaven/deephaven-core/envoy/contents/envoy.yaml /deephaven/config

1. Remove references to grpc-proxy:

note

At the time of this writing, the Docker set up has a third utility called grpc-web. For our setup, we won't need it. gRPC-web enables gRPC over websockets, but we are going to utilize HTTP2, which requires SSL.

It is not strictly necessary to remove them from your configuration. I don't need them, so I prefer to remove them.

• Remove the route entry that forwards calls to grpc-proxy (from listener_0).
• Remove the cluster named grpc-proxy.

2. Replace listener_0's socket_address port from 10000 to default HTTPS port of 443.

3. Configure SSL.

note

Obtaining an SSL certificate is out of scope for this post. However, there are numerous tutorials on how to generate a self-signed certificate. For our purposes, I generated a wildcard cert for my personal domain, at no cost, with the help of certbot. This specific path required that I add TEXT DNS entries to the domain that I control to prove ownership.

I added this SSL configuration block under listener_0's filter_chains (at the same level as filters):

transport_socket:
  name: envoy.transport_sockets.tls
  typed_config:
  '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
  common_tls_context:
    alpn_protocols: ['h2']
    tls_certificates:
      - certificate_chain:
          filename: /deephaven/config/fullchain.pem
        private_key:
          filename: /deephaven/config/privkey.pem

danger

It's important to keep tight restrictions on your SSL certificate's private key; it should not be readable by the deephaven user.

4. Change the endpoint.address.socket_address.address for both server and web clusters. I've set these both to localhost.

5. Change the socket_address.port_value for the web cluster from 80 to the same port you configured nginx to listen to. (I used port 1818.)

6. Either remove the admin port entry, or restrict access to localhost (or 127.0.0.1).

For your convenience, here are the initial and final configurations.

Finally, let's start Envoy:

/$ sudo envoy -c /deephaven/config/envoy.yaml

We should now be able to navigate to the https version of our page and land on that same 404 that nginx served us earlier.

Automating the launch of Envoy is left as an exercise to the reader.

Install Community Core

Now that we have built the native distribution, we can populate our host with the necessary artifacts and resources:

/$ export VERSION=0.4.1
/$ mkdir /deephaven/lib
/$ tar xvf /deephaven/deephaven-core/server/server/native/build/distributions/server-server-native-$VERSION.tar -C /deephaven/lib --strip-components=2 "server-server-native-$VERSION/lib/"

note

To redeploy, be sure to clear that lib directory before extracting the new jars:

/$ rm -rf /deephaven/lib/*

Let's initialize our runtime logback.xml. Note that you can start with the template provided at /deephaven/deephaven-core/server/server/native/build/resources/logback.xml. If you choose to roll your own, be sure to copy the LogBufferAppender section or else REPL users won't get logs. I prefer rolling logs, so I went with this:

/$ mkdir /deephaven/resources
/$ cat > /deephaven/resources/logback.xml
<configuration debug="false">

  <property name="LOG_FILE" value="dhvn" />
  <property name="LOG_DIR" value="/deephaven/logs/" />
  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${LOG_DIR}/${LOG_FILE}.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>${LOG_DIR}/archive/%d{yyyyMMdd}/%d{yyyyMMdd-HH-}${LOG_FILE}.gz</fileNamePattern>
      <totalSizeCap>300MB</totalSizeCap>
    </rollingPolicy>

    <encoder>
      <pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSS'Z', UTC} | %green(%-20.20thread) | %highlight(%5level) | %yellow(%-25.25logger{25}) | %m%n</pattern>
    </encoder>
  </appender>

  <!-- System.out / System.err may be redirected (and captured by LogBuffer).
       By referencing PrintStreamGlobalsConsole, we can be sure that we avoid that redirection so
       we don not double up messages to the LogBuffer. -->
  <appender name="STDOUT" class="io.deephaven.logback.PrintStreamGlobalsConsole">
    <encoder>
      <pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSS'Z', UTC} | %green(%-20.20thread) | %highlight(%5level) | %yellow(%-25.25logger{25}) | %m%n</pattern>
    </encoder>
  </appender>

  <appender name="LOGBUFFER" class="io.deephaven.logback.LogBufferAppender">
    <encoder>
      <!-- LogBufferRecord has timestamp and level, so no need to encode -->
      <pattern>%-20.20thread | %-25.25logger{25} | %m</pattern>
    </encoder>
  </appender>

  <root level="info">
    <appender-ref ref="FILE" />
    <appender-ref ref="LOGBUFFER" />
  </root>
</configuration>

Here's a mini-script that wraps starting up the Deephaven server:

#!/usr/bin/env bash

java_home=$(update-java-alternatives -l | grep -w "1.8.0" | sed 's/  */ /g' | cut -f 3 -d ' ')

pushd /
$java_home/bin/java -server                         \
         -XX:+UseG1GC                               \
         -XX:MaxGCPauseMillis=100                   \
         -XX:+UseStringDeduplication                \
         -Xmx8G                                     \
         -XshowSettings:vm                          \
         -cp /deephaven/resources/:/deephaven/lib/* \
         -Ddeephaven.console.type=groovy            \
         io.deephaven.grpc_api.runner.Main

Install the Community web client

We'll utilize the web-client-ui build that deephaven-core depends on.

1. Build and install the jsapi:

/$ cd /deephaven/deephaven-core
/deephaven/deephaven-core$ rm -rf /deephaven/html/jsapi
/deephaven/deephaven-core$ ./gradlew webpackSources
/deephaven/deephaven-core$ mkdir /deephaven/html/jsapi
/deephaven/deephaven-core$ cp proto/raw-js-openapi/build/dhapi/dh-internal.js /deephaven/html/jsapi
/deephaven/deephaven-core$ ./gradlew prepareCompose
/deephaven/deephaven-core$ cp -rf /deephaven/deephaven-core/web/client-ide/build/docker/dhapi/* /deephaven/html/jsapi/

2. Extract Deephaven's Code Studio from the released npm package:

We'll use npm to download Deephaven's code-studio. Alternatively, you can download the tar file directly from npmjs.

/$ sudo apt install npm

Let's check the version that deephaven-core is going to expect:

/$ cat /deephaven/deephaven-core/web/client-ui/Dockerfile

The results tell us this was built with version 0.3.1:

FROM docker.io/library/node:14.16.0 AS node
WORKDIR /usr/src/app

FROM node as build
ARG WEB_VERSION=0.3.1

# Pull in the published package from npmjs and extract is
RUN set -eux; \
    npm pack @deephaven/code-studio@${WEB_VERSION}; \
    tar --touch -xf deephaven-code-studio-${WEB_VERSION}.tgz; \
    rm deephaven-code-studio-${WEB_VERSION}.tgz;%

Fetch and unpack the Deephaven IDE:

/$ cd /deephaven/html
/deephaven/html$ rm -rf ide
/deephaven/html$ mkdir ide
/deephaven/html$ npm pack @deephaven/code-studio@0.3.1
/deephaven/html$ tar zxvf deephaven-code-studio-0.3.1.tgz -C ide package/build --strip-components=2
/deephaven/html$ rm deephaven-code-studio-0.3.1.tgz

Extra Credit

If you've been following along, your server is now running nginx for static assets, envoy to enable SSL and route between nginx and deephaven, and a JVM running deephaven in Groovy REPL mode. See below for instructions to install a custom version of the web-client. Integrating with Python is a bit involved and therefore out of scope for this post.

Further reading

• Learn more about Application Mode.
• Learn more about the Deephaven Core API.
• Explore the configuration files in this post.

Build the web client

Testing out changes to the web-client-ui is quite easy to accomplish in our new environment. Let me walk you through the steps.

1. The web client uses npm to build. Let's install it:

/$ sudo apt install npm

2. At the time of this article, this build requires npm version 6.14.14. Let's install that now:

/$ sudo npm install -g [email protected]

Restart your terminal to allow the npm version change to take effect.

/$ npm -v
6.14.14

3. Let's checkout the source:

/$ cd /deephaven
/deephaven$ git clone https://github.com/deephaven/web-client-ui

4. Let's prepare the repository to build the release artifacts:

/deephaven$ cd web-client-ui
/deephaven/web-client-ui$ npm install

5. Let's produce the release artifacts:

/deephaven/web-client-ui$ npm run build

6. Finally, let's replace the existing ide with the one we just built:

/deephaven/web-client-ui$ rm -rf /deephaven/html/ide
/deephaven/web-client-ui$ cp -rf packages/code-studio/build /deephaven/html/ide

7. After a quick refresh of the browser, you're using your own build of the web client!