Installing Deephaven without Docker

Typically, when you start Deephaven, you would go through these steps, which use Docker and Docker Compose to set up a Deephaven server and its dependencies.

However, running inside of Docker is not the only way you can set up Deephaven. Today, I'm going to walk through the steps necessary to get a Deephaven Community Core server running on a fresh install of Linux.

Here's the kernel I just installed on a recently flashed machine:

Preparing the machine

Pre-Installed dependencies

Although we'll install most dependencies as we go, a few tools we need were already installed on my machine:

Creating a deephaven user

Deephaven's server is very powerful. You probably already know, but users can write and execute custom code within a REPL. You should assume that any user will have the same privileges as the user that owns the JVM process. For a little extra protection, I like to create a new user that has relatively limited access to this system.

Let's create a root level directory and transfer ownership to our new user:

Ensure this directory is writable by the deephaven user group and that new files inherit the group ownership:

Let's add our user to the deephaven group.

Restart your terminal to allow the group change to take effect.

You can verify that your user is now in the deephaven group:

Build Community Core

The following section can be skipped if you don't intend to build on this machine. Instead, you can upload and install artifacts built from another host if that suits your style better. Nothing in this section is required to run a Deephaven instance.

If you aren't building on this machine, jump to the Nginx section.

Installing JDK 8

Deephaven requires jdk8 to build.

Installing Docker

Although our end product won't be using Docker, the build system still utilizes Docker to ensure that any of our users can build the same artifacts that we do.

Here I follow the official instructions as listed for the hirsute 21.04 distribution:

1. Enable apt to use a repository over https:

2. Install Docker's official GPG key:

3. Add Docker's stable hirsute repository:

4. Install the latest and greatest version of Docker (see instructions for alternatives):

5. Run docker hello-world test (as super user):

This is the response I received:

Docker response

Sudo-less Docker

Let's follow Docker's post-install instructions that enable sudo-less use of Docker.

1. Create the docker group:

2. Add your user to the docker group:

3. Restart your terminal to allow the group change to take effect:

4. Run the docker hello-world test (no longer as a super user!):

Building

Let's obtain the source!

Let's check out the most recently released version of Deephaven (as of this post):

Finally, we build and package Deephaven Community Core:

Nginx

To serve-up the static assets of our web-client-ui, Deephaven's Docker setup uses nginx. We'll do the same.

You can follow Ubuntu's nginx tutorial to learn more about nginx.

Install nginx:

Configure Deephaven as an nginx enabled-site. Let's seed our configuration from the existing configuration that Deephaven's docker compose uses:

Edit the file to your taste. In particular:

• I'm changing the port from the default HTTP port of 80 to 1818. This helps us avoid using HTTP when we want to HTTPS.
• I'm replacing the location root from /usr/share/nginx/html to /deephaven/html.
• I'm replacing the storage location for notebooks/layouts from /data to /deephaven/storage.
• I'm replacing the server_name from localhost to a DNS entry on my personal domain that resolves to its IP address on my internal network.

My version of this file can be found here.

We also need to configure the notebooks limit:

• Place dav_ext_lock_zone zone=notebooks:10m; near the end of the http section of /etc/nginx/nginx.conf.

I remove the default site that is installed with nginx. It binds to port 80, and I would prefer this machine to only respond to HTTPS requests:

Let's create the necessary folders to make the configuration valid:

Let's also add the www-data user to our deephaven group, so that it also can read and write to /deephaven:

Finally, let's restart the nginx service:

If you're successful, then hitting port 80 responds with This site cannot be reached, and hitting port 1818 should give you a 404.

Envoy

Deephaven uses Envoy to send static content requests to nginx and gRPC requests to the server process. To install, I'll follow along with the official directions:

Let's grab the Envoy configuration that the Docker setup uses as a starting point:

1. Remove references to grpc-proxy:

• Remove the route entry that forwards calls to grpc-proxy (from listener_0).
• Remove the cluster named grpc-proxy.

2. Replace listener_0's socket_address port from 10000 to default HTTPS port of 443.

3. Configure SSL.

I added this SSL configuration block under listener_0's filter_chains (at the same level as filters):

4. Change the endpoint.address.socket_address.address for both server and web clusters. I've set these both to localhost.

5. Change the socket_address.port_value for the web cluster from 80 to the same port you configured nginx to listen to. (I used port 1818.)

6. Either remove the admin port entry, or restrict access to localhost (or 127.0.0.1).

For your convenience, here are the initial and final configurations.

Finally, let's start Envoy:

We should now be able to navigate to the https version of our page and land on that same 404 that nginx served us earlier.

Automating the launch of Envoy is left as an exercise to the reader.

Install Community Core

Now that we have built the native distribution, we can populate our host with the necessary artifacts and resources:

Let's initialize our runtime logback.xml. Note that you can start with the template provided at /deephaven/deephaven-core/server/server/native/build/resources/logback.xml. If you choose to roll your own, be sure to copy the LogBufferAppender section or else REPL users won't get logs. I prefer rolling logs, so I went with this:

Here's a mini-script that wraps starting up the Deephaven server:

Install the Community web client

We'll utilize the web-client-ui build that deephaven-core depends on.

1. Build and install the jsapi:

2. Extract Deephaven's Code Studio from the released npm package:

We'll use npm to download Deephaven's code-studio. Alternatively, you can download the tar file directly from npmjs.

Let's check the version that deephaven-core is going to expect:

The results tell us this was built with version 0.3.1:

Fetch and unpack the Deephaven IDE:

Extra Credit

If you've been following along, your server is now running nginx for static assets, envoy to enable SSL and route between nginx and deephaven, and a JVM running deephaven in Groovy REPL mode. See below for instructions to install a custom version of the web-client. Integrating with Python is a bit involved and therefore out of scope for this post.

Further reading

• Learn more about Application Mode.
• Learn more about the Deephaven Core API.
• Explore the configuration files in this post.

Build the web client

Testing out changes to the web-client-ui is quite easy to accomplish in our new environment. Let me walk you through the steps.

1. The web client uses npm to build. Let's install it:

2. At the time of this article, this build requires npm version 6.14.14. Let's install that now:

Restart your terminal to allow the npm version change to take effect.

3. Let's checkout the source:

4. Let's prepare the repository to build the release artifacts:

5. Let's produce the release artifacts:

6. Finally, let's replace the existing ide with the one we just built:

7. After a quick refresh of the browser, you're using your own build of the web client!