Installing Envoy

This guide provides instructions for installing, upgrading, and restarting Envoy as Deephaven's front proxy in all three supported Deephaven installation types. Instructions will be given for Kubernetes and Podman first, as they have a much simpler operational model.

Install Envoy on Kubernetes

Envoy is always used with Kubernetes.

All you need to do is supply a signing certificate and configure the hostname and port where Deephaven clients can reach the exposed Envoy Kubernetes service.

Install Envoy on Podman

Podman installation supports using Envoy by passing the --envoy|-e flag to start_command.sh when creating your pod. If you build custom images, you must provide the Envoy binary to build_images.sh.

By default, Envoy on Podman will use the same TLS certificate as all other processes on the dh-infra pod. You may optionally place a PEM-formatted envoy.pem bundle containing the certificate chain and private key in the --tls-volume directory passed to start_command.sh.

Install Envoy on native installations

In order to use Envoy when installing natively, you must enable Envoy in cluster.cnf, supply a TLS signing certificate, and declare how Envoy is installed.

The minimum required configuration to use Envoy without root permission or containers:

DH_CONFIGURE_ENVOY=true
DH_ENVOY_INSTALL_MODE=NATIVE
# PODMAN or DOCKER options are also available.

If you are using default values for all other properties, the only other requirement is to supply your X.509 PEM-formatted certificate chain and private key bundle on the Installation host at DH_LOCAL_DIR/certs/envoy.pem.

Once you run the native installer, you can connect to Deephaven via Envoy at your configured https:// DH_ENVOY_FQDN:DH_ENVOY_PORT.

This minimal example assumes that you use Deephaven-managed Envoy installation to automatically install, update, and restart the Envoy reverse proxy.

If you choose to Manually manage Envoy, your system administrator must install and manage the Envoy process.

Manually-managed Envoy

By default, Envoy is neither installed nor restarted when the native installer runs, due to DH_ENVOY_INSTALL_MODE having a default value of NONE.

This policy ensures smooth upgrades for customers who have already set up Envoy on their own.

In the manually managed scenario, the Deephaven native installer configures the cluster to speak to Envoy for you, and generates the appropriate envoy.yaml for Envoy to use, but it does not install or restart Envoy.

Your system administrator will need to Manually install Envoy, restart it whenever you perform minor upgrades of Deephaven, and upgrade the Envoy version during major version upgrades of Deephaven.

Deephaven-managed Envoy

When DH_CONFIGURE_ENVOY=true, the installer can manage Envoy using the DH_ENVOY_INSTALL_MODE options:

DOCKER / PODMAN: pulls and runs the Envoy image and mounts the generated YAML and the Envoy endpoint certificate PEM. It creates a systemd service unit called deephaven_envoy to simplify process restarts and integrate with journalctl logging.
NATIVE: installs Envoy as a native systemd user service run by DH_ADMIN_USER.
NONE: does not install, configure, or restart Envoy.

Select a single node within the Deephaven cluster to run Envoy and ensure it has network access to the Deephaven services it will front.

Note

Before you begin, check the Deephaven version support matrix to identify the Envoy version compatible with your Deephaven release.

Envoy frontend URL

All Deephaven clients connect to Envoy through a configured fully qualified domain name and port. The method to supply the Envoy frontend URL varies based on Deephaven installation type.

Kubernetes installations set envoyFrontProxyUrl to specify both hostname and port in values.yaml. Podman installations supply -h|--hostname <hostname> -e|--envoy <port> to start_command.sh. Native installations set DH_ENVOY_FQDN and DH_ENVOY_PORT in cluster.cnf.

The Envoy frontend URL forms the machine identity, which all Deephaven clients must trust when making TLS connections.

Required TLS certificate

Envoy requires a TLS signing certificate and private key, provided as follows:

Native Installation: provide DH_ENVOY_LOCAL_CERT on the Installer host, by default DH_LOCAL_DIR/certs/envoy.pem.
Podman Installation: provide envoy.pem in the --tls-volume directory passed to start_command.sh.
Kubernetes Installation: provide the deephaven-tls Kubernetes secret.

The Envoy certificate must include a subject alternative name (SAN) entry matching the Envoy frontend URL which Deephaven clients use to reach Envoy. Hostname verification is SAN-based; neither certificate Common Names nor authority overrides are used.

The provided Envoy certificate is verified by clients using an optional truststore file, or using the client's system truststore

Envoy installation methods

Tip

This section is only relevant to native installation methods. Podman and Kubernetes install Envoy into container images.

While many Envoy installation methods exist, Deephaven recommends using Deephaven-managed PODMAN if available, as Podman quadlets offer the most elegant systemd integration. On Ubuntu installations, DOCKER has good integration with systemd, while NATIVE is harder to use, but works without a container or root permission.

The following sections provide instructions for common installation methods.

Example Podman quadlet

When DH_ENVOY_INSTALL_MODE=PODMAN, Deephaven will generate a Podman quadlet file.

The sample Podman quadlet file below is configured with default Envoy port and Envoy admin port, run as the default Admin service account, using default locations for envoy.yaml and Envoy certificates.

/etc/containers/systemd/deephaven_envoy.container

[Unit]
Description=Envoy proxy server (deephaven_envoy)
Wants=network-online.target
After=network-online.target

[Container]
ContainerName=deephaven_envoy
Image=docker.io/envoyproxy/envoy:v1.36.4

PublishPort=8000:8000
# this example only opens the sensitive admin port to localhost connections on the container's host machine
PublishPort=127.0.0.1:8001:8001

Volume=/etc/sysconfig/illumon.d/resources/envoy3.yaml:/config.yaml:rw
Volume=/etc/deephaven/certs/envoy.pem:/envoy.pem:ro

User=9002
Exec=-c /config.yaml

[Service]
Restart=always
TimeoutStopSec=70

[Install]
WantedBy=default.target

Be sure to check that you are using the correct Envoy version, that your necessary ports are exposed, and that your envoy.yaml and certificates are correctly mounted.

After creating the quadlet file, Deephaven will update systemd:

# reload systemd to generate deephaven_envoy.container
systemctl daemon-reload
# no need to enable the service as it is generated
systemctl start deephaven_envoy.service

The envoy3.yaml should specify the certificate location it expects within the container; in this example, /envoy.pem.

Example Docker service

When DH_ENVOY_INSTALL_MODE=DOCKER, Deephaven will generate a systemd service file which attaches directly to the invocation of docker run --rm. This allows systemd to immediately detect and respond to process failures.

The sample systemd service unit below uses the default Envoy port and Envoy admin port, run as the default Admin service account, using default locations for envoy.yaml and Envoy certificates.

/etc/systemd/system/deephaven_envoy.service

[Unit]
Description=Envoy proxy server (deephaven_envoy)
Wants=network-online.target
After=network-online.target

[Service]
Restart=always
RestartSec=5
TimeoutStopSec=70

# Ensure we don't fail due to a stale container name
ExecStartPre=/usr/bin/docker rm -f deephaven_envoy

ExecStart=/usr/bin/docker run \
--name deephaven_envoy \
--rm \
-p 8000:8000 \
-p 8001:8001 \
-v /etc/sysconfig/illumon.d/resources/envoy3.yaml:/config.yaml:rw \
-v /etc/deephaven/certs/envoy.pem:/envoy.pem:ro \
-u 9002 \
docker.io/envoyproxy/envoy:v1.36.4 \
-c /config.yaml

ExecStop=/usr/bin/docker stop --timeout 10 deephaven_envoy

[Install]
WantedBy=multi-user.target

Be sure to check that you are using the correct Envoy version, necessary ports are exposed, and that your envoy.yaml and certificates are correctly mounted.

After creating the service file, Deephaven will reload systemd:

# Tell systemd to process our new service
systemctl daemon-reload
# Enable and start the service.
systemctl enable --now deephaven_envoy.service

The envoy3.yaml should specify the certificate location it expects within the container; in this example, /envoy.pem.

Example native Envoy installation

When using DH_ENVOY_INSTALL_MODE=NATIVE, Deephaven will create a systemd service unit that does not require any root privileges and starts no containers. Deephaven directly downloads the Envoy binary from Envoy's GitHub releases page, and runs it as the default Administrative service account, irisadmin.

The sample systemd user unit service file below uses the default Admin service account username of irisadmin, and requires that you have downloaded an Envoy binary file to ~irisadmin/bin/envoy-1.36.4-linux-x86_64.

~irisadmin/.config/systemd/user/deephaven_envoy.service

[Unit]
Description=Envoy proxy server (deephaven_envoy) - native binary
Wants=network-online.target
After=network-online.target

[Service]
Restart=always
RestartSec=5
TimeoutStopSec=70

# Run Envoy using the installer-managed config + cert locations.
ExecStart=%h/bin/envoy-1.36.4-linux-x86_64 -c /etc/sysconfig/illumon.d/resources/envoy3.yaml
WorkingDirectory=%h

[Install]
WantedBy=default.target

When setting up a native process in systemd, you do not need to worry about port mapping or volume mounting, but you must be sure that your envoy.yaml uses the correct ports and paths to your TLS signing certificate.

After creating the user service file, Deephaven will enable and start the service using systemctl:

sudo -u irisadmin bash -c '
  # This variable must be set to use systemd user units
  export XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}
  # Tell systemd to process our service file
  systemctl --user daemon-reload
  # Enable and start the unit
  systemctl --user enable --now deephaven_envoy.service
'

In order to use systemd user units, special care must be taken to set the XDG_RUNTIME_DIR, especially when sudo is used:

# systemd user unit operations require extra care:
sudo -u irisadmin XDG_RUNTIME_DIR=/run/user/$(id -u irisadmin) \
      systemctl --user status deephaven_envoy
# Journalctl does not require --user flags or sudo -u
journalctl -eu deephaven_envoy

While this installation method is a little harder to use once installed, it requires no special permissions.

Manual Envoy installation

There are many documented installation methods that are maintained by the Envoy team.

You may choose any of those options, and should explicitly set DH_ENVOY_INSTALL_MODE=NONE; even though NONE is the default installation mode, it is best to be explicit about your configuration.

However you choose to install Envoy, use envoy --version to verify you are using the Deephaven-tested Envoy version listed in the product support matrix.

For example, on Ubuntu systems, if you follow Envoy's installation instructions, make sure to specify the Envoy version when installing with apt-get:

sudo apt-get update
sudo apt-get install envoy=1.36.4
envoy --version

Manual Envoy upgrade

In order to manually upgrade Envoy, you should first Uninstall Envoy and then Reinstall Envoy.

Uninstall Envoy

If you are upgrading a system that already has Manually-managed Envoy, before you turn on Deephaven managed Envoy, you should first remove or at least disable your previous Envoy installation.

Tip

Podman and Kubernetes installations embed the Envoy binary in container images and do not require uninstallation.

If you change which machine runs Envoy, you must manually uninstall Envoy from the previous machine using the instructions below.

If you installed Envoy using Deephaven documentation from previous versions, you may need to remove both a running container and a systemd service unit.

The first step to uninstalling Envoy is to determine the installation method used.

Once you have determined how Envoy is installed, you can decide how to uninstall it.

For containerized installation:

docker stop deephaven_envoy
docker rm deephaven_envoy
#or
podman stop deephaven_envoy
podman rm deephaven_envoy

Many Envoy installation methods include a systemd service that must be stopped and disabled.

# Most installation methods include a root systemd `deephaven_envoy` service
systemctl disable --now deephaven_envoy

# When using DH_ENVOY_INSTALL_MODE=NATIVE, Envoy is installed as a systemd user unit.
# The user unit service runs as DH_ADMIN_USER (irisadmin), and you must take extra steps to use systemctl:
source /usr/illumon/latest/bin/dh_users
sudo -u $DH_ADMIN_USER XDG_RUNTIME_DIR=/run/user/$(id -u $DH_ADMIN_USER) \
    systemctl --user disable --now deephaven_envoy

Once uninstallation is complete, verify that Envoy is no longer running:

ps -ef | grep -v grep | grep envoy

Restarting Envoy

For Kubernetes installations, Envoy is restarted with the scaleAll.sh script.

For podman installations, Envoy can be restarted via podman exec -it dh-infra monit restart envoy.

For native installations, how you restart Envoy depends on the ownership model:

For Deephaven-managed DOCKER and PODMAN modes, restart the deephaven_envoy systemd root service:
```
sudo systemctl restart deephaven_envoy
```

For Deephaven managed NATIVE mode, restart the system user service:

# if you use custom service account names, replace irisadmin with your DH_ADMIN_USER
sudo -u irisadmin XDG_RUNTIME_DIR=/run/user/$(id -u irisadmin) \
   systemctl --user restart deephaven_envoy

For DH_ENVOY_INSTALL_MODE=NONE, use whatever restart method your manual deployment requires.

Caution

A full installation or upgrade of your cluster will overwrite configuration and certificate files and requires all processes, including Envoy, to be restarted. Using dh_keygen.sh to rotate certificates will update certificates but does not restart Envoy or Deephaven services; you will need to manually restart Envoy yourself.

When Deephaven-managed Envoy is used, full installations will restart Envoy automatically.

If you have only changed envoy.yaml, you can hot reload the Envoy configuration to load the new configuration and drain existing connections without restarting.