--- title: Installing Envoy sidebar_label: Installation --- This guide provides instructions for installing, upgrading, and restarting Envoy as Deephaven's front proxy in all three supported [Deephaven installation types](../admin-glossary.md#deephaven-installation-type). Instructions will be given for Kubernetes and Podman first, as they have a much simpler operational model. ## Install Envoy on Kubernetes Envoy is always used with Kubernetes. All you need to do is [supply a signing certificate](../kubernetes/kubernetes-install-guide.md#create-a-kubernetes-secret-for-the-tls-certificate) and [configure the hostname and port](../kubernetes/kubernetes-configuration-settings.md#configuring-envoy) where [Deephaven clients](../admin-glossary.md#deephaven-client) can reach the exposed Envoy Kubernetes service. ## Install Envoy on Podman Podman installation supports using Envoy by passing the `--envoy|-e` flag to [`start_command.sh`](../installation/podman-installation.md#start_commandsh-details) when creating your pod. If you build custom images, you must [provide the Envoy binary to `build_images.sh`](../installation/podman-installation.md#dh-infra). By default, Envoy on Podman will [use the same TLS certificate](../installation/podman-installation.md#supplying-signing-certificates-to-podman) as all other processes on the `dh-infra` pod. You may optionally place a PEM-formatted `envoy.pem` bundle containing the certificate chain and private key in the `--tls-volume` directory passed to [`start_command.sh`](../installation/podman-installation.md#start_commandsh-details). ## Install Envoy on native installations In order to use Envoy when installing natively, you must [enable Envoy in `cluster.cnf`](../installation/cluster-config-guide.md#envoy-properties), supply a [TLS signing certificate](../installation/cluster-config-guide.md#dh_envoy_local_cert), and [declare how Envoy is installed](../installation/cluster-config-guide.md#dh_envoy_install_mode). The minimum required configuration to use Envoy without root permission or containers: ```properties DH_CONFIGURE_ENVOY=true DH_ENVOY_INSTALL_MODE=NATIVE # PODMAN or DOCKER options are also available. ``` If you are using default values for all other properties, the only other requirement is to supply your [X.509 PEM-formatted](../admin-glossary.md#certificate-x509) certificate chain and private key bundle on the Installation host at [`DH_LOCAL_DIR/certs/envoy.pem`](../installation/cluster-config-guide.md#dh_envoy_local_cert). Once you run the native installer, you can connect to Deephaven via Envoy at your configured `https://` [DH_ENVOY_FQDN](../installation/cluster-config-guide.md#dh_envoy_fqdn):[DH_ENVOY_PORT](../installation/cluster-config-guide.md#dh_envoy_port). This minimal example assumes that you use [Deephaven-managed Envoy installation](#deephaven-managed-envoy) to automatically install, update, and restart the Envoy reverse proxy. If you choose to [Manually manage Envoy](#manually-managed-envoy), your system administrator must install and manage the Envoy process. ## Manually-managed Envoy By default, Envoy is neither installed nor restarted when the native installer runs, due to [`DH_ENVOY_INSTALL_MODE`](../installation/cluster-config-guide.md#dh_envoy_install_mode) having a default value of `NONE`. This policy ensures smooth upgrades for customers who have already set up Envoy on their own. In the manually managed scenario, the Deephaven native installer [configures the cluster to speak to Envoy](configuring-envoy.md#envoy-configuration-properties) for you, and generates the appropriate [`envoy.yaml`](configuring-envoy.md#envoy-yaml-file) for Envoy to use, but it does not install or restart Envoy. Your system administrator will need to [manually install Envoy](#envoy-installation-methods), restart it whenever you perform minor upgrades of Deephaven, and upgrade the Envoy version during major version upgrades of Deephaven. ### Deephaven-managed Envoy When [`DH_CONFIGURE_ENVOY`=true](../installation/cluster-config-guide.md#dh_configure_envoy), the installer can manage Envoy using the [`DH_ENVOY_INSTALL_MODE`](../installation/cluster-config-guide.md#dh_envoy_install_mode) options: - `DOCKER` / `PODMAN`: pulls and runs the Envoy image and mounts the generated YAML and the Envoy endpoint certificate PEM. It creates a systemd service unit called `deephaven_envoy` to simplify process restarts and integrate with `journalctl` logging. - `NATIVE`: installs Envoy as a native systemd user service run by `DH_ADMIN_USER`. - `NONE`: does not install, configure, or restart Envoy. Select a single node within the Deephaven cluster to run Envoy and ensure it has network access to the Deephaven services it will front. > [!NOTE] > Before you begin, check the [Deephaven version support matrix](../../reference/supported-versions.md#envoy-versions) to identify the Envoy version compatible with your Deephaven release. ## Envoy frontend URL All [Deephaven clients](../admin-glossary.md#deephaven-client) connect to Envoy through a configured [fully qualified domain name](../admin-glossary.md#fqdn) and port. The method to supply the Envoy frontend URL varies based on [Deephaven installation type](../admin-glossary.md#deephaven-installation-type). Kubernetes installations set [`envoyFrontProxyUrl`](../kubernetes/kubernetes-configuration-settings.md#configuring-envoy) to specify both hostname and port in `values.yaml`. Podman installations supply [`-h|--hostname -e|--envoy ` to `start_command.sh`](../installation/podman-installation.md#start_commandsh-details). Native installations set [`DH_ENVOY_FQDN`](../installation/cluster-config-guide.md#dh_envoy_fqdn) and [`DH_ENVOY_PORT`](../installation/cluster-config-guide.md#dh_envoy_port) in `cluster.cnf`. The Envoy frontend URL forms the [machine identity](../admin-glossary.md#machine-identity), which all [Deephaven clients must trust](../security/tls-certificates.md#external-tls-trust) when making TLS connections. ## Required TLS certificate Envoy requires a [TLS signing certificate and private key](../security/tls-certificates.md#required-tls-certificates), provided as follows: - [**Native Installation:**](installing-envoy.md#install-envoy-on-native-installations) provide [`DH_ENVOY_LOCAL_CERT`](../installation/cluster-config-guide.md#dh_envoy_local_cert) on the Installer host, by default `DH_LOCAL_DIR/certs/envoy.pem`. - [**Podman Installation:**](installing-envoy.md#install-envoy-on-podman) provide `envoy.pem` in the `--tls-volume` directory passed to [`start_command.sh`](../installation/podman-installation.md#start_commandsh-details). - [**Kubernetes Installation:**](installing-envoy.md#install-envoy-on-kubernetes) provide the [`deephaven-tls` Kubernetes secret](../kubernetes/kubernetes-configuration-settings.md#configuring-envoy). The Envoy certificate must include a [subject alternative name (SAN)](../admin-glossary.md#san) entry [matching the Envoy frontend URL](#envoy-frontend-url) which [Deephaven clients](../admin-glossary.md#deephaven-client) use to reach Envoy. [Hostname verification](../admin-glossary.md#hostname-verification) is SAN-based; neither certificate Common Names nor authority overrides are used. The provided Envoy certificate is verified by clients using an [optional truststore file](../security/tls-certificates.md#optional-truststore), or using the client's [system truststore](../admin-glossary.md#system-trust-vs-a-custom-truststore) ## Envoy installation methods > [!TIP] > This section is only relevant to native installation methods. > [Podman](#install-envoy-on-podman) and [Kubernetes](#install-envoy-on-kubernetes) install Envoy into container images. While [many Envoy installation methods](https://www.envoyproxy.io/docs/envoy/latest/start/install) exist, Deephaven recommends using [Deephaven-managed `PODMAN`](#deephaven-managed-envoy) if available, as [Podman quadlets](https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html) offer the most elegant systemd integration. On Ubuntu installations, `DOCKER` has good integration with systemd, while `NATIVE` is harder to use, but works without a container or root permission. The following sections provide instructions for common installation methods. ### Example Podman quadlet When `DH_ENVOY_INSTALL_MODE=PODMAN`, Deephaven will generate a [Podman quadlet](https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html) file. The sample Podman quadlet file below is configured with default [Envoy port](../installation/cluster-config-guide.md#dh_envoy_port) and [Envoy admin port](../installation/cluster-config-guide.md#dh_envoy_admin_port), run as the default [Admin service account](../installation/cluster-config-guide.md#dh_admin_user), using default locations for [`envoy.yaml`](configuring-envoy.md#envoy-yaml-file) and [Envoy certificates](../installation/basic-install.md#tls-certificates). `/etc/containers/systemd/deephaven_envoy.container` ```properties [Unit] Description=Envoy proxy server (deephaven_envoy) Wants=network-online.target After=network-online.target [Container] ContainerName=deephaven_envoy Image=docker.io/envoyproxy/envoy:v1.36.4 PublishPort=8000:8000 # this example only opens the sensitive admin port to localhost connections on the container's host machine PublishPort=127.0.0.1:8001:8001 Volume=/etc/sysconfig/illumon.d/resources/envoy3.yaml:/config.yaml:rw Volume=/etc/deephaven/certs/envoy.pem:/envoy.pem:ro User=9002 Exec=-c /config.yaml [Service] Restart=always TimeoutStopSec=70 [Install] WantedBy=default.target ``` Be sure to check that you are using the [correct Envoy version](../../reference/supported-versions.md#envoy-versions), that your [necessary ports are exposed](configuring-envoy.md#envoy-ports), and that your `envoy.yaml` and certificates are correctly mounted. After creating the `quadlet` file, Deephaven will update systemd: ```bash # reload systemd to generate deephaven_envoy.container systemctl daemon-reload # no need to enable the service as it is generated systemctl start deephaven_envoy.service ``` The `envoy3.yaml` should specify the certificate location it expects within the container; in this example, `/envoy.pem`. ### Example Docker service When `DH_ENVOY_INSTALL_MODE=DOCKER`, Deephaven will generate a systemd service file which attaches directly to the invocation of `docker run --rm`. This allows systemd to immediately detect and respond to process failures. The sample systemd service unit below uses the default [Envoy port](../installation/cluster-config-guide.md#dh_envoy_port) and [Envoy admin port](../installation/cluster-config-guide.md#dh_envoy_admin_port), run as the default [Admin service account](../installation/cluster-config-guide.md#dh_admin_user), using default locations for [`envoy.yaml`](configuring-envoy.md#envoy-yaml-file) and [Envoy certificates](../installation/basic-install.md#tls-certificates). `/etc/systemd/system/deephaven_envoy.service` ```properties [Unit] Description=Envoy proxy server (deephaven_envoy) Wants=network-online.target After=network-online.target [Service] Restart=always RestartSec=5 TimeoutStopSec=70 # Ensure we don't fail due to a stale container name ExecStartPre=/usr/bin/docker rm -f deephaven_envoy ExecStart=/usr/bin/docker run \ --name deephaven_envoy \ --rm \ -p 8000:8000 \ -p 8001:8001 \ -v /etc/sysconfig/illumon.d/resources/envoy3.yaml:/config.yaml:rw \ -v /etc/deephaven/certs/envoy.pem:/envoy.pem:ro \ -u 9002 \ docker.io/envoyproxy/envoy:v1.36.4 \ -c /config.yaml ExecStop=/usr/bin/docker stop --timeout 10 deephaven_envoy [Install] WantedBy=multi-user.target ``` Be sure to check that you are using the [correct Envoy version](../../reference/supported-versions.md#envoy-versions), [necessary ports are exposed](configuring-envoy.md#envoy-ports), and that your `envoy.yaml` and certificates are correctly mounted. After creating the `service` file, Deephaven will reload systemd: ```bash # Tell systemd to process our new service systemctl daemon-reload # Enable and start the service. systemctl enable --now deephaven_envoy.service ``` The `envoy3.yaml` should specify the certificate location it expects within the container; in this example, `/envoy.pem`. ### Example native Envoy installation When using [`DH_ENVOY_INSTALL_MODE=NATIVE`](../installation/cluster-config-guide.md#dh_envoy_install_mode), Deephaven will create a systemd service unit that does not require any root privileges and starts no containers. Deephaven directly downloads the Envoy binary from [Envoy's GitHub releases](https://github.com/envoyproxy/envoy/releases) page, and runs it as the default [Administrative service account](../installation/cluster-config-guide.md#dh_admin_user), `irisadmin`. The sample systemd user unit service file below uses the default [Admin service account](../installation/cluster-config-guide.md#dh_admin_user) username of `irisadmin`, and requires that you have downloaded an Envoy binary file to `~irisadmin/bin/envoy-1.36.4-linux-x86_64`. `~irisadmin/.config/systemd/user/deephaven_envoy.service` ```properties [Unit] Description=Envoy proxy server (deephaven_envoy) - native binary Wants=network-online.target After=network-online.target [Service] Restart=always RestartSec=5 TimeoutStopSec=70 # Run Envoy using the installer-managed config + cert locations. ExecStart=%h/bin/envoy-1.36.4-linux-x86_64 -c /etc/sysconfig/illumon.d/resources/envoy3.yaml WorkingDirectory=%h [Install] WantedBy=default.target ``` When setting up a native process in systemd, you do not need to worry about port mapping or volume mounting, but you must be sure that your [`envoy.yaml`](configuring-envoy.md#envoy-yaml-file) uses the correct ports and paths to your TLS signing certificate. After creating the user service file, Deephaven will enable and start the service using `systemctl`: ```bash sudo -u irisadmin bash -c ' # This variable must be set to use systemd user units export XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)} # Tell systemd to process our service file systemctl --user daemon-reload # Enable and start the unit systemctl --user enable --now deephaven_envoy.service ' ``` In order to use systemd user units, special care must be taken to set the `XDG_RUNTIME_DIR`, especially when `sudo` is used: ```bash # systemd user unit operations require extra care: sudo -u irisadmin XDG_RUNTIME_DIR=/run/user/$(id -u irisadmin) \ systemctl --user status deephaven_envoy # Journalctl does not require --user flags or sudo -u journalctl -eu deephaven_envoy ``` While this installation method is a little harder to use once installed, it requires no special permissions. ### Manual Envoy installation There are many [documented installation methods](https://www.envoyproxy.io/docs/envoy/latest/start/install) that are maintained by the Envoy team. You may choose any of those options, and should explicitly set [`DH_ENVOY_INSTALL_MODE=NONE`](../installation/cluster-config-guide.md#dh_envoy_install_mode); even though `NONE` is the default installation mode, it is best to be explicit about your configuration. However you choose to install Envoy, use `envoy --version` to verify you are using the [Deephaven-tested Envoy version](../../reference/supported-versions.md#envoy-versions) listed in the product support matrix. For example, on Ubuntu systems, if you follow [Envoy's installation instructions](https://www.envoyproxy.io/docs/envoy/latest/start/install#install-envoy-on-debian-based-linux), make sure to specify the Envoy version when installing with `apt-get`: ```bash sudo apt-get update sudo apt-get install envoy=1.36.4 envoy --version ``` ### Manual Envoy upgrade In order to manually upgrade Envoy, you should first [Uninstall Envoy](#uninstall-envoy) and then [Reinstall Envoy](#manual-envoy-installation). ## Envoy Health Check Deephaven managed Envoy installations expose a health check endpoint on the public Envoy listener at `/healthz`. This is also part of the generated `envoy3.yaml` configuration file for manual installations. This endpoint only tells you about Envoy's health, not the health of the Deephaven services behind it. It is served by Envoy's [`envoy.filters.http.health_check`](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/health_check_filter) filter. Envoy responds with `200` while healthy and `503` once it begins draining or its own health flags fail. ## Uninstall Envoy If you are upgrading a system that already has [Manually-managed Envoy](#manually-managed-envoy), before you turn on [Deephaven managed Envoy](#deephaven-managed-envoy), you should first remove or at least disable your previous Envoy installation. > [!TIP] > Podman and Kubernetes installations embed the Envoy binary in container images and do not require uninstallation. If you change [which machine runs Envoy](../installation/cluster-config-guide.md#dh_node_n_role_envoy), you must manually uninstall Envoy from the previous machine using the instructions below. If you installed Envoy using Deephaven documentation from previous versions, you may need to remove both a running container and a `systemd` service unit. The first step to uninstalling Envoy is to [determine the installation method used](troubleshooting-envoy.md#determine-envoy-installation-method). Once you have determined how Envoy is installed, you can decide how to uninstall it. For containerized installation: ```bash docker stop deephaven_envoy docker rm deephaven_envoy #or podman stop deephaven_envoy podman rm deephaven_envoy ``` Many Envoy installation methods include a systemd service that must be stopped and disabled. ```bash # Most installation methods include a root systemd `deephaven_envoy` service systemctl disable --now deephaven_envoy # When using DH_ENVOY_INSTALL_MODE=NATIVE, Envoy is installed as a systemd user unit. # The user unit service runs as DH_ADMIN_USER (irisadmin), and you must take extra steps to use systemctl: source /usr/illumon/latest/bin/dh_users sudo -u $DH_ADMIN_USER XDG_RUNTIME_DIR=/run/user/$(id -u $DH_ADMIN_USER) \ systemctl --user disable --now deephaven_envoy ``` Once uninstallation is complete, verify that Envoy is no longer running: ```bash ps -ef | grep -v grep | grep envoy ``` ## Restarting Envoy For Kubernetes installations, Envoy is restarted with the [`scaleAll.sh` script](../kubernetes/troubleshooting-kubernetes.md#how-do-i-scale-my-deephaven-deployment-down-or-up). For podman installations, Envoy can be restarted via `podman exec -it dh-infra monit restart envoy`. For native installations, how you restart Envoy depends on the ownership model: - For [Deephaven-managed](#deephaven-managed-envoy) `DOCKER` and `PODMAN` modes, restart the `deephaven_envoy` systemd root service: ```bash sudo systemctl restart deephaven_envoy ``` - For [Deephaven managed](#deephaven-managed-envoy) `NATIVE` mode, restart the system user service: ```bash # if you use custom service account names, replace irisadmin with your DH_ADMIN_USER sudo -u irisadmin XDG_RUNTIME_DIR=/run/user/$(id -u irisadmin) \ systemctl --user restart deephaven_envoy ``` - For `DH_ENVOY_INSTALL_MODE=NONE`, use whatever [restart method](troubleshooting-envoy.md#restarting-envoy) your manual deployment requires. > [!CAUTION] > A full installation or upgrade of your cluster will overwrite configuration and certificate files and requires all processes, including Envoy, to be restarted. > Using `dh_keygen.sh` to [rotate certificates](../security/tls-certificates.md#rotate-certificates-for-native-installations) will update certificates but does **not** restart Envoy or Deephaven services; you will need to manually restart Envoy yourself. When [Deephaven-managed Envoy](#deephaven-managed-envoy) is used, full installations will restart Envoy automatically. If you have only changed [`envoy.yaml`](configuring-envoy.md#envoy-yaml-file), you can [hot reload the Envoy configuration](troubleshooting-envoy.md#reloading-envoy) to load the new configuration and drain existing connections without restarting. ## Related documentation - [Use Envoy as a front proxy](./envoy.md) - [Configuring Envoy](./configuring-envoy.md) - [Envoy configuration flags for native installations](../installation/cluster-config-guide.md#envoy-properties) - [Troubleshooting Envoy](./troubleshooting-envoy.md) - [TLS Certificates](../security/tls-certificates.md)