Configuring Envoy
After installing Envoy, the next step is to configure it to work with your Deephaven instance. This configuration enables Envoy to act as a secure and robust front proxy, intelligently routing incoming traffic to the appropriate backend Deephaven services.
Deephaven simplifies this process by integrating with Envoy's dynamic configuration APIs (known as xDS). This allows Deephaven to automatically publish its service locations and routing rules to Envoy, minimizing the need for manual updates.
This guide covers two methods for setting up this integration:
- Automatic: The recommended method for most users. Deephaven manages the Envoy configuration for you.
- Manual: For users who require a customized setup or are integrating Envoy into an existing infrastructure.
Select a tab below based on your needs.
When using Podman or Kubernetes, Envoy is configured automatically from the inputs you normally provide.
Note
The various DH_*ENVOY* cluster.cnf properties only apply to native installations of Deephaven.
They are used to configure both Deephaven properties and Envoy yaml from a single source of truth (cluster.cnf).
Native installer Envoy properties
When configuring Envoy automatically, the installer sets the necessary properties in iris-endpoints.prop, adjusts getdown.global, generates envoy3.yaml, and writes the configured Envoy certificate path into that YAML. For installer-managed Envoy, the PEM bundle comes from DH_ENVOY_LOCAL_CERT and Envoy reads it from DH_ENVOY_PEM_PATH at runtime.
DH_CONFIGURE_ENVOY
The default value is false.
When set to true, the installation is configured to use Envoy for access to Deephaven services. The installer updates iris-endpoints.prop, generates envoy3.yaml, and can also install and manage Envoy depending on DH_ENVOY_INSTALL_MODE.
When DH_CONFIGURE_ENVOY is set to true, these additional flags control how Envoy is installed, how its certificate bundle is provided, and how the generated configuration is written.
DH_ENVOY_INSTALL_MODE
Controls whether the Deephaven installs and manages Envoy on the Envoy node.
NONE- Envoy remains manually managed (default).PODMAN- the installer manages an Envoy container undersystemd.- Recommended for Rocky and RedHat-based Operating Systems.
- Sets up a Podman quadlet for smooth systemd integration.
- Container lifecycle is managed by
systemctl <operation> deephaven_envoy.
DOCKER- the installer manages an Envoy container undersystemd.- Recommended for all Ubuntu and Debian-based Operating Systems
- Sets up a root systemd service that uses
docker run --rmto stay. directly attached to the container, for graceful restarts. - Container lifecycle is managed by
systemctl <operation> deephaven_envoy.
NATIVE- the installer manages a native Envoy service.- Downloads an Envoy binary from Envoy's GitHub releases page.
- Creates a systemd user service to run Envoy as DH_ADMIN_USER.
- Container lifecycle is managed by the DH_ADMIN_USER :
Warning
If you are upgrading a pre-existing Envoy-enabled cluster,
before you set DH_ENVOY_INSTALL_MODE to any value other than NONE,
make sure you Uninstall any manually-managed Envoy processes.
DH_GENERATE_CERTIFICATES
Only used when DH_CONFIGURE_ENVOY=true.
This flag defaults to true when Envoy is enabled. When DH_GENERATE_CERTIFICATES=true, Deephaven will generate per-node TLS certificates for every machine, so you only need to provide one certificate regardless of the size of your cluster.
When DH_GENERATE_CERTIFICATES=false, you must provide one certificate/key pair per machine, and each machine’s certificate must cover the value of DH_NODE_N_DOMAIN for that node.
DH_ENVOY_LOCAL_CERT
Path on the Installation Host to the Envoy certificate input.
This file must be a single concatenated PEM bundle containing:
- the certificate chain Envoy presents to clients, followed by
- the private key for that certificate.
DH_ENVOY_REMOTE_CERT
Path on the Envoy node where the installer copies the Envoy PEM bundle.
DH_ENVOY_PEM_PATH
Runtime path that Envoy reads, and the value written into the generated envoy3.yaml.
Default runtime paths:
NATIVE:/etc/deephaven/certs/envoy.pemDOCKER/PODMAN/ 'NONE':/envoy.pem
The FQDN settings below are used when configuring properties, generated YAML, and client-facing URLs so Envoy and Deephaven services can find each other.
Warning
When using Deephaven managed Envoy installation with PODMAN or DOCKER installation methods, the DH_ENVOY_PEM_PATH is mounted as a volume in your container, but only when Envoy is first installed. If you change this location, you will need to systemctl edit deephaven_envoy to alter this volume mount as well.
DH_ENVOY_FQDN
DH_ENVOY_FQDN determines the hostname that all Deephaven clients will use to connect to your cluster.
Defaults to the fully qualified domain name (FQDN) of the DH_NODE_N_ROLE_ENVOY node, which defaults to the first Web API node, if no _ROLE_ENVOY node has been specified.
The DH_ENVOY_FQDN is used to configure the envoy.front.proxy.url Deephaven property, which is what the product uses to determine whether Deephaven clients will be allowed to connect directly to Deephaven servers or not.
Additionally, DH_ENVOY_FQDN is written to /etc/deephaven/FRONTEND_HOST on every Deephaven server, while the DH_ENVOY_PORT (default 8000) is written to /etc/deephaven/FRONTEND_PORT. When Envoy is not used, DH_WEB_FQDN is written instead.
The FRONTEND_HOST and FRONTEND_PORT files can be used by Deephaven client processes running on the remote server to easily find a FQDN:port to connect to. For example, the Core+ Python client SessionManager class will use the FRONTEND_HOST and FRONTEND_PORT files if you do not specify your own URL to connect to.
DH_ENVOY_PORT
The port where the Envoy server receives Deephaven client traffic.
If set to 443, worker urls will omit the port, but you will require administrative privileges to use this port.
This value is also used, in conjunction with DH_ENVOY_FQDN to define the default host and port used by Deephaven client processes that are running on a Deephaven server.
DH_ENVOY_ENABLE_ADMIN
Whether to enable the Envoy administration page.
Caution
Opening the Envoy admin port exposes your cluster's information, stability, and integrity to your network. While it is a valuable debugging tool, be sure to enable it with minimum scope, and to disable it when you are done.
The admin interface must be enabled in Envoy YAML, Envoy must be restarted, and the relevant ports exposed before you can interact with the Envoy administration page.
DH_ENVOY_ADMIN_PORT
If DH_ENVOY_ENABLE_ADMIN=true, choose which port to host the Envoy administration server.
This value will be written into envoy.yaml, and if DH_ENVOY_INSTALL_MODE is used to install Envoy with Podman or Docker, the admin port will be opened through the container. If you later change the admin port, you may need to use systemctl edit deephaven_envoy to update the container with your desired port mapping. Make sure to systemctl daemon-reload when you are done editing.
DH_ENVOY_CLUSTER_TYPE="STRICT_DNS"
Determines how Envoy will use DNS. Options are:
STATIC- Use only IP Addresses for endpoints. WhenSTATICis used, the installer will set IP addresses instead of host names in the Envoy config yaml.STRICT_DNS- Make a DNS call each time an address needs to be resolved.LOGICAL_DNS- Make a DNS call once for an address and cache the results.
DH_ENVOY_COPY_YAML
Whether the installer should try to copy the generated envoy3.yaml Envoy configuration file to the host where Envoy will be running. The default and most common configuration is for Envoy to be running on the Web API node. If this setting is enabled, the installer will copy the file to /etc/sysconfig/illumon.d/resources on the configured node.
Tip
If Envoy is running on a separate system, the file copy step can be disabled by setting DH_ENVOY_COPY_YAML=false in the cluster.cnf file. An alternate location for Envoy can be specified by setting the DH_ENVOY_FQDN property in cluster.cnf.
Manual Envoy configuration
Manual configuration requires two main steps: creating an Envoy configuration file and updating several Deephaven property files to ensure all services can communicate through the proxy.
An example Envoy yaml is given, below.
Manual Envoy yaml configuration
Create your own envoy.yaml
When manually configuring Envoy, the location of the envoy.yaml only matters to the running Envoy process, who reads the file when the proxy is started. The default location used by the Deephaven installer is /etc/sysconfig/illumon.d/resources/envoy3.yaml.
Caution
The native installer will automatically create the file /etc/sysconfig/illumon.d/resources/envoy3.yaml when DH_CONFIGURE_ENVOY=true.
This file is altered to match the current network topology defined in cluster.cnf.
If you make any changes to this file, those changes will be lost every time you upgrade Deephaven.
You can disable this by setting DH_ENVOY_COPY_YAML=false, but you will be responsible for maintaining your Envoy configuration.
Whenever editing Envoy YAML, always consult the Envoy documentation.
A sample envoy.yaml file is given below.
Static resources
-
Clusters: In the
static_resourcessection, configure the targetaddressandporton which the Configuration Service listens for xDS requests. If the Configuration Service is on a different node than Envoy, or if Envoy is in a Docker container, change this to the appropriate address. The port must match the Deephaven system propertyenvoy.xds.port, which defaults to8124. -
Listeners: The
listenerssection tells Envoy what addresses and ports to listen on for forwarding. In the example,0.0.0.0and port8000tell Envoy to listen for connections from any interface on port 8000.
TLS Certificates
When using TLS, Envoy needs access to a single PEM bundle containing the certificate chain and private key. The example uses /envoy.pem, which matches the default container runtime path. For installer-managed deployments, the runtime path is controlled by DH_ENVOY_PEM_PATH.
The Envoy certificate must include a SAN entry matching the hostname clients use to reach Envoy. Hostname verification is SAN-based; the certificate Common Name is not used. For the full certificate model, see TLS Certificates.
Administration Interface
Administrators can set up the Envoy administration port to inspect the running configuration. The example Envoy YAML enables this on port 8001. Be careful with this in production, as it allows access to internal Envoy configuration.
Configure Deephaven to route through Envoy
Altering how Deephaven routes client requests requires changes to Deephaven properties in iris-environment.prop, iris-endpoints.prop, and getdown.global.
Note
Once Envoy is enabled for Deephaven, all services must be configured to use it, or the Configuration Server service will fail to start.
Tip
All customizations described below are done automatically with the native installer when DH_CONFIGURE_ENVOY=true.
Update iris-environment.prop
Export, edit, and reimport this file:
Add the following properties:
Take special note of the tls.truststore= section in the above YAML.
This value can vary depending on how your administrator configured TLS trust for your cluster, and should already be configured automatically.
Update iris-endpoints.prop
Services like the Persistent Query Controller need to know about all dispatchers. Add configuration_server to the iris_controller stanza and define the websocket ports for merge servers.
Warning
iris-endpoints.prop is regenerated and reimported automatically every time the native installer upgrades your system. The values below will be added automatically, are very rarely changed, and will be overwritten on every upgrade. If there is a value that you want to change which is overwritten by values from cluster.cnf, instead edit your cluster.cnf so the installation process is aware of your changes.
Export, edit, and reimport the file:
Example for a three-node cluster:
Note
The iris-endpoints.prop file is regenerated by the Deephaven installer. After any reconfiguration or upgrade, you must reapply any modifications.
Update getdown.global
Update /etc/sysconfig/illumon.d/getdown.global to tell the Web UI where to find the server:
Update /etc/sysconfig/illumon.d/client_update_service/getdown.global so the Swing client reads properties from disk:
Note
The changes to getdown.global are done automatically when using the Deephaven installer. If you make any customizations, you may need to re-apply those changes after every upgrade.
Envoy yaml file
An example Envoy configuration YAML file is provided below.
Envoy Configuration File
Whenever you change the contents of envoy.yaml, you must notify the Envoy process for the changes to take effect.
The Envoy troubleshooting guide contains additional advice on how to configure Envoy application log levels, increase Envoy request logging, and more.
Envoy ports
Using Envoy as a reverse proxy allows you to isolate all internal network traffic behind one or two ports.
The exposed Envoy port is where all Deephaven client TLS connections will be made.
In addition to this port, when you need to debug your Envoy installation, you may need to expose the Envoy admin interface as well.
Because the admin interface exposes sensitive information and can modify the runtime behavior of your cluster, you should be careful to expose this port to the narrowest audience possible. For example, using the admin interface using curl from the Envoy host machine means you do not need to expose the admin port to the network interfaces that Deephaven clients can reach.
For containerized installations, you may need to set the admin.address.socket_address to use IP 0.0.0.0 within your container, then adjust your container settings to expose your admin port (default 8001) to a port on the host machine which is not exposed to your network.
Advanced Envoy configuration
In case you have additional needs for Envoy to function in your network, there are a variety of Deephaven properties that can be set to control the inner workings of the reverse proxy.
Any additional configuration needed for your networks should be well-documented and stored in source control, so any customizations done can be maintained as your network or needs change.
Custom HTTP Headers
You can add custom HTTP headers to responses from the Deephaven server. Deephaven has pre-defined headers (e.g., Strict-Transport-Security, X-Frame-Options) that can be enabled in iris-environment.prop:
You can also add custom headers not pre-defined in iris-defaults.prop. For each header, create a pair of properties to enable it and set its value:
Extra XDS Routes
Additional routes can be added to the Envoy configuration to expose other services. For each route, create a set of properties in iris-environment.prop:
WebSocket Message Size
If you encounter errors that a WebSocket message size exceeds the maximum, you can increase the limit with the following property in iris-environment.prop:
Restart services after any configuration change
After making changes to Deephaven properties, those properties are cached by running processes, so it is the most reliable to restart all services on all nodes after any changes:
If need to perform selective service restarts in a production environment, consult a Deephaven representative about your specific use case.
Restarting Envoy is covered in the Envoy troubleshooting guide.
Envoy configuration properties
The following table summarizes the key Deephaven properties used when manually configuring Deephaven to work with Envoy.
These properties are all set automatically when using the native installer with DH_CONFIGURE_ENVOY=true.
| Property | File(s) | Description |
|---|---|---|
global.websocket.server.enabled | iris-environment.prop | Enables the WebSocket server required for proxying. |
envoy.front.proxy.url | iris-environment.prop | The public-facing URL of the Envoy proxy. |
envoy.terminate.ssl | iris-environment.prop | Informs Deephaven that Envoy is handling TLS termination. |
Webapi.server.host | iris-environment.prop | Specifies the host of the Web API server for the Configuration Server. |
global.websocket.client.enabled | iris-environment.prop | Enables the WebSocket client for Swing applications. |
authentication.server.list | iris-environment.prop | Points Swing clients to Envoy for authentication. |
opt=-Ddeephaven.web.client.publicUrl | getdown.global | Sets the public URL for the Web UI client to connect to the server. |
jvmarg | getdown.global | Forces the Swing client to use local property files. |
websocket.client.max.message.size | iris-environment.prop | Increases the maximum WebSocket message size to prevent errors. |
envoy.add.header.* | iris-environment.prop | Enables and sets custom HTTP headers in responses. |
envoy.xds.extra.routes.* | iris-environment.prop | Defines additional routes for exposing other services through Envoy. |
Security Considerations
When deploying Envoy in a production environment, it is critical to consider the following security best practices:
-
TLS Certificates: The example configuration uses
envoy.pemfor convenience. For any production system, use a properly signed certificate chain and private key whose SAN entries match the public Envoy hostname. -
Admin Interface: The Envoy admin interface on port
8001provides powerful debugging capabilities but also exposes sensitive configuration details. In a production environment, access to this port should be heavily restricted. Use firewall rules (e.g.,iptablesor your cloud provider's security groups) to ensure that the admin port is only accessible from trusted IP addresses, such as a specific bastion host or an internal management network. -
Network Security: Ensure that the network communication between Envoy and the backend Deephaven services is secure. If Envoy and Deephaven are running on different hosts, consider using a private network or firewall rules to limit traffic between them to only the necessary ports.