Configuring Envoy

After installing Envoy, the next step is to configure it to work with your Deephaven instance. This configuration enables Envoy to act as a secure and robust front proxy, intelligently routing incoming traffic to the appropriate backend Deephaven services.

Deephaven simplifies this process by integrating with Envoy's dynamic configuration APIs (known as xDS). This allows Deephaven to automatically publish its service locations and routing rules to Envoy, minimizing the need for manual updates.

This guide covers two methods for setting up this integration:

Automatic: The recommended method for most users. Deephaven manages the Envoy configuration for you.
Manual: For users who require a customized setup or are integrating Envoy into an existing infrastructure.

Select a tab below based on your needs.

When using Podman or Kubernetes, Envoy is configured automatically from the inputs you normally provide.

Note

The various DH_*ENVOY* cluster.cnf properties only apply to native installations of Deephaven. They are used to configure both Deephaven properties and Envoy YAML from a single source of truth (cluster.cnf).

Native installer Envoy properties

When using the native installer, Envoy-related settings are sourced from cluster.cnf and translated into generated configuration artifacts.

At a high level:

Deephaven service configuration is written into iris-endpoints.prop Deephaven properties file.
Envoy itself is configured via a generated /etc/sysconfig/illumon.d/resources/envoy3.yaml.
The legacy client URL and port are updated in /etc/sysconfig/illumon.d/client_update_service/getdown.global.

A variety of DH_*ENVOY* variables in cluster.cnf (plus the cluster topology / node roles) are used to control how Envoy is set up and run.

`cluster.cnf` property	Target file(s)	Notes
`DH_CONFIGURE_ENVOY`	`iris-endpoints.prop`	Causes the installer to update Deephaven properties to force clients to route through Envoy rather than connect directly to services.
`DH_ENVOY_FQDN`, `DH_ENVOY_PORT`	`getdown.global`	Used to construct the client-facing host/port that clients use to reach Deephaven via Envoy.
`DH_ENVOY_CLUSTER_TYPE`	`envoy3.yaml`	Chooses how the generated Envoy configuration resolves the Configuration Server address; by default, `STRICT_DNS` is used.
`DH_ENVOY_PORT`	`envoy3.yaml`	Sets the port Envoy listens on for incoming client connections.
`DH_ENVOY_ENABLE_ADMIN`, `DH_ENVOY_ADMIN_PORT`	`envoy3.yaml`	Controls whether the Envoy admin interface is included in the generated config, and which port it uses when enabled.
`DH_ENVOY_PEM_PATH`	`envoy3.yaml`	Sets the filesystem path Envoy uses for its TLS certificate/key PEM bundle.
`DH_ENVOY_LOCAL_CERT`	Installation host filesystem	Controls where the installer finds the PEM to copy to the Envoy host.
`DH_ENVOY_REMOTE_CERT`	Envoy node filesystem	Controls where the installer places the PEM bundle on the Envoy host.

Instead of maintaining iris-endpoints.prop, getdown.global, or envoy3.yaml manually (and re-applying changes on every upgrade), set the Envoy-related cluster.cnf properties, place the required certificate files on the Installation host, and re-run the native installer scripts.

With this approach, upgrades and reconfiguration can safely regenerate the various property files and YAML to match your current cluster topology.

Manual Envoy configuration

Manual configuration requires two main steps: creating an Envoy configuration file and updating several Deephaven property files to ensure all services can communicate through the proxy.

An example Envoy YAML is given below.

Manual Envoy YAML configuration

Create your own `envoy.yaml`

When manually configuring Envoy, the location of the envoy.yaml file matters only to the running Envoy process, which reads it when the proxy starts. The default location used by the Deephaven installer is /etc/sysconfig/illumon.d/resources/envoy3.yaml.

Caution

The native installer will automatically create the file /etc/sysconfig/illumon.d/resources/envoy3.yaml when DH_CONFIGURE_ENVOY=true. This file is altered to match the current network topology defined in cluster.cnf. If you make any changes to this file, those changes will be lost every time you upgrade Deephaven. You can disable this by setting DH_ENVOY_COPY_YAML=false, but you will be responsible for maintaining your Envoy configuration.

Whenever editing Envoy YAML, always consult the Envoy documentation.

A sample envoy.yaml file is given below.

Static resources

Clusters: In the static_resources section, configure the target address and port on which the Configuration Service listens for xDS requests. If the Configuration Service is on a different node than Envoy, or if Envoy is in a Docker container, change this to the appropriate address. The port must match the Deephaven system property envoy.xds.port, which defaults to 8124.
Listeners: The listeners section tells Envoy what addresses and ports to listen on for forwarding. In the example, 0.0.0.0 and port 8000 tell Envoy to listen for connections from any interface on port 8000.

TLS certificates

Manually configured Envoy installations must supply TLS certificates in the same manner as automatic Envoy configuration.

Administration Interface

Administrators can set up the Envoy administration port to inspect the running configuration. The example Envoy YAML enables this on port 8001. Be careful with this in production, as it allows access to internal Envoy configuration.

Configure Deephaven to route through Envoy

Altering how Deephaven routes client requests requires changes to Deephaven properties in iris-environment.prop, iris-endpoints.prop, and getdown.global.

Note

Once Envoy is enabled for Deephaven, all services must be configured to use it, or the Configuration Server service will fail to start.

Tip

All customizations described below are done automatically with the native installer when DH_CONFIGURE_ENVOY=true.

Update `iris-environment.prop`

Export, edit, and reimport this file:

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties export --etcd -f iris-environment.prop -d /tmp/
sudo -u irisadmin vi /tmp/iris-environment.prop
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import --etcd -f iris-environment.prop -d /tmp/

Add the following properties:

# Enables the WebSocket server, which is required for Envoy to proxy WebSocket traffic.
global.websocket.server.enabled=true

# The public-facing URL of the Envoy proxy. This is used by services to construct correct URLs.
envoy.front.proxy.url=user-facing-host.company.com:8000

# The Configuration Server needs to know the host of the Web API server.
Webapi.server.host=web-api-host.company.com

# This section configures Swing clients (e.g., Iris Console) to connect through Envoy.
[service.name=iris_console|interactive_console] {
    # Enables the WebSocket client for Swing applications.
    global.websocket.client.enabled=true
    # Points the client to the Envoy proxy for authentication.
    authentication.server.list=user-facing-host.company.com
    authentication.server.port.ssl=8000
    authentication.server.overrideAuthority=

    # Configure how remote Deephaven clients will verify TLS connections to Deephaven servers.
    # Truststore can be left blank if Envoy's certificate is trusted by the client's system.
    #tls.truststore=
    # if you provided an explicit truststore, instead uncomment below:
    #tls.truststore=truststore-iris.p12
}

# Each service that uses WebSockets needs a unique port assigned for internal communication.
# This port is used by the RemoteQueryDispatcher.
RemoteQueryDispatcherParameters.websocket.port=22052

[service.name=dbmerge|db_dis_merge|tailer1_merge] {
    RemoteQueryDispatcherParameters.websocket.port=22060
}

Take special note of the tls.truststore= section in the above YAML.

This value can vary depending on how your administrator configured TLS trust for your cluster, and should already be configured automatically.

Update `iris-endpoints.prop`

Services like the Persistent Query Controller need to know about all dispatchers. Add configuration_server to the iris_controller stanza and define the websocket ports for merge servers.

Warning

iris-endpoints.prop is regenerated and reimported automatically every time the native installer upgrades your system. The values below are automatically added, are rarely changed, and will be overwritten on every upgrade. If there is a value that you want to change which is overwritten by values from cluster.cnf, instead edit your cluster.cnf so the installation process is aware of your changes.

Export, edit, and reimport the file:

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties export --etcd -f iris-endpoints.prop -d /tmp/
sudo -u irisadmin vi /tmp/iris-endpoints.prop
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import --etcd -f iris-endpoints.prop -d /tmp/

Example for a three-node cluster:

# The iris_controller and configuration_server must be aware of all query and merge nodes.
[service.name=iris_controller|configuration_server] {
    iris.db.nservers=3

    iris.db.1.host=query-1.company.com
    iris.db.1.class=Query

    iris.db.2.host=query-2.company.com
    iris.db.2.class=Query

    iris.db.3.host=infra-1.company.com
    iris.db.3.port=30002
    iris.db.3.class=Merge
    # Define the websocket port for the merge server.
    iris.db.3.websocket.port=22060
}

Note

The iris-endpoints.prop file is regenerated by the Deephaven installer. After any reconfiguration or upgrade, you must reapply any modifications.

Update `getdown.global`

Update /etc/sysconfig/illumon.d/getdown.global to tell the Web UI where to find the server:

# This property tells the Web UI client the public URL to connect back to the server via Envoy.
opt=-Ddeephaven.web.client.publicUrl=https://user-facing-host.company.com:8000

Update /etc/sysconfig/illumon.d/client_update_service/getdown.global so the Swing client reads properties from disk:

# This forces the Swing client to use local property files instead of fetching them from the server.
# This is necessary when connecting through a proxy like Envoy.
jvmarg = -Dcom.fishlib.configuration.PropertyInputStreamLoader.override=com.fishlib.configuration.PropertyInputStreamLoaderTraditional

Note

The changes to getdown.global are done automatically when using the Deephaven installer. The Deephaven-supplied getdown.txt.pre contains the token #<Envoy properties loader property> which is automatically translated into the appropriate jvmarg. If you make any customizations to property loading, you may need to reapply your changes after every upgrade.

Envoy YAML file

An example Envoy configuration YAML file is provided below.

Envoy Configuration File

# Every Envoy instance needs a node identifier.
node: { id: "envoynode", cluster: "envoycluster" }

# This section tells Envoy that there is a dynamic cluster discovery service, 'xds_service',
# that communicates via gRPC using the V3 API.
dynamic_resources:
  # cds_config tells Envoy how to get Cluster Discovery Service updates.
  # Clusters are the backend services that Envoy can route traffic to.
  cds_config:
    resource_api_version: V3
    api_config_source:
      api_type: GRPC
      transport_api_version: V3
      grpc_services:
        envoy_grpc: { cluster_name: xds_service }

# This section defines the static cluster for the xDS service itself. In Deephaven's case,
# this is the single Configuration Server listening on port 8124.

# This configuration assumes the web_api_service is running on the same host as Envoy.
# If Envoy is instead running within a docker container, or another host the address should be updated.
static_resources:
  clusters:
    # This is the static cluster definition for the xDS service itself.
    # Envoy needs to know how to connect to the Deephaven Configuration Server.
    - name: xds_service
      connect_timeout: 0.25s
      type: STRICT_DNS
      typed_extension_protocol_options:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicit_http_config:
            http2_protocol_options: {}
      load_assignment:
        cluster_name: "xds_service"
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      # This must be the address of the Deephaven Configuration Server.
                      address: <config-server-hostname>
                      port_value: 8124

  listeners:
    # This section defines a listener that accepts incoming connections.
    - address:
        socket_address:
          address: 0.0.0.0
          port_value: 8000
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                codec_type: AUTO
                use_remote_address: true
                stat_prefix: egress_http
                request_timeout: 0s
                stream_idle_timeout: 600s
                upgrade_configs:
                  - upgrade_type: websocket
                # rds tells Envoy to get its route configuration dynamically from the xDS service.
                rds:
                  route_config_name: rds_config
                  config_source:
                    resource_api_version: V3
                    api_config_source:
                      api_type: GRPC
                      transport_api_version: V3
                      grpc_services:
                        envoy_grpc:
                          cluster_name: "xds_service"
                access_log:
                  - name: envoy.file_access_log
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                      path: "/tmp/envoy-rds.log"
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          # This configures TLS for the listener.
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              require_client_certificate: false
              common_tls_context:
                alpn_protocols: "h2,http/1.1"
                tls_certificates:
                  - certificate_chain:
                      filename: "/envoy.pem"
                    private_key:
                      filename: "/envoy.pem"

# This section configures the Envoy admin interface.
admin:
  access_log_path: "/tmp/envoy-admin.log"
  # Setting these will allow a sysadmin to dump Envoy configurations from localhost of the Envoy host.
  # We suggest you disable this in production, as the admin interface can be used to modify the running configuration.
  # Select one of the two cases below: without docker or with docker.
  #
  # Without docker: listen only on the localhost interface.
  # address:
  #  socket_address: { address: 127.0.0.1, port_value: 8001 }
  #
  # With docker, and being able to connect to the Envoy admin port from the host machine,
  # use the configuration below instead (potentially replacing 8001 with the port you prefer to use).
  # address:
  #  socket_address: { address: 0.0.0.0, port_value: 8001 }

Whenever you change the contents of envoy.yaml, you must notify the Envoy process for the changes to take effect.

The Envoy troubleshooting guide contains additional advice on how to configure Envoy application log levels, increase Envoy request logging, and more.

Envoy ports

Using Envoy as a reverse proxy allows you to isolate all internal network traffic behind one or two ports.

The exposed Envoy port is where all Deephaven client TLS connections will be made.

Native installations: the Envoy port is specified by DH_ENVOY_PORT.
Podman installations: the Envoy port is passed as the argument to the -e|--envoy flag passed to start_command.sh.
Kubernetes installations: the Envoy port is set as part of the envoyFrontProxyUrl in values.yaml.

In addition to the main Envoy port, when you need to debug your Envoy installation, you may need to expose the Envoy admin interface as well.

Because the admin interface exposes sensitive information and can modify your cluster's runtime behavior, you should be careful to expose this port to the narrowest possible audience. For example, using the admin interface using curl from the Envoy host machine means you do not need to expose the admin port to the network interfaces that Deephaven clients can reach.

Advanced Envoy configuration

In case you have additional needs for Envoy to function in your network, there are a variety of Deephaven properties that can be set to control the inner workings of the reverse proxy.

Any additional configuration needed for your networks should be well documented and stored in source control, so any customizations can be maintained as your network or needs change.

Custom HTTP Headers

You can add custom HTTP headers to responses from the Deephaven server. Deephaven has pre-defined headers (e.g., Strict-Transport-Security, X-Frame-Options) that can be enabled in iris-environment.prop:

envoy.add.header.Strict-Transport-Security.enabled=true
envoy.add.header.Strict-Transport-Security.value=max-age=31536000; includeSubDomains

You can also add custom headers that are not predefined in iris-defaults.prop. For each header, create a pair of properties to enable it and set its value:

envoy.add.header.Access-Control-Max-Age.enabled = true
envoy.add.header.Access-Control-Max-Age.value = 7200

Extra XDS Routes

Additional routes can be added to the Envoy configuration to expose other services. For each route, create a set of properties in iris-environment.prop:

envoy.xds.extra.routes.dis.host=infra-1.company-name.com
envoy.xds.extra.routes.dis.port=8086
envoy.xds.extra.routes.dis.prefix=/dis/
envoy.xds.extra.routes.dis.prefixRewrite=/
envoy.xds.extra.routes.dis.tls=false
envoy.xds.extra.routes.dis.exactPrefix=false

WebSocket Message Size

If you encounter errors that a WebSocket message size exceeds the maximum, you can increase the limit with the following property in iris-environment.prop:

websocket.client.max.message.size=100000000

Restart services after any configuration change

After making changes to Deephaven properties, those properties are cached by running processes, so it is the most reliable to restart all services on all nodes after any changes:

/usr/illumon/latest/bin/dh_monit restart all --block

If you need to perform selective service restarts in a production environment, consult a Deephaven representative about your specific use case.

Restarting Envoy is covered in the Envoy troubleshooting guide.

Envoy configuration properties

The following table summarizes the key Deephaven properties used when manually configuring Deephaven to work with Envoy.

These properties are all set automatically when using the native installer with DH_CONFIGURE_ENVOY=true.

Property	File(s)	Description
`global.websocket.server.enabled`	`iris-endpoints.prop`	Enables the WebSocket server required for proxying.
`envoy.front.proxy.url`	`iris-endpoints.prop`	The public-facing URL of the Envoy proxy.
`Webapi.server.host`	`iris-endpoints.prop`	Specifies the host of the Web API server for the Configuration Server.
`global.websocket.client.enabled`	`iris-endpoints.prop`	Enables the WebSocket client for Swing applications.
`authentication.server.list`	`iris-endpoints.prop`	Points Swing clients to Envoy for authentication.
`appbase`	`getdown.global`	Configure the base URL Swing and legacy enterprise clients use to connect to Deephaven.
`websocket.client.max.message.size`	`iris-endpoints.prop`	Increases the maximum WebSocket message size to prevent errors.
`envoy.add.header.*`	`iris-endpoints.prop`	Enables and sets custom HTTP headers in responses.
`envoy.xds.extra.routes.*`	`iris-endpoints.prop`	Defines additional routes for exposing other services through Envoy.

Security Considerations

When deploying Envoy in a production environment, it is critical to consider the following security best practices:

TLS Certificates: The example configuration uses "a file mounted at /envoy.pem" for convenience. Your actual certificate must meet the TLS certificate requirements.
Admin Interface: The Envoy admin interface on port 8001 provides powerful debugging capabilities but also exposes sensitive configuration details. In a production environment, access to this port should be heavily restricted. Use firewall rules (e.g., iptables or your cloud provider's security groups) to ensure the admin port is accessible only from trusted IP addresses, such as a specific bastion host or an internal management network.
Network Security: Ensure that the network communication between Envoy and the backend Deephaven services is secure. If Envoy and Deephaven are running on different hosts, consider using a private network or firewall rules to limit traffic between them to only the necessary ports.

Configuring Envoy

Native installer Envoy properties

Manual Envoy configuration

Manual Envoy YAML configuration

Create your own envoy.yaml

Static resources

TLS certificates

Administration Interface

Configure Deephaven to route through Envoy

Update iris-environment.prop

Update iris-endpoints.prop

Update getdown.global

Envoy YAML file

Envoy ports

Advanced Envoy configuration

Custom HTTP Headers

Extra XDS Routes

WebSocket Message Size

Restart services after any configuration change

Envoy configuration properties

Security Considerations

Related documentation

Create your own `envoy.yaml`

Update `iris-environment.prop`

Update `iris-endpoints.prop`

Update `getdown.global`