Configuring Envoy

After installing Envoy, the next step is to configure it to work with your Deephaven instance. This configuration enables Envoy to act as a secure and robust front proxy, intelligently routing incoming traffic to the appropriate backend Deephaven services.

Deephaven simplifies this process by integrating with Envoy's dynamic configuration APIs (known as xDS). This allows Deephaven to automatically publish its service locations and routing rules to Envoy, minimizing the need for manual updates.

This guide covers two methods for setting up this integration:

Automatic: The recommended method for most users. Deephaven manages the Envoy configuration for you.
Manual: For users who require a customized setup or are integrating Envoy into an existing infrastructure.

Select a tab below based on your needs.

When configuring Envoy automatically, the installer sets the necessary properties in the iris-endpoints.prop file and adjusts the Envoy settings in getdown.global to direct the Client Update Service to use Envoy. Additionally, the installer generates an Envoy configuration YAML file, named envoy3.yaml, and copies it to the Deephaven node where Envoy is installed.

This method is primarily addressed in the Deephaven installation documentation, specifically in the Installation and Upgrade Guide and in the cluster configuration documentation under the section titled DH_CONFIGURE_ENVOY.

Tip

If Envoy is running on a separate system, the file copy step can be disabled by setting DH_ENVOY_COPY_YAML=false in the cluster.cnf file. An alternate location for Envoy can be specified by setting the DH_ENVOY_FQDN property in cluster.cnf.tip

Manual configuration requires two main steps: creating an Envoy configuration file and updating several Deephaven property files to ensure all services can communicate through the proxy.

Step 1: Create the Envoy Configuration File

Create an Envoy YAML file named envoy3.yaml and place it in /etc/sysconfig/illumon.d/resources. An example Envoy configuration is provided below.

Envoy Configuration File

# Every Envoy instance needs a node identifier.
node: { id: 'envoynode', cluster: 'envoycluster' }

# This section tells Envoy that there is a dynamic cluster discovery service, 'xds_service',
# that communicates via gRPC using the V3 API.
dynamic_resources:
  # cds_config tells Envoy how to get Cluster Discovery Service updates.
  # Clusters are the backend services that Envoy can route traffic to.
  cds_config:
    resource_api_version: V3
    api_config_source:
      api_type: GRPC
      transport_api_version: V3
      grpc_services:
        envoy_grpc: { cluster_name: xds_service }

# This section defines the static cluster for the xDS service itself. In Deephaven's case,
# this is the single Configuration Server listening on port 8124.

# This configuration assumes the web_api_service is running on the same host as Envoy.
# If envoy is instead running within a docker container, or another host the address should be updated.
static_resources:
  clusters:
    # This is the static cluster definition for the xDS service itself.
    # Envoy needs to know how to connect to the Deephaven Configuration Server.
    - name: xds_service
      connect_timeout: 0.25s
      type: STRICT_DNS
      typed_extension_protocol_options:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          '@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicit_http_config:
            http2_protocol_options: {}
      load_assignment:
        cluster_name: 'xds_service'
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      # This must be the address of the Deephaven Configuration Server.
                      address: <enter-ip-here>
                      port_value: 8124

  listeners:
    # This section defines a listener that accepts incoming connections.
    - address:
        socket_address:
          address: 0.0.0.0
          port_value: 8000
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                codec_type: AUTO
                use_remote_address: true
                stat_prefix: egress_http
                request_timeout: 0s
                stream_idle_timeout: 600s
                upgrade_configs:
                  - upgrade_type: websocket
                # rds tells Envoy to get its route configuration dynamically from the xDS service.
                rds:
                  route_config_name: rds_config
                  config_source:
                    resource_api_version: V3
                    api_config_source:
                      api_type: GRPC
                      transport_api_version: V3
                      grpc_services:
                        envoy_grpc:
                          cluster_name: 'xds_service'
                access_log:
                  - name: envoy.file_access_log
                    typed_config:
                      '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                      path: '/tmp/envoy-rds.log'
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          # This configures TLS for the listener.
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              require_client_certificate: false
              common_tls_context:
                alpn_protocols: 'h2,http/1.1'
                tls_certificates:
                  - certificate_chain:
                      filename: '/lighttpd.pem'
                    private_key:
                      filename: '/lighttpd.pem'

# This section configures the Envoy admin interface.
admin:
  access_log_path: '/tmp/envoy-admin.log'
  # Setting these will allow a sysadmin to dump Envoy configurations from localhost of the Envoy host.
  # We suggest you disable this in production, as the admin interface can be used to modify the running configuration.
  # Select one of the two cases below: without docker or with docker.
  #
  # Without docker: listen only on the localhost interface.
  # address:
  #  socket_address: { address: 127.0.0.1, port_value: 8001 }
  #
  # With docker, and being able to connect to the Envoy admin port from the host machine,
  # use the configuration below instead (potentially replacing 8001 with the port you prefer to use).
  # address:
  #  socket_address: { address: 0.0.0.0, port_value: 8001 }

Static resources

Clusters: In the static_resources section, configure the target address and port on which the Configuration Service listens for xDS requests. If the Configuration Service is on a different node than Envoy, or if Envoy is in a Docker container, change this to the appropriate address. The port must match the Deephaven system property envoy.xds.port, which defaults to 8124.
Listeners: The listeners section tells Envoy what addresses and ports to listen on for forwarding. In the example, 0.0.0.0 and port 8000 tell Envoy to listen for connections from any interface on port 8000.

TLS Certificates

When using TLS, Envoy needs access to the TLS Certificate chain and private key file. This is configured in the common_tls_context stanza. The example uses Deephaven’s default SSL files, but administrators should install their own for production.

Administration Interface

Administrators can set up the Envoy administration port to inspect the running configuration. The example enables this on port 8001. Be careful with this in production, as it allows access to internal Envoy configuration.

Step 2: Configure Deephaven Services

This step requires additions to Deephaven properties in iris-environment.prop, iris-endpoints.prop, and getdown.global.

Note

Once Envoy is enabled for Deephaven, all services must be configured to use it, or the Configuration Server service will fail to start.

Update `iris-environment.prop`

Export, edit, and reimport this file:

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties export --etcd -f iris-environment.prop -d /tmp/
sudo -u irisadmin vi /tmp/iris-environment.prop
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import --etcd -f iris-environment.prop -d /tmp/

Add the following properties:

# Enables the WebSocket server, which is required for Envoy to proxy WebSocket traffic.
global.websocket.server.enabled=true

# The public-facing URL of the Envoy proxy. This is used by services to construct correct URLs.
envoy.front.proxy.url=user-facing-host.company.com:8000

# Tells Deephaven that Envoy is handling TLS termination.
envoy.terminate.ssl=true

# The Configuration Server needs to know the host of the Web API server.
Webapi.server.host=web-api-host.company.com

# This section configures Swing clients (e.g., Iris Console) to connect through Envoy.
[service.name=iris_console|interactive_console] {
    # Enables the WebSocket client for Swing applications.
    global.websocket.client.enabled=true
    # Points the client to the Envoy proxy for authentication.
    authentication.server.list=user-facing-host.company.com
    authentication.server.port.ssl=8000
    # Truststore can be left blank if Envoy's certificate is trusted by the system.
    tls.truststore=
    authentication.server.overrideAuthority=
}

# Each service that uses WebSockets needs a unique port assigned for internal communication.
# This port is used by the RemoteQueryDispatcher.
RemoteQueryDispatcherParameters.websocket.port=22052

[service.name=dbmerge|db_dis_merge|tailer1_merge] {
    RemoteQueryDispatcherParameters.websocket.port=22060
}

Update `iris-endpoints.prop`

Services like the Persistent Query Controller need to know about all dispatchers. Add configuration_server to the iris_controller stanza and define the websocket ports for merge servers.

Export, edit, and reimport the file:

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties export --etcd -f iris-endpoints.prop -d /tmp/
sudo -u irisadmin vi /tmp/iris-endpoints.prop
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import --etcd -f iris-endpoints.prop -d /tmp/

Example for a three-node cluster:

# The iris_controller and configuration_server must be aware of all query and merge nodes.
[service.name=iris_controller|configuration_server] {
    iris.db.nservers=3

    iris.db.1.host=query-1.company.com
    iris.db.1.class=Query

    iris.db.2.host=query-2.company.com
    iris.db.2.class=Query

    iris.db.3.host=infra-1.company.com
    iris.db.3.port=30002
    iris.db.3.class=Merge
    # Define the websocket port for the merge server.
    iris.db.3.websocket.port=22060
}

Note

The iris-endpoints.prop file is regenerated by the Deephaven installer. After any reconfiguration or upgrade, you must reapply these modifications.

Update `getdown.global`

Update /etc/sysconfig/illumon.d/getdown.global to tell the Web UI where to find the server:

# This property tells the Web UI client the public URL to connect back to the server via Envoy.
opt=-Ddeephaven.web.client.publicUrl=https://user-facing-host.company.com:8000

Update /etc/sysconfig/illumon.d/client_update_service/getdown.global so the Swing client reads properties from disk:

# This forces the Swing client to use local property files instead of fetching them from the server.
# This is necessary when connecting through a proxy like Envoy.
jvmarg = -Dcom.fishlib.configuration.PropertyInputStreamLoader.override=com.fishlib.configuration.PropertyInputStreamLoaderTraditional

Step 3: Advanced Configuration (Optional)

Custom HTTP Headers

You can add custom HTTP headers to responses from the Deephaven server. Deephaven has pre-defined headers (e.g., Strict-Transport-Security, X-Frame-Options) that can be enabled in iris-environment.prop:

envoy.add.header.Strict-Transport-Security.enabled=true
envoy.add.header.Strict-Transport-Security.value=max-age=31536000; includeSubDomains

You can also add custom headers not pre-defined in iris-defaults.prop. For each header, create a pair of properties to enable it and set its value:

envoy.add.header.Access-Control-Max-Age.enabled = true
envoy.add.header.Access-Control-Max-Age.value = 7200

Extra XDS Routes

Additional routes can be added to the Envoy configuration to expose other services. For each route, create a set of properties in iris-environment.prop:

envoy.xds.extra.routes.dis.host=infra-1.company-name.com
envoy.xds.extra.routes.dis.port=8086
envoy.xds.extra.routes.dis.prefix=/dis/
envoy.xds.extra.routes.dis.prefixRewrite=/
envoy.xds.extra.routes.dis.tls=false
envoy.xds.extra.routes.dis.exactPrefix=false

WebSocket Message Size

If you encounter errors that a WebSocket message size exceeds the maximum, you can increase the limit with the following property in iris-environment.prop:

websocket.client.max.message.size=100000000

Step 4: Restart Services

After making these changes, restart all services on all nodes for them to take effect:

sudo -u irisadmin monit stop all
sudo -u irisadmin monit start all

Configuration properties

The following table summarizes the key properties used when manually configuring Deephaven to work with Envoy.

Property	File(s)	Description
`global.websocket.server.enabled`	`iris-environment.prop`	Enables the WebSocket server required for proxying.
`envoy.front.proxy.url`	`iris-environment.prop`	The public-facing URL of the Envoy proxy.
`envoy.terminate.ssl`	`iris-environment.prop`	Informs Deephaven that Envoy is handling TLS termination.
`Webapi.server.host`	`iris-environment.prop`	Specifies the host of the Web API server for the Configuration Server.
`global.websocket.client.enabled`	`iris-environment.prop`	Enables the WebSocket client for Swing applications.
`authentication.server.list`	`iris-environment.prop`	Points Swing clients to Envoy for authentication.
`opt=-Ddeephaven.web.client.publicUrl`	`getdown.global`	Sets the public URL for the Web UI client to connect to the server.
`jvmarg`	`getdown.global`	Forces the Swing client to use local property files.
`websocket.client.max.message.size`	`iris-environment.prop`	Increases the maximum WebSocket message size to prevent errors.
`envoy.add.header.*`	`iris-environment.prop`	Enables and sets custom HTTP headers in responses.
`envoy.xds.extra.routes.*`	`iris-environment.prop`	Defines additional routes for exposing other services through Envoy.

Security Considerations

When deploying Envoy in a production environment, it is critical to consider the following security best practices:

TLS Certificates: The example configuration uses Deephaven’s default lighttpd.pem file for convenience. For any production system, you should replace this with your own properly signed TLS certificate and private key to ensure secure communication.
Admin Interface: The Envoy admin interface on port 8001 provides powerful debugging capabilities but also exposes sensitive configuration details. In a production environment, access to this port should be heavily restricted. Use firewall rules (e.g., iptables or your cloud provider's security groups) to ensure that the admin port is only accessible from trusted IP addresses, such as a specific bastion host or an internal management network.
Network Security: Ensure that the network communication between Envoy and the backend Deephaven services is secure. If Envoy and Deephaven are running on different hosts, consider using a private network or firewall rules to limit traffic between them to only the necessary ports.