--- title: Troubleshooting Envoy sidebar_label: Troubleshooting --- This guide provides steps for troubleshooting common issues with Envoy when used as a front proxy for Deephaven. ## General Diagnostic Checklist Before diving into specific error codes, start with the checks that require the fewest assumptions. In particular, do not begin with the admin-interface commands unless you already know that Envoy's admin interface is enabled and reachable. If you are not sure how Envoy is installed, start with [Determine Envoy installation method](#determine-envoy-installation-method), then use the matching process and log commands for that installation type. A good first pass is: confirm that Envoy is running, check for obvious startup or configuration errors in the logs, and only then move on to admin-interface checks such as `/clusters` or `/config_dump`. Those admin endpoints are described in more detail in [Using the Admin Interface](#using-the-admin-interface). If the admin interface is not available yet, you may find your answers in the [Envoy request logs](#envoy-request-logs). 1. **Determine how Envoy is installed.** If you are not sure whether this is a native, systemd-managed, Docker, or Podman install, follow [Determine Envoy installation method](#determine-envoy-installation-method) first. 2. **Confirm that Envoy is running.** Use the command that matches your installation method. ```bash # docker sudo docker ps -f name=deephaven_envoy # podman sudo podman ps -f name=deephaven_envoy # systemd root service systemctl status deephaven_envoy # systemd user unit sudo -u irisadmin XDG_RUNTIME_DIR=/run/user/$(id -u irisadmin) systemctl --user status deephaven_envoy ``` 3. **Check for obvious startup or configuration errors in the logs.** Again, use the command that matches your installation method. ```bash # systemd root-or-user service sudo journalctl -u deephaven_envoy --no-pager -S "-30 min" # docker sudo docker logs --since 30m deephaven_envoy # podman sudo podman logs --since 30m deephaven_envoy ``` 4. **Check whether backend clusters are healthy.** ```bash curl http://localhost:8001/clusters ``` This endpoint requires the [admin interface to be enabled and reachable](#using-the-admin-interface). Look for healthy upstreams, including `xds_service` members showing `health_flags::healthy`. 5. **Check whether the expected configuration is loaded.** ```bash curl http://localhost:8001/config_dump ``` This endpoint requires the [admin interface to be enabled and reachable](#using-the-admin-interface). Verify that listeners, routes, and clusters match your expectations. For route-specific problems, you can narrow the output: ```bash curl http://localhost:8001/config_dump?resource=rds_config ``` 6. **Inspect the access/request log for failing requests, unexpected routes, or upstream mismatches.** The exact collection method depends on how Envoy is installed and how the access log is configured; see [Envoy request logs](#envoy-request-logs) for the supported locations and commands. When reading request logs, look for: - the request path and method, - the returned status code, - the selected upstream host, - any timeout, reset, or transport-failure indicators. Each request to Envoy results in a log line like the following: ```text [2025-08-14T19:00:39.725Z] "POST /io.deephaven.proto.auth.grpc.AuthApi/getToken HTTP/2" 200 - 109 74 1 0 "10.1.2.3" "grpc-python/1.74.0 grpc-c/49.0.0 (linux; chttp2)" "7217a86b-9964-4484-a53c-3bba48a7c81d" "host.example.com:8000" "10.1.2.4:9031" ``` Among other fields, Envoy log lines contain the request path (for example, `/io.deephaven.proto.auth.grpc.AuthApi/getToken`), HTTP status code (for example, `200`), and backend address (for example, `10.1.2.4:9031`). ## Determine Envoy installation method To manage your Envoy installation, you need to know how it is installed. If you do not know this information, you can inspect the processes running on your Envoy host machine: ```bash ps -ef --forest | grep -v grep | grep "envoy" -B 2 ``` The above command will show the running process and its parent processes. A containerized installation will look like this: ```bash root 52336 1 0 17:45 ? 00:00:00 /usr/bin/conmon --api-version 1 -c 27809a4897def0645f28b635b258b614765d86f1ac703174b7699dc7ead52f36 -u ... irisadm+ 52342 52336 0 17:45 ? 00:00:15 \_ envoy -c /config.yaml ``` A native installation will look something like this: ```bash irisadm+ 49671 1 0 17:44 ? trou-en 00:00:00 /usr/lib/systemd/systemd --user irisadm+ 49673 49671 0 17:44 ? 00:00:00 \_ (sd-pam) irisadm+ 56394 49671 0 17:47 ? 00:00:11 \_ /db/TempFiles/irisadmin/bin/envoy-1.36.4-linux-x86_64 -c /etc/sysconfig/illumon.d/resources/envoy3.yaml ``` For clusters using the native Deephaven installer, you can find the expected installation mode from your `cluster.cnf`: ```bash grep DH_ENVOY_INSTALL_MODE /etc/sysconfig/deephaven/cluster.cnf ``` ## Reloading Envoy When you change the [`envoy.yaml`](configuring-envoy.md#envoy-yaml-file) configuration, it is possible to tell the Envoy process to hot-reload the new configuration without restarting the process or cancelling any connections. [Envoy's hot reload process](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/hot_restart) is fairly complex, as it requires you to keep track of how many times you have previously performed a hot reload. The simplest option is to restart Envoy, but all in-progress connections will be dropped. ## Restarting Envoy If you have rotated your certificates or made any non-trivial changes to your system, you should restart the Envoy process. If you have only [edited `envoy.yaml` configuration](configuring-envoy.md#envoy-yaml-file), you may be able to simply [Reload Envoy configuration](#reloading-envoy) rather than restarting the process. The method to restart Envoy depends on your [Envoy installation method](#determine-envoy-installation-method). If you used any of the suggested installation methods, one of the following commands will work for you: ```bash # preferred: use systemd systemctl restart deephaven_envoy # if using a user unit, you may need additional variables: sudo -u irisadmin XDG_RUNTIME_DIR=/run/user/$(id -u irisadmin) \ systemctl --user restart deephaven_envoy # if using podman without systemd: podman stop deephaven_envoy podman start deephaven_envoy # if using docker without systemd: docker stop deephaven_envoy docker start deephaven_envoy ``` When using the Deephaven native installer with [Deephaven-managed Envoy](installing-envoy.md#deephaven-managed-envoy), the Envoy process automatically restarts during cluster upgrades. When managing Envoy yourself, you must restart Envoy after the installation completes. ## Using the admin interface The Envoy admin interface is a useful tool for debugging client interaction with the Deephaven server. When enabled, it listens on port `8001`. It also serves a web application for browsing Envoy state, but because it can expose internal state and sensitive information in logs, access should be restricted. > [!IMPORTANT] > Secure the Envoy admin port to localhost connections only. Sample [envoy.yaml](configuring-envoy.md#envoy-yaml-file): > > ```yaml > admin: > address: > socket_address: > address: 127.0.0.1 # local install > # address: 0.0.0.0 # containerized install > port_value: 8001 > ``` > > For containerized Envoy installs, access can be retricted to `127.0.0.1` when you publish the port: > > - `docker run -p 127.0.0.1:8001:8001` or `podman run -p 127.0.0.1:8001:8001`. > - For Podman Quadlets: `PublishPort=127.0.0.1:8001:8001`. > - You may need to destroy and recreate the container for port publishing changes to take effect. > - Check published ports via `docker port deephaven_envoy` or `podman port deephaven_envoy`. > > Your admin port can also be restricted with firewall or security-group rules. - **`/config_dump`**: Shows the entire loaded configuration. This is useful for verifying that your `envoy3.yaml` and dynamic xDS updates have been applied correctly. You can filter it for specific resources, like routes (`?resource=rds_config`). - **`/clusters`**: Provides a detailed status of all upstream clusters, including IP addresses, health status, and connection statistics. This is the best way to check if Envoy can connect to the backend Deephaven services. - **`/stats`**: Outputs a large number of performance metrics. You can use `grep` to find specific stats, like `upstream_cx_total` for connection counts or `http.downstream_rq_5xx` for server errors. - **`/server_info`**: Displays the running Envoy version and its uptime, which is useful for confirming that a restart was successful. ## Envoy application logs Envoy emits application logs for startup/configuration events, routing decisions, and detailed upstream/downstream error messages. You can adjust log levels and component-specific logging through the admin interface, which [must be enabled](#using-the-admin-interface) for these commands to work. Envoy's documentation on the [admin interface logging endpoints](https://www.envoyproxy.io/docs/envoy/latest/operations/admin#post--logging) includes additional options and examples. How you view application logs depends on how Envoy is installed: - **systemd / native install**: ```bash sudo journalctl -u deephaven_envoy --no-pager sudo journalctl -u deephaven_envoy -S "-30 min" --no-pager ``` - **Containerized install**: ```bash sudo docker logs --since 30m deephaven_envoy sudo podman logs --since 30m deephaven_envoy ``` You can temporarily increase logging via the admin interface: ```bash curl -sSd '' "http://localhost:8001/logging?level=debug" ``` To return to the default level: ```bash curl -sSd '' "http://localhost:8001/logging?level=info" ``` When you need details for a specific issue (for example, routing or upstream connection problems), enable debug logging for specific subsystems using `paths=...`: ```bash curl -sSd '' "http://localhost:8001/logging?paths=http2:debug,router:debug,upstream:debug,connection:debug" ``` Common log levels to increase are `http`, `http2`, `router`, `upstream`, `connection`, `grpc` and `filter`. > [!CAUTION] > Debug and trace logging can be very noisy and may include request metadata. > Limit the time window, reproduce the issue, then revert the log level. ## Envoy request logs Envoy request logs are recorded via the access log configured on the route/listener (for Deephaven-managed Envoy, this is typically set in the RDS route configuration). The [admin interface must be enabled](#using-the-admin-interface) to use the troubleshooting commands in this guide that inspect live config and cluster health. Envoy's documentation on [access logs](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage) covers additional output formats and fields. Access logs are ideal for quickly answering: - Which requests are failing (paths/methods), and with what status codes? - Which route was selected? - Which upstream cluster/host handled the request? - Whether Envoy saw a transport failure, timeout, or reset. ### What to collect for a troubleshooting bundle When reporting an Envoy issue, collect the following (keep the window small: typically 5–30 minutes around the failure): 1. **Access log** (request log) - Why: best high-level signal for request failures and routing/upstream selection. - Example (common location in these docs): ```bash sudo podman exec /deephaven_envoy sh -c 'tail -n 2000 /tmp/envoy-rds.log' ``` 2. **Application logs** (journal/container logs) - Why: includes warnings/errors that do not appear in access logs (config reload errors, upstream TLS failures, disconnect reasons). 3. **Admin interface snapshots** - Why: captures the loaded config and current upstream health at the time of failure. - Examples: ```bash curl -sS http://localhost:8001/server_info curl -sS http://localhost:8001/clusters curl -sS http://localhost:8001/config_dump curl -sS http://localhost:8001/stats ``` > [!IMPORTANT] > Logs and admin output may contain sensitive information (hostnames, IPs, request headers, and possibly user identifiers). > Collect only the minimum time window needed and review/redact before sharing. ### Improve access logs with JSON lines For troubleshooting intermittent failures, JSON access logs are easier to search and parse. The following is an example access log format block that emits one JSON object per line (unescaped JSON lines). The snippet is shown in minimal context; placement may vary depending on whether your configuration uses per-route or per-virtual-host logging. ```yaml static_resources: listeners: - filter_chains: - filters: - name: envoy.filters.network.http_connection_manager typed_config: access_log: - name: envoy.file_access_log typed_config: "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog path: "/tmp/envoy-rds.log" log_format: json_format: start_time: "%START_TIME%" method: "%REQ(:METHOD)%" path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%" authority: "%REQ(:AUTHORITY)%" protocol: "%PROTOCOL%" response_code: "%RESPONSE_CODE%" response_code_details: "%RESPONSE_CODE_DETAILS%" response_flags: "%RESPONSE_FLAGS%" duration_ms: "%DURATION%" request_id: "%REQ(X-REQUEST-ID)%" route_name: "%ROUTE_NAME%" upstream_cluster: "%UPSTREAM_CLUSTER%" upstream_host: "%UPSTREAM_HOST%" upstream_transport_failure_reason: "%UPSTREAM_TRANSPORT_FAILURE_REASON%" grpc_status: "%GRPC_STATUS%" ``` Key fields to pay attention to: - `request_id`: correlates retries and makes it easier to match a single request across logs. - `route_name`: confirms which route matched. - `upstream_cluster`: identifies which backend service Envoy routed to. - `upstream_transport_failure_reason`: often includes a concrete reason when the upstream connection fails (for example, TLS, reset, or connect failures). - `response_flags`: compact flags that indicate common failure categories (timeouts, resets, upstream failures). - `response_code_details`: additional detail behind the response code. - `grpc_status`: for gRPC calls, helps distinguish transport success from application-level status. > [!IMPORTANT] > Ensure the access log is emitted as JSON lines (one JSON object per line). Avoid double-escaped JSON strings, which are much harder to search and parse. ## Common Issues and Resolutions ### Connection Refused - **Symptom**: Your browser or client shows a "Connection Refused" error when trying to connect to the Envoy port (e.g., `8000`). - **Cause**: This typically means the Envoy process is not running or not listening on the correct port. - **Troubleshooting Steps**: 1. Verify that the Envoy process or container is running using the checklist above. 2. Check the Envoy logs for startup errors, such as a port conflict or a syntax error in the configuration file. 3. Ensure no firewall rules on the host or network are blocking access to the port. ### 503 Service Unavailable - **Symptom**: You receive a `503 Service Unavailable` error. This is often accompanied by `no healthy upstream` messages in the logs. - **Cause**: Envoy is running but cannot establish a healthy connection to the backend Deephaven services. - **Troubleshooting Steps**: 1. Use the `/clusters` admin endpoint to identify which cluster is unhealthy. 2. Verify that the backend Deephaven services (e.g., `web-api`, `xds_service`) are running and accessible from the Envoy host. 3. Check for network connectivity issues (e.g., firewall rules, incorrect IP addresses in `envoy3.yaml`). ### 404 Not Found - **Symptom**: You receive a `404 Not Found` error for a specific URL. - **Cause**: Envoy is running and connected, but the requested URL path does not match any configured route. - **Troubleshooting Steps**: 1. Verify the URL you are trying to access is correct. 2. Dump the route configuration to ensure the routes are correctly defined and loaded from the Deephaven RDS. ```bash curl http://localhost:8001/config_dump?resource=rds_config ``` 3. Check the Deephaven Configuration Server logs to ensure it is correctly publishing routes to Envoy. ### WebSocket Connection Failures - **Symptom**: The Deephaven Web UI loads, but you cannot open a query console, or data does not update in real-time. Browser developer tools show a failed WebSocket handshake. - **Cause**: The WebSocket upgrade request is being blocked or misconfigured. - **Troubleshooting Steps**: 1. Verify that the `upgrade_configs` section is present in the `http_connection_manager` filter in your `envoy3.yaml` file. 2. Check for any intermediate network devices (like corporate firewalls or other proxies) between the client and Envoy that might be blocking WebSocket traffic. 3. Inspect the Envoy logs for errors related to `upgrade failure`. ### TLS/SSL Certificate Issues - **Symptom**: The browser shows a security warning (e.g., `NET::ERR_CERT_AUTHORITY_INVALID`), or connections fail with a TLS handshake error. - **Cause**: The TLS certificate is not correctly configured, trusted, or presented by Envoy. - **Troubleshooting Steps**: 1. Verify the certificate input that Envoy is expected to read. - For installer-managed Envoy, confirm `DH_ENVOY_LOCAL_CERT` points to the correct PEM bundle and that the runtime path in `DH_ENVOY_PEM_PATH` matches the generated YAML. - For manual container-based Envoy, confirm the volume mount matches the runtime path in the YAML, commonly `/envoy.pem`. 2. Ensure the PEM bundle contains the certificate chain followed by the private key and is readable by the Envoy process. 3. Use a command-line tool like `openssl` to inspect the certificate that Envoy is presenting: ```bash openssl s_client -connect your-envoy-host:8000 ``` ## Related documentation - [Use Envoy as a front proxy](./envoy.md) - [Installing Envoy](./installing-envoy.md) - [Configuring Envoy](./configuring-envoy.md) - [Envoy configuration flags for native installations](../installation/cluster-config-guide.md#envoy-properties) - [TLS Certificates](../security/tls-certificates.md)