Services and dependencies

A resilient Deephaven cluster is built upon a collection of interdependent services, each vital to the real-time data platform's operation. Understanding the role of each service within the cluster, its dependencies, and its specific contributions to high availability is crucial for comprehensive resilience planning. This document outlines these key services, their functions, and their HA mechanisms.

Core infrastructure services

These services form the backbone of the Deephaven cluster, managing state, configuration, and worker processes.

etcd

  • Role: etcd is a distributed key-value store that serves as the central source of truth for the entire cluster. It stores configuration, persistent query definitions, and the current state of workers.
  • High Availability: etcd is inherently fault-tolerant. A cluster with an odd number of nodes (three or more) can automatically withstand the loss of nodes while maintaining availability. In a typical production Deephaven installation with three nodes, etcd runs on each node, allowing the cluster to survive the loss of any single node without impacting etcd's operation. Three nodes is the minimum recommended configuration for production high availability.
  • Configuration: See etcd configuration for detailed setup and management.

Controller

  • Role: The Controller is responsible for managing the lifecycle of persistent queries (PQs) and the workers that execute them.

  • High Availability: Only one Controller process acts as the leader at any given time. Additional Controller instances can be configured as standby servers. If the active leader fails to renew its etcd lease (due to a crash, network issue, or having been restarted), the remaining servers participate in a leader election, and a new leader is chosen to take over.

    • In the case of a failover, running Core+ workers will continue running and be associated with the new leader. Running Legacy workers will shut down immediately.
    • If there is only a single instance of the Controller, running Core+ workers will survive a restart of the controller, while Legacy workers will immediately shut down.
    • Core+ workers will wait and retry for up to one minute before shutting down if they lose their controller connection.

  • Configuration: Some aspects of controller configuration can be reloaded without restarting the controller itself. See Persistent Query Controller for detailed configuration and management.

Authentication and configuration services

  • Role: The authentication_server handles user authentication, while the configuration_server provides configuration settings to other services.
  • High Availability: In bare metal deployments, you can set up multiple authentication server and configuration server instances in order to provide fault tolerance, with clients configured to connect to any available instance. In Kubernetes, these services are deployed as statefulsets to provide high availability.
  • Configuration: See Authentication service and Configuration server for detailed setup.

Data processing and query services

These services are the workhorses of the cluster, responsible for launching workers and managing data.

Query servers

  • Role: Query servers launch query workers (read-only access to system tables), which execute user queries, including scripts, ticking tables, and other computations.
  • High Availability: Query servers are stateless compute resources; they do not own any data. If a query server fails, the queries that it started will also fail; they can then be restarted on other available query servers, and the Controller has the capability to do this automatically. Each query host in a bare metal or Podman cluster has its own db_query_server process. A common HA strategy is the n-1 approach: provisioning enough query servers to handle the full production workload even if one server is lost. In Kubernetes environments, there is one query_server for the deployment. If a query server process is restarted, all workers it had launched will shut down. If these are backing PQs, the PQs may be configured to automatically restart on any available query server.
  • Best practices: Use automated server selection to ensure Persistent Queries are not configured to run in only one place. Configure PQ scheduling for automatic restart of failed queries. Deploy Persistent Query replicas and spares for redundancy.

Merge servers

  • Role: Merge servers launch query workers (read/write access to system tables) which handle data ingestion and running merge routines that combine intraday data into historical tables.
  • High Availability: Merge servers are stateless compute resources; they do not own any data. If a merge server fails, the queries that it started will also fail; they can then be restarted on other available merge servers, and the Controller has the capability to do this automatically. However, note that merge queries often require access to local data files that must also be configured for a backup merge server to be a valid host for merge queries being restarted after a merge server has failed. Each query host in a bare metal or Podman cluster has its own db_merge_server process. A common HA strategy is the n-1 approach: provisioning enough merge servers to handle the full production workload even if one server is lost. In Kubernetes environments, there is one merge_server for the deployment. If a merge server process is restarted, all workers it had launched will shut down. If these are backing PQs, the PQs may be configured to automatically restart on any available merge server.
  • Best practices: For Persistent Query failover, similar mechanisms can be used for merge servers as for query servers if multiple merge servers are available.

Data ingestion and access services

These services are responsible for bringing data into Deephaven from external sources.

Data Import Server (DIS)

  • Role: The DIS handles ticking data streams, writing events to disk and publishing them to clients. This is the system DIS, as opposed to optional in-worker DISes that are often implemented for ingesters and data transformation queries. You can use this system DIS for customer-created binary log data streams, and also handles live user tables, input tables, and Deephaven internal tables, such as the Process Event Log.
  • High Availability: In bare metal installations, redundant DIS instances can be configured into a failover group in the data routing setup, allowing for round-robin failover, and multiple DISes can also be configured to separate the types of data and features they handle.

Tailer

  • Role: A Tailer reads from binary log files (often produced by custom logger processes) and streams the data to a DIS.
  • High Availability: An individual tailer is not fault-tolerant, but it checkpoints its read position. If a tailer fails, a new one can be started, and it will resume from the last checkpoint. Redundancy can also be achieved by having multiple tailers process the same data stream to different destinations.
  • Configuration: See Tailer configuration for detailed setup and management.

Log Aggregator Service (LAS)

  • Role: The LAS provides an endpoint for Deephaven processes to append log data. This is used for workers to log to internal tables, like the Process Event Log, and also to allow workers to manage and append to live user tables.

  • High Availability: Each Deephaven query or merge server has a local LAS. Kubernetes clusters have one LAS instance for the Deephaven cluster. There is no high availability for the LAS for individual servers, but the one-per-server deployment of them in bare metal and Podman deployments limits the impact of a failed LAS. monit or Kubernetes will restart a LAS if it crashes.

    • The LAS is required for query workers to be able to start, so a failed LAS will also deny service for its associated query or merge servers.
    • Running Core+ workers can survive through the restart of their LAS and will reconnect to it when it resumes running.
    • Running Legacy workers will immediately shut down if their LAS is restarted.

Local Table Data Service (LTDS)

  • Role: In some Deephaven installations, the LTDS provides access to table data on a specific server. See the Local Table Data Service guide for an architectural overview and configuration considerations. By default, this service is not configured or used.
  • High Availability: Redundant LTDSs can be configured for round-robin failover in the data routing setup.

Table Data Cache Proxy (TDCP)

  • Role: The TDCP caches table data locally on each host. This allows worker processes to share cached data and reduces load on upstream data sources.
  • High Availability: Each host typically runs its own TDCP instance. If a TDCP fails, workers on that host lose access to intraday data, may fail to start, or encounter query failures. The impact is isolated to that host — other hosts with functioning TDCPs are unaffected.

Supporting and optional services

  • web_api_service: This service is not fault-tolerant. Its loss impacts Web and API access.
  • db_acl_writer: In bare metal and Podman this service is not fault-tolerant. Its loss prevents changes to database permissions and accounts. See ACL Troubleshooting. In Kubernetes, it is deployed as a statefulset, which provides high availability.
  • Envoy: A third-party reverse proxy that is optional in bare metal and Podman deployments, but required for Kubernetes deployments. Envoy itself supports clustered and statefulset deployments, but is deployed as a single instance as part of Deephaven deployments.

Failure analysis and troubleshooting

For comprehensive analysis of failure scenarios and step-by-step troubleshooting procedures, see: