Services and dependencies

A resilient Deephaven cluster is built upon a collection of interdependent services, each vital to the real-time data platform's operation. Understanding the role of each service within the cluster, its dependencies, and its specific contributions to high availability is crucial for comprehensive resilience planning. This document outlines these key services, their functions, and their HA mechanisms.

Core infrastructure services

These services form the backbone of the Deephaven cluster, managing state, configuration, and worker processes.

etcd

  • Role: etcd is a distributed key-value store that serves as the central source of truth for the entire cluster. It stores configuration, persistent query definitions, and the current state of workers.
  • High Availability: etcd is inherently fault-tolerant. A cluster with an odd number of nodes (three or more) can automatically withstand the loss of nodes while maintaining availability. In a typical production Deephaven installation with three nodes, etcd runs on each node, allowing the cluster to survive the loss of any single node without impacting etcd's operation. Three nodes is the minimum recommended configuration for production high availability.
  • Configuration: See etcd configuration for detailed setup and management.

Controller

  • Role: The Controller is responsible for managing the lifecycle of persistent queries (PQs) and the workers that execute them.
  • High Availability: Only one Controller process acts as the leader at any given time. Additional Controller instances can be configured as hot-spares. If the active leader fails to renew its etcd lease (due to a crash or network issue), the remaining spares participate in a leader election, and a new leader is chosen to take over.
  • Configuration: See Persistent Query Controller for detailed configuration and management.

Authentication and configuration services

  • Role: The authentication_server handles user authentication, while the configuration_server provides configuration settings to other services.
  • High Availability: Multiple authentication server instances can be run for fault tolerance, with clients configured to connect to any available instance. The configuration server typically runs as a single instance and is not designed for multi-instance deployment.
  • Configuration: See Authentication service and Configuration server for detailed setup.

Data processing and query services

These services are the workhorses of the cluster, responsible for executing user code and managing data.

Query servers

  • Role: Query servers execute user queries, including scripts, ticking tables, and other computations.
  • High Availability: Query servers are stateless compute resources; they do not own any data. If a query server fails, the queries that were running on it can be restarted on other available query servers. A common HA strategy is the n-1 approach: provisioning enough query servers to handle the full production workload even if one server is lost.
  • Best practices: Use automated server selection to ensure Persistent Queries are not configured to run in only one place. Configure PQ scheduling for automatic restart of failed queries. Deploy Persistent Query replicas and spares for redundancy.

Merge servers

  • Role: Merge servers are responsible for data ingestion and running merge routines that combine intraday data into historical tables.
  • High Availability: While individual merge servers are not fault-tolerant, redundancy is achieved by running multiple instances that can be assigned different parts of a workflow.
  • Best practices: For Persistent Query failover, similar mechanisms can be used for merge servers as for query servers if multiple merge servers are available.

Data ingestion and access services

These services are responsible for bringing data into Deephaven from external sources.

Data Import Server (DIS)

  • Role: The DIS handles ticking data streams, writing events to disk and publishing them to clients.
  • High Availability: Redundant DIS instances can be configured into a failover group in the data routing setup, allowing for round-robin failover.

Tailer

  • Role: A Tailer reads from binary log files (often produced by custom logger processes) and streams the data to a DIS.
  • High Availability: An individual tailer is not fault-tolerant, but it checkpoints its read position. If a tailer fails, a new one can be started, and it will resume from the last checkpoint. Redundancy can also be achieved by having multiple tailers process the same data stream to different destinations.
  • Configuration: See Tailer configuration for detailed setup and management.

Local Table Data Service (LTDS)

Table Data Cache Proxy (TDCP)

  • Role: The TDCP caches table data locally on each host. This allows worker processes to share cached data and reduces load on upstream data sources.
  • High Availability: Each host typically runs its own TDCP instance. If a TDCP fails, workers on that host lose access to intraday data, may fail to start, or encounter query failures. The impact is isolated to that host — other hosts with functioning TDCPs are unaffected.

Failure analysis and troubleshooting

For comprehensive analysis of failure scenarios and step-by-step troubleshooting procedures, see:

Supporting and optional services

These services support cluster operations but are not designed for multi-instance high availability:

  • web_api_service: Hosts the Web UI and serves the Client Update Service. Its loss impacts web and OpenAPI access but does not affect running queries or data ingestion.
  • log_aggregator_service: Serializes binary log writes from multiple workers to prevent file contention. Workers are typically configured to require LAS connectivity; if LAS is down, new workers cannot start and running workers may terminate if they cannot buffer log data. See the Log Aggregator Service runbook for details.
  • db_acl_write_server: Administrative interface for managing users, groups, and permissions. Its loss prevents ACL changes but does not affect existing permissions. See ACL Write Server runbook.
  • Envoy: An optional third-party proxy for unified cluster ingress. It can be configured as a cluster for high availability and is often run in containers for easy replacement.

For incident response severity classification, see the Process runbooks.