Resilience planning

High availability, fault tolerance, and disaster recovery

Deephaven is a distributed platform designed for high availability and fault tolerance. Effective resilience planning requires understanding how Deephaven's architecture handles failures, protects data, and ensures continuous operation. This guide provides a comprehensive framework for designing and managing a resilient Deephaven deployment.

The guide is organized into the following articles, each focusing on a critical aspect of resilience:

  • Failure Modes: An analysis of potential hardware, software, and environmental failures, and strategies for disaster recovery.
  • Services and Dependencies: A detailed look at the roles of core Deephaven services and their high-availability mechanisms.
  • Data and Configuration Storage: An overview of how different types of data and configuration are stored and protected against loss.
  • Data Ingestion: A guide to building resilient data ingestion pipelines that can withstand component failures.

These topics have some amount of overlap. In the context of this document:

  • High availability means continued operations with short downtime and some manual actions needed by administrators and/or users.
  • Fault tolerance means continued operations with no user awareness of an issue other than possibly degraded performance.
  • Disaster recovery means resuming operations after a significant loss of data center functionality or connectivity.

Besides the consideration of system availability, there is also planning for recovery time, tolerable data loss, and what amount of data reloading, if any, is acceptable in case of a failure. Typically, these are referred to as:

  • Recovery Time Objective (RTO) - maximum down time after a system failure.
  • Recovery Point Objective (RPO) - maximum acceptable data loss from a system failure.

Differences among Deephaven deployments for high availability features

You can deploy Deephaven to bare metal (running on one or more machines or VMs), to Kubernetes, or with Podman. All deployment models (as long as there are sufficient compute resources deployed) support Persistent Query replicas and spares; but, for service-level redundancy, the available options differ by deployment type.

Podman

Podman has no service-level HA capabilities. While you can scale the cluster capacity by adding additional "query" nodes, the Podman deployment supports only a single node in the Deephaven cluster running infrastructure services. Included among the services of the "infra" node is a single instance of etcd, so, even at a configuration data preservation level, Podman deployments are the least resilient of the deployment types.

Within a Podman container, the deployment is very similar to a bare metal installation, including the presence of monit, which manages Deephaven system processes and restarts them if they crash.

Kubernetes

Kubernetes inherently provides isolation of Deephaven processes: each worker and each infrastructure service has its own pod. Kubernetes monitors these pods to ensure they are running correctly. You can also add additional health and liveness checks. If a Deephaven infrastructure service crashes, Kubernetes will attempt to restart its container or replace its pod.

In addition, Deephaven Kubernetes clusters configure statefulsets to run multiple concurrent instances of key services. The defaults are:

  • etcd - 3 replicas
  • controller - 2 replicas
  • authserver - 2 replicas
  • aclwriter - 2 replicas
  • configuration-server - 2 replicas

Another advantage of Kubernetes is that it is entirely isolated from actual compute hardware. If a server in the Kubernetes cluster fails, the server can be replaced with minimal disruption to Deephaven activities and with no need to reconfigure or reinstall anything in the Deephaven deployment itself.

Bare metal

Bare metal deployments are the most flexible and customizable. While they do not have the level of process isolation available in Kubernetes, many processes can be separated onto different servers, and monit provides service monitoring and restarts if a process crashes.

Redundancy in a bare metal installation is configured by the person doing the installation, by assigning the same services to multiple nodes in the cluster configuration file. A minimal bare metal deployment would be everything on one server - one instance of each service, including etcd - and no HA capability. For production installations, the recommended minimum is three servers, so there can be three nodes in the etcd cluster (providing high availability and fault tolerance).

On multi-node bare metal installs, infrastructure services such as the controller and authentication_server can have multiple instances deployed. Multi-node installs also allow deployment of additional dis processes, in groups for HA, or for specialized applications, such as dedicated centralized user table handling. Lastly, bare metal installs automatically provide a tdcp, a las, and a tailer per node, which reduces potential disruption should any of these fail.