Persistent Query redundancy and failover

Deephaven's Persistent Queries let you create applications for processing and sharing data with users. These Persistent Queries can be configured with Replicas and Spares to balance the user load across identical copies and have spares available to take over in the event of a failure.

Replicas

A Replica is an identical copy of a Persistent Query that Deephaven uses to balance and compartmentalize user load. When a query is configured with more than one Replica, Deephaven assigns users to individual replica slots based on an Assignment Policy. Deephaven's default assignment policy uses Round Robin to distribute users evenly across the available Replicas.

Once a user has been assigned to a slot, they only interact with tables and plots from the worker in that slot. This compartmentalizes users so that if a particular user causes a query worker to crash, they do not cause cascading failures of other slots. If capacity is added or removed, the Assignment Policy may choose to reassign users. In the query monitor, viewers only see their assigned slot. The owner and administrator have a tree view showing the state of all Replicas and Spares for a given query.

Spares

A Spare is an identical copy of a Persistent Query that Deephaven uses to replace a failed worker. As soon as a failure is detected, the controller takes any available spare and immediately assigns it to the failed slot. This minimizes the downtime any user experiences from a query problem.

Behavior

Replicas and Spares are identical copies of the original query that the system runs on whichever hosts the query configuration specifies. If you have $R$ replicas and $S$ spares, then the query needs $(R+S) * Heap Size$ of host memoery to run. If, for example, the query is set to run on a specific host like Query_1 then there must be at least that much memory available there. When Auto Query is selected the system may distribute the replicas and spares across all available hosts.

All workers start at the same time and are assigned to a slot. Replicas may not move between slots, while Spares may take the slot of any failed Replica. If a Replica fails at any point, the controller immediately assigns any running Spare to take its place and notify clients. After the Spare has been inserted, a new worker fills the used Spare slot.

The Error Restart Attempts setting on the scheduling tab controls the maximum number of times a Replica or Spare can fail. If a replica or spare in a slot fails more times, it remains in the failed state until an administrator manually restarts it. This prevents a problematic slot from consuming and failing all the remaining available spares.

Worker Failures

When a worker fails, the Controller attempts to replace it with an available running spare as mentioned earlier. Automatic replacement is limited by Error Restart Attempts.

For example, say there is a Persistent Query with five replicas and five spares configured with a maximum error restart attempt count set to two. A user, 'Greg', connects to the system and does some operation that causes the worker to crash. The controller takes one of the five available running spares, swaps it into the failed slot, and then starts a new worker to replace the consumed spare. Greg's client detects the new worker and re-issues the command that crashed the worker, causing it to crash again. The worker has now failed twice, so the controller tries again with the same results. When Greg crashes the worker a third time, the controller does not consume another spare, leaving the slot in a failed state. The other four replicas and three spares continue to run unaffected, so if there is a failure on a remaining replica, there will still be spares available to replace it.

If instead the number of restart attempts was set to unlimited, Greg's broken operation would continue to crash every replacement when it reconnects, and there would be no available spares for any other slot to use should they fail.

If a slot exceeds the number of allowed restart attempts, you may restart it manually using the Restart only selected replicas option in the UI. A manual restart also resets the failure count, enabling the controller to replace it with further spares when needed.

Administrators and query owners can restart single Replicas or Spares using the drop-down menu of the Restart button in the Query Monitor

Configuration

Replicas and Spares are configured in the Load Balancing tab of the Persistent Query configuration panel.

• Replica workers field is used to set the number of active replicas.
• Failover spares field set the number of spare queries to run.
• Replica assignment field lets you choose an assignment policy for the query. Deephaven includes one default Round Robin policy.

The Round Robin assignment policy determines which slot a user is assigned to when the controller first sends them information about a query. This can happen when the user logs in, or if the query is changed such that they are now permitted to view it. The assignment is stored only in the memory of the controller, and is not preserved across controller restarts. The round robin policy does not take into account which users have connected to a particular query, or the load of any replica. Users are assigned starting at slot 0, with each subsequent user assigned to the next slot in numerical order (wrapping back around to zero).

When the replica count is decreased, the users on the removed replicas are assigned to the remaining replicas. When the replica count is increased, then all users are reassigned to a new slot to make use of the additional replicas. Finally, the table on the bottom shows the assignment of users to Replicas. This is informational only; you can not make changes manually.

Adjusting capacity

You may dynamically add or remove Replicas and Spares during the day to increase or decrease the capacity of your query when you change the Replica workers or Failover spares settings and select the Save without restart option. When you add or remove Replicas this way, the Assignment policy may choose to redistribute the assigned users to spread the load across the new capacity.

Status

When Replicas and Spares are configured, the query Status display shows additional information about the status of each running Replica and Spare.

The summary section shows a concise status for each Replica and Spare separated by a dot. The leftmost items are Replicas; the rightmost are Spares.

The Worker Summary section gives you detailed information about the status of each worker. You can click on the worker name to see its process info id, and start and end times.

Assignment Policies

An Assignment Policy is used to assign Deephaven users to a single Replica of a PQ. Deephaven includes a single default implementation of the Round Robin policy that assigns users to Replicas evenly.

You may implement your own Assignment Policy by implementing the io.deephaven.enterprise.controller.assignment.AssignmentPolicy interface (javadoc). Implementations must include a constructor that accepts a io.deephaven.enterprise.controller.assignment.QueryUserAssignmentLogger and a String for policy parameters.

public MyAssignmentPolicy(@NotNull final QueryUserAssignmentLogger logger, @Nullable final String policyParameters){}

The logger is used to record user-to-slot assignment changes for the UI. It has one method:

public void log(final long timestamp,
                final String userToAssign,
                final long querySerial,
                final String queryOwner,
                final String[] adminGroups,
                final int assignedSlot,
                final String policyName,
                final String details) {}

Most of the parameters can be taken from the PersistentQueryConfiguration object used during assignment:

• assignedSlot should be passed to the slot the user was just assigned to.
• policyName should be set to the current policy name.
• details (optional) can be used to describe the reason for assignment.

The second parameter, policyParameters, is a free-form string passed directly in from the Query configuration. It may take any form that suits your needs and can be used to configure policy options that you define.

Deephaven creates a single Assignment Policy instance for each Persistent Query configured with Replicas. When Replicas are added or removed, the Assignment Policy is allowed to react both before and after the change is applied with the beforeConfigurationChanged and afterConfigurationChanged callbacks.

To add your Assignment Policy implementation to the list of available policies, you need to add the following configuration properties to your Deephaven configuration:

PersistentQueryController.AssignmentPolicy.PolicyName.class=io.mycompany.assignment.MyAssignmentPolicy
PersistentQueryController.AssignmentPolicy.PolicyName.displayName=My Assignment Policy
PersistentQueryController.AssignmentPolicy.PolicyName.description=Assigns users in a special way

• class must be your implementation's fully qualified Java class name.
• displayName is the value users see in the Persistent Query Configuration.
• description is the description users are presented when they hover over the name.

You may change the default Assignment Policy using the following properties:

AssignmentPolicy.defaultPolicy=RoundRobin
AssignmentPolicy.defaultPolicyParams=

Related documentation

• Creating Persistent Queries