Protect infrastructure processes from worker CPU load

On native, non-containerized bare-metal and VM installations, Deephaven workers share CPUs with the infrastructure processes that manage them. Starting many workers at once is CPU-intensive: each new JVM loads classes, JIT-compiles code, and initializes its script session. When enough workers start simultaneously, they can starve the Remote Query Dispatcher, the Controller, etcd, and other infrastructure processes of CPU time. Symptoms include worker startup timeouts, etcd lease or leader-election warnings, and Controller or dispatcher heartbeat failures.

This guide shows how to use Linux control groups (cgroups) and process scheduling priority so that infrastructure processes keep running when workers saturate the CPUs. If your cluster has more than one host, the most robust fix is to run workers on separate hosts; the other steps protect hosts that must run both.

Note

Do not apply these host-level instructions to Podman deployments, where services run in containers and the host does not have the native dh-etcd.service layout described here. For Kubernetes, set CPU requests for infrastructure pods in your Helm values, and set worker CPU per Persistent Query with CPU Shares; see Kubernetes CPU configuration.

How Deephaven processes are grouped

On a bare-metal installation, systemd places each service in its own cgroup, and the Linux scheduler divides CPU time between cgroups according to their CPU weight. By default, every service has the same weight.

  • etcd runs as the dh-etcd systemd service, in its own cgroup (see the dh-etcd service unit file).
  • Deephaven infrastructure processes: Monit starts the Controller, the query and merge dispatchers, the Data Import Server, the Authentication Service, the Web API Service, and the other Deephaven services, so they all run in the monit.service cgroup.
  • Workers are child processes of the dispatcher that launched them, so they also run in the monit.service cgroup, alongside the infrastructure processes.

You can inspect the grouping on any host:

Because workers and infrastructure processes share a cgroup, cgroup weights alone cannot favor one over the other. This guide combines two techniques:

  1. Raise the CPU weight of the dh-etcd service so that etcd wins CPU contention against everything under Monit.
  2. Lower the scheduling priority (nice value) of workers so that, within the monit.service cgroup, infrastructure processes win CPU contention against workers.

Both changes only matter when the CPUs are saturated. When there is spare CPU capacity, workers can still use all of it. To reduce the load itself, also reduce the startup burst.

Raise the CPU weight of etcd

CPUWeight is relative; the default for every service is 100, so a weight of 1000 gives etcd ten times the share of a default service when the CPUs are saturated. Use CPUWeight on systems using cgroups v2. On older systemd versions using cgroups v1, use CPUShares instead; its default is 1024, so set it to 10240 for the same tenfold relative weight.

Caution

Restarting etcd on more than one host at a time can cause the etcd cluster to lose quorum. Complete every step below on one etcd host, including the health check, before you move on to the next host.

On each etcd host, one host at a time:

  1. Create a drop-in override for the dh-etcd service:

    Add the setting for your cgroups version in the space between the comment blocks, then save and exit. To check which version the host uses, run stat -fc %T /sys/fs/cgroup: cgroup2fs means cgroups v2, and tmpfs means cgroups v1.

    Use CPUWeight for cgroups v2:

    For cgroups v1, use CPUShares instead:

  2. Restart etcd and confirm the new weight. Only the property for your cgroups version shows a value; the other one shows [not set] or its default.

    On a single-node etcd installation, this restart briefly takes down the configuration store that the Controller, the Configuration Service, and the dispatchers depend on. Expect those processes to log connection errors until etcd is back, and schedule the restart for a maintenance window.

  3. Confirm that every etcd node is healthy before you continue to the next host:

    If irisadmin does not own the etcd client configuration on your system, run the command as the account that does. See etcdctl command overview.

Lower the scheduling priority of Core+ workers

This section applies only to Core+ workers. Legacy workers do not support commandPrefix, so they always run at the same priority as the dispatcher that launches them. On a host that runs legacy workers, raising etcd's CPU weight still protects etcd, but the Controller and dispatchers keep competing with legacy workers on equal terms inside monit.service. For those hosts, rely on separate infrastructure and query hosts and on reducing the startup burst.

Use the commandPrefix property of a Core+ worker kind to launch every worker with nice, which lowers the worker's scheduling priority relative to the dispatcher and other infrastructure processes. The dispatcher splits the prefix into executable and argument tokens on spaces by default, then appends the full worker command line as one final argument:

With this setting, the dispatcher launches workers like this:

If you need a different separator between prefix tokens, set WorkerKind.<name>.commandPrefixDelimiter. For example, use commas instead of spaces:

If your installation already uses commandPrefix (for example, to run workers with a Kerberos keytab through k5start), put nice -n 10 in front of the existing prefix rather than replacing it, for example: nice -n 10 k5start -f my-krb5.keytab -U -- /bin/sh -c.

Nice values range from -20 (highest priority) to 19 (lowest priority), and Deephaven infrastructure processes run at the default of 0. Nice values adjust scheduling weight per runnable thread: a thread at nice 10 gets about 10% of the scheduling weight of a thread at nice 0 when they compete for the same core. This does not reserve CPU for infrastructure or cap worker CPU use; a worker with many runnable threads can collectively receive substantially more CPU than a single infrastructure thread. Unprivileged users can raise a process's nice value but not lower it, so this setting requires no additional permissions.

Export iris-environment.prop from the Configuration Service, add the property for each Core+ worker kind you have defined (replacing DeephavenCommunity with the worker kind name), and import the updated file. For example, on a server with access to the Configuration Service:

See editing configuration files for details. Newly started workers pick up the setting after you restart the dispatchers:

Restarting a dispatcher stops every worker it is running, so schedule the restart for a maintenance window.

To confirm that the setting is in effect, start a worker and list every process with a positive nice value:

The worker's java process should appear with 10 in the NI column. Depending on the shell and on whether the worker runs as a different user, you might also see sh or sudo rows with the same nice value. Read the NI value on the java row, because that is the process doing the work. This check does not depend on the worker kind name or install path.

If the nice value has no effect

A nice value adjusts each thread's scheduling weight; it does not reserve CPU for infrastructure or cap what a worker uses. A worker with many runnable threads, or many workers starting at once, can still take most of the CPU from a dispatcher with only a few busy threads. If workers show 10 in the NI column but still starve infrastructure processes, reduce the load itself: reduce the startup burst, and if that is not enough, separate infrastructure and query hosts.

Reduce the startup burst

Cgroup weights and nice values decide who gets the CPU during contention, but they do not reduce the total work of starting many workers. To shorten and soften the startup burst itself, also consider:

  • Lowering RemoteQueryDispatcher.maxConcurrentStartups so each dispatcher starts fewer workers at once. See Query Dispatcher configuration.
  • Lowering the number of JIT compiler threads each worker uses. See JIT compiler optimization.
  • Staggering Persistent Query start times so that fewer queries start at the same moment.

Separate infrastructure and query hosts

The most robust protection is to keep workers off the hosts that run etcd and the Controller. In a multi-node cluster, run query and merge dispatchers on dedicated query hosts, and run etcd and the other infrastructure processes on hosts that do not launch workers. See Capacity planning for guidance on sizing and placing servers.