---
title: Protect infrastructure processes from worker CPU load
---

On native, non-containerized bare-metal and VM installations, Deephaven workers share CPUs with the infrastructure processes that manage them. Starting many workers at once is CPU-intensive: each new JVM loads classes, JIT-compiles code, and initializes its script session. When enough workers start simultaneously, they can starve the Remote Query Dispatcher, the Controller, etcd, and other infrastructure processes of CPU time. Symptoms include worker startup timeouts, etcd lease or leader-election warnings, and Controller or dispatcher heartbeat failures.

This guide shows how to use Linux control groups (cgroups) and process scheduling priority so that infrastructure processes keep running when workers saturate the CPUs. If your cluster has more than one host, the most robust fix is to [run workers on separate hosts](#separate-infrastructure-and-query-hosts); the other steps protect hosts that must run both.

> [!NOTE]
> Do not apply these host-level instructions to [Podman deployments](../installation/podman-installation.md), where services run in containers and the host does not have the native `dh-etcd.service` layout described here. For Kubernetes, set CPU requests for infrastructure pods in your Helm values, and set worker CPU per Persistent Query with **CPU Shares**; see [Kubernetes CPU configuration](./cpu-optimization.md#kubernetes-cpu-configuration).

## How Deephaven processes are grouped

On a bare-metal installation, `systemd` places each service in its own cgroup, and the Linux scheduler divides CPU time between cgroups according to their CPU weight. By default, every service has the same weight.

- **etcd** runs as the `dh-etcd` systemd service, in its own cgroup (see [the `dh-etcd` service unit file](../installation/basic-install.md#appendix-e-dh-etcd-service-unit-file)).
- **Deephaven infrastructure processes**: Monit starts the Controller, the query and merge dispatchers, the Data Import Server, the Authentication Service, the Web API Service, and the other Deephaven services, so they all run in the `monit.service` cgroup.
- **Workers** are child processes of the dispatcher that launched them, so they also run in the `monit.service` cgroup, alongside the infrastructure processes.

You can inspect the grouping on any host:

```bash
systemd-cgls --unit monit.service
systemd-cgls --unit dh-etcd.service
```

Because workers and infrastructure processes share a cgroup, cgroup weights alone cannot favor one over the other. This guide combines two techniques:

1. [Raise the CPU weight of the `dh-etcd` service](#raise-the-cpu-weight-of-etcd) so that etcd wins CPU contention against everything under Monit.
2. [Lower the scheduling priority (`nice` value) of workers](#lower-the-scheduling-priority-of-core-workers) so that, within the `monit.service` cgroup, infrastructure processes win CPU contention against workers.

Both changes only matter when the CPUs are saturated. When there is spare CPU capacity, workers can still use all of it. To reduce the load itself, also [reduce the startup burst](#reduce-the-startup-burst).

## Raise the CPU weight of etcd

`CPUWeight` is relative; the default for every service is `100`, so a weight of `1000` gives etcd ten times the share of a default service when the CPUs are saturated. Use `CPUWeight` on systems using cgroups v2. On older `systemd` versions using cgroups v1, use `CPUShares` instead; its default is `1024`, so set it to `10240` for the same tenfold relative weight.

> [!CAUTION]
> Restarting etcd on more than one host at a time can cause the etcd cluster to lose quorum. Complete every step below on one etcd host, including the health check, before you move on to the next host.

On each etcd host, one host at a time:

1. Create a drop-in override for the `dh-etcd` service:

   ```bash
   sudo systemctl edit dh-etcd.service
   ```

   Add the setting for your cgroups version in the space between the comment blocks, then save and exit. To check which version the host uses, run `stat -fc %T /sys/fs/cgroup`: `cgroup2fs` means cgroups v2, and `tmpfs` means cgroups v1.

   Use `CPUWeight` for cgroups v2:

   ```text
   [Service]
   CPUWeight=1000
   ```

   For cgroups v1, use `CPUShares` instead:

   ```text
   [Service]
   CPUShares=10240
   ```

2. Restart etcd and confirm the new weight. Only the property for your cgroups version shows a value; the other one shows `[not set]` or its default.

   On a single-node etcd installation, this restart briefly takes down the configuration store that the Controller, the Configuration Service, and the dispatchers depend on. Expect those processes to log connection errors until etcd is back, and schedule the restart for a maintenance window.

   ```bash
   sudo systemctl restart dh-etcd
   systemctl show dh-etcd.service --property=CPUWeight --property=CPUShares
   ```

3. Confirm that every etcd node is healthy before you continue to the next host:

   ```bash
   sudo -u irisadmin DH_ETCD_DIR=/etc/sysconfig/deephaven/etcd/client/root /usr/illumon/latest/bin/etcdctl.sh endpoint health --cluster
   ```

   If `irisadmin` does not own the etcd client configuration on your system, run the command as the account that does. See [`etcdctl` command overview](../troubleshooting/troubleshooting-etcd.md#etcdctl-command-overview).

## Lower the scheduling priority of Core+ workers

This section applies only to Core+ workers. Legacy workers do not support `commandPrefix`, so they always run at the same priority as the dispatcher that launches them. On a host that runs legacy workers, raising etcd's CPU weight still protects etcd, but the Controller and dispatchers keep competing with legacy workers on equal terms inside `monit.service`. For those hosts, rely on [separate infrastructure and query hosts](#separate-infrastructure-and-query-hosts) and on [reducing the startup burst](#reduce-the-startup-burst).

Use the `commandPrefix` property of a Core+ worker kind to launch every worker with `nice`, which lowers the worker's scheduling priority relative to the dispatcher and other infrastructure processes. The dispatcher splits the prefix into executable and argument tokens on spaces by default, then appends the full worker command line as one final argument:

```properties
WorkerKind.DeephavenCommunity.commandPrefix=nice -n 10 /bin/sh -c
```

With this setting, the dispatcher launches workers like this:

```bash
nice -n 10 /bin/sh -c '<worker command line>'
```

If you need a different separator between prefix tokens, set `WorkerKind.<name>.commandPrefixDelimiter`. For example, use commas instead of spaces:

```properties
WorkerKind.DeephavenCommunity.commandPrefixDelimiter=,
WorkerKind.DeephavenCommunity.commandPrefix=nice,-n,10,/bin/sh,-c
```

If your installation already uses `commandPrefix` (for example, to run workers with a Kerberos keytab through `k5start`), put `nice -n 10` in front of the existing prefix rather than replacing it, for example: `nice -n 10 k5start -f my-krb5.keytab -U -- /bin/sh -c`.

Nice values range from `-20` (highest priority) to `19` (lowest priority), and Deephaven infrastructure processes run at the default of `0`. Nice values adjust scheduling weight per runnable thread: a thread at nice `10` gets about 10% of the scheduling weight of a thread at nice `0` when they compete for the same core. This does not reserve CPU for infrastructure or cap worker CPU use; a worker with many runnable threads can collectively receive substantially more CPU than a single infrastructure thread. Unprivileged users can raise a process's nice value but not lower it, so this setting requires no additional permissions.

Export `iris-environment.prop` from the Configuration Service, add the property for each Core+ worker kind you have defined (replacing `DeephavenCommunity` with the worker kind name), and import the updated file. For example, on a server with access to the Configuration Service:

```bash
/usr/illumon/latest/bin/dhconfig properties export --directory /tmp iris-environment.prop
vi /tmp/iris-environment.prop
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import /tmp/iris-environment.prop
```

See [editing configuration files](../ops-guide/configuration-files.md#edit-iris-environmentprop) for details. Newly started workers pick up the setting after you restart the dispatchers:

```bash
/usr/illumon/latest/bin/dh_monit restart db_query_server
/usr/illumon/latest/bin/dh_monit restart db_merge_server
```

Restarting a dispatcher stops every worker it is running, so schedule the restart for a maintenance window.

To confirm that the setting is in effect, start a worker and list every process with a positive nice value:

```bash
ps -eo pid,ni,user,args | awk 'NR == 1 || $2 > 0'
```

The worker's `java` process should appear with `10` in the `NI` column. Depending on the shell and on whether the worker runs as a different user, you might also see `sh` or `sudo` rows with the same nice value. Read the `NI` value on the `java` row, because that is the process doing the work. This check does not depend on the worker kind name or install path.

### If the nice value has no effect

A nice value adjusts each thread's scheduling weight; it does not reserve CPU for infrastructure or cap what a worker uses. A worker with many runnable threads, or many workers starting at once, can still take most of the CPU from a dispatcher with only a few busy threads. If workers show `10` in the `NI` column but still starve infrastructure processes, reduce the load itself: [reduce the startup burst](#reduce-the-startup-burst), and if that is not enough, [separate infrastructure and query hosts](#separate-infrastructure-and-query-hosts).

## Reduce the startup burst

Cgroup weights and nice values decide who gets the CPU during contention, but they do not reduce the total work of starting many workers. To shorten and soften the startup burst itself, also consider:

- Lowering `RemoteQueryDispatcher.maxConcurrentStartups` so each dispatcher starts fewer workers at once. See [Query Dispatcher configuration](../pq-controller/dispatcher.md).
- Lowering the number of JIT compiler threads each worker uses. See [JIT compiler optimization](./cpu-optimization.md#jit-compiler-optimization).
- Staggering Persistent Query start times so that fewer queries start at the same moment.

## Separate infrastructure and query hosts

The most robust protection is to keep workers off the hosts that run etcd and the Controller. In a multi-node cluster, run query and merge dispatchers on dedicated query hosts, and run etcd and the other infrastructure processes on hosts that do not launch workers. See [Capacity planning](./capacity-planning.md) for guidance on sizing and placing servers.

## Related documentation

- [CPU optimization](./cpu-optimization.md)
- [Capacity planning](./capacity-planning.md)
- [Query Dispatcher configuration](../pq-controller/dispatcher.md)
