---
title: CPU optimization
---

This guide covers strategies for optimizing CPU performance in your Deephaven cluster through proper resource allocation, JVM tuning, and concurrency configuration.

> [!NOTE]
> For query-level optimization (writing efficient queries), see [Monitor queries](../../performance/monitor-queries.md).

## Understanding CPU usage in Deephaven

Deephaven is a compute-intensive system that relies heavily on CPU resources for data processing, query evaluation, and real-time updates.

### Application processes vs. worker processes

Deephaven distributes CPU load across different process types:

- **Application processes** include the Controller, Query Dispatchers, Data Import Server, and Web API Service. These coordinate work but typically have modest CPU requirements.
- **Worker processes** execute user queries, table operations, and update cycles. These are usually the primary CPU consumers.

### Query types and CPU patterns

Different query workloads exhibit distinct CPU characteristics:

- **Real-time ticking queries**: Continuous CPU usage during update cycles as new data arrives.
- **Large batch/historical queries**: High CPU bursts during initial computation, followed by minimal CPU when results are cached.
- **Interactive queries**: Sporadic CPU usage tied to user actions like sorting, filtering, or chart interactions.

### JIT compilation

The JVM uses Just-In-Time (JIT) compilation to optimize frequently-executed code paths:

- **Warm-up period**: During initial execution, the JIT compiler analyzes code and compiles hot paths to native code, causing higher CPU usage temporarily.
- **Steady state**: After warm-up, compiled code executes more efficiently.
- **Compilation threads**: The JIT compiler uses dedicated threads (controlled by `-XX:CICompilerCount`) that consume CPU during compilation.

You can control [JIT compiler](#jit-compiler-optimization) thread count using [remote processing profiles](../pq-controller/remote-processing-profiles.md).

## Identifying CPU bottlenecks

### Using htop to identify high-CPU processes

```bash
# Install htop if not present
sudo yum install htop  # RHEL/CentOS
sudo apt install htop  # Debian/Ubuntu

# Run htop
htop
```

Key metrics to observe:

- **Process CPU %**: Individual process utilization
- **Load averages**: System-wide CPU load over 1, 5, and 15 minutes
- **CPU bar colors**: User processes (green), system/kernel (red), I/O wait (blue)

To identify specific Deephaven processes:

```bash
# List all Deephaven Java processes
ps -ef | grep java | grep deephaven

# Show CPU usage for a specific worker
top -p $(pgrep -f "worker_.*" | head -n 1)
```

For Kubernetes deployments:

```bash
# View pod CPU usage
kubectl top pods

# View node CPU usage
kubectl top nodes
```

### Analyzing thread dumps

Thread dumps reveal what each thread in a Java process is doing. Use `jstack` to capture:

```bash
# Capture thread dump to a file
sudo jstack -F <pid> > /tmp/threaddump.txt
```

For CPU analysis, capture multiple (3-5) thread dumps spaced 5-10 seconds apart to identify consistently active threads.

**Thread states relevant to CPU analysis:**

- **RUNNABLE** — Thread is executing on CPU or ready to execute.
- **BLOCKED** — Thread is waiting to acquire a lock.
- **WAITING/TIMED_WAITING** — Thread is parked or waiting.

**What to look for:**

- **Hot threads**: Same thread ID in RUNNABLE across multiple dumps
- **Hot methods**: Same method appearing repeatedly in stack traces
- **Thread pool saturation**: All threads in a pool executing simultaneously

See [Troubleshoot Java processes](../troubleshooting/troubleshooting-java.md) for additional techniques.

### Monitoring internal performance tables

Deephaven's [internal tables](../internal-tables/internal-tables.md) provide CPU-related metrics:

```python
from deephaven import agg

# Query performance with CPU time
query_perf = (
    db.live_table("DbInternal", "QueryPerformanceLogCoreV2")
    .where("Date = today()")
    .view(["EvaluationNumber", "UsageNanos", "Description", "PrimaryEffectiveUser"])
    .sort_descending("UsageNanos")
)

# Update cycle performance
update_perf = (
    db.live_table("DbInternal", "UpdatePerformanceLogCoreV2")
    .where("Date = today()")
    .where("UsageNanos > 1_000_000_000")  # Updates taking > 1 second
)
```

```groovy
// Query performance with CPU time
queryPerf = db.liveTable("DbInternal", "QueryPerformanceLogCoreV2")
    .where("Date = today()")
    .view("EvaluationNumber", "UsageNanos", "Description", "PrimaryEffectiveUser")
    .sortDescending("UsageNanos")

// Update cycle performance
updatePerf = db.liveTable("DbInternal", "UpdatePerformanceLogCoreV2")
    .where("Date = today()")
    .where("UsageNanos > 1_000_000_000")  // Updates taking > 1 second
```

**Key tables for CPU analysis:**

| Table                                | Purpose                                              |
| ------------------------------------ | ---------------------------------------------------- |
| `QueryPerformanceLogCoreV2`          | Query-level CPU metrics (`CpuNanos`, `UserCpuNanos`) |
| `QueryOperationPerformanceLogCoreV2` | Per-operation CPU metrics                            |
| `UpdatePerformanceLogCoreV2`         | Update cycle CPU metrics                             |

> [!TIP]
> Compare `CpuNanos` to `UsageNanos`. If `CpuNanos` is close to `UsageNanos`, the operation is **CPU-bound**. If `CpuNanos` is much smaller, it's likely **I/O-bound** or waiting on resources.

## CPU tuning strategies

### Resource allocation

#### Kubernetes CPU configuration

For Kubernetes deployments, configure CPU requests and limits in your Helm values file. See [Kubernetes configuration settings](../kubernetes/kubernetes-configuration-settings.md).

```yaml
resources:
  controller:
    requests:
      cpu: 2000m # 2 CPU cores
      memory: 4Gi
    limits:
      memory: 4Gi

  dis:
    requests:
      cpu: 1000m # 1 CPU core
      memory: 12Gi
```

Worker CPU allocation is specified when creating a Persistent Query under **Advanced Settings** → **CPU Shares**.

#### Bare-metal and VM deployments

For non-Kubernetes deployments, manage CPU allocation through process scheduling and resource limits.

Standard Linux tools (`cgroups`, `nice`/`renice`) apply to Deephaven processes as with any JVM workload.

### JVM tuning

#### Garbage collection tuning

GC parallelism can be tuned to balance CPU overhead against collection efficiency:

```properties
RemoteProcessingRequestProfile.custom.G1Tuned.include.1=G1 GC
RemoteProcessingRequestProfile.custom.G1Tuned.jvmParameter.gcThreads=-XX:ParallelGCThreads=4
RemoteProcessingRequestProfile.custom.G1Tuned.jvmParameter.concGcThreads=-XX:ConcGCThreads=2
```

- `ParallelGCThreads` — Threads used during stop-the-world GC phases
- `ConcGCThreads` — Threads used for concurrent marking (typically 1/4 of ParallelGCThreads)

See [Memory management](./memory-management.md#choosing-and-tuning-garbage-collectors) for comprehensive GC configuration.

#### JIT compiler optimization

Control JIT compiler threads via remote processing profiles:

```properties
# Limit JIT compiler threads for all workers
RemoteProcessingRequestProfile.JitCompilerCount=2

# Different settings per profile
RemoteProcessingRequestProfile.JitCompilerCount.G1 GC=3
```

**Default:** The default value for `RemoteProcessingRequestProfile.JitCompilerCount` is **2**.

**When to adjust:**

- **Reduce to 1** if you observe high CPU during worker startup or have limited cores per worker.
- **Increase to 3-4** if workers with large heaps need faster warm-up and you have CPU headroom.

If this property is removed entirely from the configuration, the JVM's built-in default applies instead. See [Remote processing profiles](../pq-controller/remote-processing-profiles.md#setting-jit-compiler-options-for-workers) for details.

### Concurrency settings

#### Persistent Query startup pool

The Controller uses a thread pool to start Persistent Queries:

```properties
PersistentQueryController.queryStartThreadPoolCoreSize=20  # Default
```

Increase to start more PQs concurrently when many are scheduled to start at the same time. See [PQ Controller](../pq-controller/pq-controller.md).

#### Dispatcher concurrent startups

Each dispatcher limits how many workers it starts simultaneously:

```properties
RemoteQueryDispatcher.maxConcurrentStartups=10  # Default
```

Increase to allow more workers to start concurrently on a single dispatcher. See [Query Dispatcher configuration](../pq-controller/dispatcher.md).

## Quick reference

### JVM parameters

| Parameter               | Purpose               | Notes                                    |
| ----------------------- | --------------------- | ---------------------------------------- |
| `-XX:+UseG1GC`          | Enable G1 GC          | Recommended for Java 11+                 |
| `-XX:ParallelGCThreads` | Full GC parallelism   | Tune based on available cores            |
| `-XX:ConcGCThreads`     | Concurrent GC threads | Tune based on GC behavior                |
| `-XX:CICompilerCount`   | JIT compiler threads  | Default: 2 via remote processing profile |
| `-XX:+UseNUMA`          | NUMA awareness        | Enable on multi-socket systems           |

### CPU utilization monitoring

Appropriate CPU utilization targets are system-dependent.

**To establish and monitor your baseline:**

1. Run `htop` or `top` during typical operations and note average CPU usage
2. Query `ProcessMetricsLogCoreV2` for CPU metrics over time:
   ```python
   cpu_metrics = db.live_table("DbInternal", "ProcessMetricsLogCoreV2").where(
       ["Date = today()", "Name.startsWith(`CPU-`)"]
   )
   ```
   ```groovy
   cpuMetrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2")
       .where("Date = today()", "Name.startsWith(`CPU-`)")
   ```
3. Set alerts when sustained utilization exceeds your observed normal range
4. For VMs, check `steal` in `top` — high steal indicates host contention

### Troubleshooting quick checks

| Symptom                | Likely cause         | Action                     |
| ---------------------- | -------------------- | -------------------------- |
| High CPU, slow queries | Inefficient query    | Check query logs, profiler |
| 100% CPU, no progress  | GC thrashing         | Check heap, GC logs        |
| Unbalanced core usage  | Thread contention    | Capture thread dump        |
| High CPU at startup    | Too many JIT threads | Reduce `CICompilerCount`   |

## Related documentation

- [Performance tuning overview](./overview.md)
- [Capacity planning](./capacity-planning.md)
- [Memory management](./memory-management.md)
- [Remote processing profiles](../pq-controller/remote-processing-profiles.md)
- [Kubernetes configuration settings](../kubernetes/kubernetes-configuration-settings.md)
- [Troubleshoot Java processes](../troubleshooting/troubleshooting-java.md)
- [Internal tables](../internal-tables/internal-tables.md)
- [Monitor queries](../../performance/monitor-queries.md)