CPU optimization

This guide covers strategies for optimizing CPU performance in your Deephaven cluster through proper resource allocation, JVM tuning, and concurrency configuration.

Note

For query-level optimization (writing efficient queries), see Monitor queries.

Understanding CPU usage in Deephaven

Deephaven is a compute-intensive system that relies heavily on CPU resources for data processing, query evaluation, and real-time updates.

Application processes vs. worker processes

Deephaven distributes CPU load across different process types:

Application processes include the Controller, Query Dispatchers, Data Import Server, and Web API Service. These coordinate work but typically have modest CPU requirements.
Worker processes execute user queries, table operations, and update cycles. These are usually the primary CPU consumers.

Query types and CPU patterns

Different query workloads exhibit distinct CPU characteristics:

Real-time ticking queries: Continuous CPU usage during update cycles as new data arrives.
Large batch/historical queries: High CPU bursts during initial computation, followed by minimal CPU when results are cached.
Interactive queries: Sporadic CPU usage tied to user actions like sorting, filtering, or chart interactions.

JIT compilation

The JVM uses Just-In-Time (JIT) compilation to optimize frequently-executed code paths:

Warm-up period: During initial execution, the JIT compiler analyzes code and compiles hot paths to native code, causing higher CPU usage temporarily.
Steady state: After warm-up, compiled code executes more efficiently.
Compilation threads: The JIT compiler uses dedicated threads (controlled by -XX:CICompilerCount) that consume CPU during compilation.

You can control JIT compiler thread count using remote processing profiles.

Identifying CPU bottlenecks

Using htop to identify high-CPU processes

# Install htop if not present
sudo yum install htop  # RHEL/CentOS
sudo apt install htop  # Debian/Ubuntu

# Run htop
htop

Key metrics to observe:

Process CPU %: Individual process utilization
Load averages: System-wide CPU load over 1, 5, and 15 minutes
CPU bar colors: User processes (green), system/kernel (red), I/O wait (blue)

To identify specific Deephaven processes:

# List all Deephaven Java processes
ps -ef | grep java | grep deephaven

# Show CPU usage for a specific worker
top -p $(pgrep -f "worker_.*" | head -n 1)

For Kubernetes deployments:

# View pod CPU usage
kubectl top pods

# View node CPU usage
kubectl top nodes

Analyzing thread dumps

Thread dumps reveal what each thread in a Java process is doing. Use jstack to capture:

# Capture thread dump to a file
sudo jstack -F <pid> > /tmp/threaddump.txt

For CPU analysis, capture multiple (3-5) thread dumps spaced 5-10 seconds apart to identify consistently active threads.

Thread states relevant to CPU analysis:

RUNNABLE — Thread is executing on CPU or ready to execute.
BLOCKED — Thread is waiting to acquire a lock.
WAITING/TIMED_WAITING — Thread is parked or waiting.

What to look for:

Hot threads: Same thread ID in RUNNABLE across multiple dumps
Hot methods: Same method appearing repeatedly in stack traces
Thread pool saturation: All threads in a pool executing simultaneously

See Troubleshoot Java processes for additional techniques.

Monitoring internal performance tables

Deephaven's internal tables provide CPU-related metrics:

// Query performance with CPU time
queryPerf = db.liveTable("DbInternal", "QueryPerformanceLogCoreV2")
    .where("Date = today()")
    .view("EvaluationNumber", "UsageNanos", "Description", "PrimaryEffectiveUser")
    .sortDescending("UsageNanos")

// Update cycle performance
updatePerf = db.liveTable("DbInternal", "UpdatePerformanceLogCoreV2")
    .where("Date = today()")
    .where("UsageNanos > 1_000_000_000")  // Updates taking > 1 second

from deephaven import agg

# Query performance with CPU time
query_perf = (
    db.live_table("DbInternal", "QueryPerformanceLogCoreV2")
    .where("Date = today()")
    .view(["EvaluationNumber", "UsageNanos", "Description", "PrimaryEffectiveUser"])
    .sort_descending("UsageNanos")
)

# Update cycle performance
update_perf = (
    db.live_table("DbInternal", "UpdatePerformanceLogCoreV2")
    .where("Date = today()")
    .where("UsageNanos > 1_000_000_000")  # Updates taking > 1 second
)

Key tables for CPU analysis:

Table	Purpose
`QueryPerformanceLogCoreV2`	Query-level CPU metrics (`CpuNanos`, `UserCpuNanos`)
`QueryOperationPerformanceLogCoreV2`	Per-operation CPU metrics
`UpdatePerformanceLogCoreV2`	Update cycle CPU metrics

Tip

Compare CpuNanos to UsageNanos. If CpuNanos is close to UsageNanos, the operation is CPU-bound. If CpuNanos is much smaller, it's likely I/O-bound or waiting on resources.

CPU tuning strategies

Resource allocation

Kubernetes CPU configuration

For Kubernetes deployments, configure CPU requests and limits in your Helm values file. See Kubernetes configuration settings.

resources:
  controller:
    requests:
      cpu: 2000m # 2 CPU cores
      memory: 4Gi
    limits:
      memory: 4Gi

  dis:
    requests:
      cpu: 1000m # 1 CPU core
      memory: 12Gi

Worker CPU allocation is specified when creating a Persistent Query under Advanced Settings → CPU Shares.

Bare-metal and VM deployments

For non-Kubernetes deployments, manage CPU allocation through process scheduling and resource limits.

Standard Linux tools (cgroups, nice/renice) apply to Deephaven processes as with any JVM workload.

JVM tuning

Garbage collection tuning

GC parallelism can be tuned to balance CPU overhead against collection efficiency:

RemoteProcessingRequestProfile.custom.G1Tuned.include.1=G1 GC
RemoteProcessingRequestProfile.custom.G1Tuned.jvmParameter.gcThreads=-XX:ParallelGCThreads=4
RemoteProcessingRequestProfile.custom.G1Tuned.jvmParameter.concGcThreads=-XX:ConcGCThreads=2

ParallelGCThreads — Threads used during stop-the-world GC phases
ConcGCThreads — Threads used for concurrent marking (typically 1/4 of ParallelGCThreads)

See Memory management for comprehensive GC configuration.

JIT compiler optimization

Control JIT compiler threads via remote processing profiles:

# Limit JIT compiler threads for all workers
RemoteProcessingRequestProfile.JitCompilerCount=2

# Different settings per profile
RemoteProcessingRequestProfile.JitCompilerCount.G1 GC=3

Default: The default value for RemoteProcessingRequestProfile.JitCompilerCount is 2.

When to adjust:

Reduce to 1 if you observe high CPU during worker startup or have limited cores per worker.
Increase to 3-4 if workers with large heaps need faster warm-up and you have CPU headroom.

If this property is removed entirely from the configuration, the JVM's built-in default applies instead. See Remote processing profiles for details.

Concurrency settings

Persistent Query startup pool

The Controller uses a thread pool to start Persistent Queries:

PersistentQueryController.queryStartThreadPoolCoreSize=20  # Default

Increase to start more PQs concurrently when many are scheduled to start at the same time. See PQ Controller.

Dispatcher concurrent startups

Each dispatcher limits how many workers it starts simultaneously:

RemoteQueryDispatcher.maxConcurrentStartups=10  # Default

Increase to allow more workers to start concurrently on a single dispatcher. See Query Dispatcher configuration.

Quick reference

JVM parameters

Parameter	Purpose	Notes
`-XX:+UseG1GC`	Enable G1 GC	Recommended for Java 11+
`-XX:ParallelGCThreads`	Full GC parallelism	Tune based on available cores
`-XX:ConcGCThreads`	Concurrent GC threads	Tune based on GC behavior
`-XX:CICompilerCount`	JIT compiler threads	Default: 2 via remote processing profile
`-XX:+UseNUMA`	NUMA awareness	Enable on multi-socket systems

CPU utilization monitoring

Appropriate CPU utilization targets are system-dependent.

To establish and monitor your baseline:

Run htop or top during typical operations and note average CPU usage

Query ProcessMetricsLogCoreV2 for CPU metrics over time:

cpuMetrics = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2")
    .where("Date = today()", "Name.startsWith(`CPU-`)")

cpu_metrics = db.live_table("DbInternal", "ProcessMetricsLogCoreV2").where(
    ["Date = today()", "Name.startsWith(`CPU-`)"]
)

Set alerts when sustained utilization exceeds your observed normal range
For VMs, check steal in top — high steal indicates host contention

Troubleshooting quick checks

Symptom	Likely cause	Action
High CPU, slow queries	Inefficient query	Check query logs, profiler
100% CPU, no progress	GC thrashing	Check heap, GC logs
Unbalanced core usage	Thread contention	Capture thread dump
High CPU at startup	Too many JIT threads	Reduce `CICompilerCount`