Performance tuning overview

This guide provides an overview of performance tuning for a Deephaven deployment. Effective performance tuning is an iterative, data-driven process that requires systematic observation, analysis, and adjustment.

This section covers:

• Capacity planning: Sizing deployments for optimal performance based on workload characteristics, including CPU, memory, storage, and network requirements.
• CPU optimization: Optimizing CPU resource allocation, JVM configuration, and parallel processing to maximize computational throughput.
• Memory management: Sizing heap and direct memory, configuring garbage collection, and managing data buffer pools to minimize overhead.
• JVM tuning: Configuring JVM parameters, garbage collection profiles, and heap sizing for Deephaven services and workers.
• Storage tuning: Selecting and configuring storage hardware, filesystems, and protocols (NFS, S3) to maximize data access throughput.
• Caching: Configuring Table Data Cache Proxy (TDCP) and other caching layers to optimize data access.
• Monitoring: Tracking cluster health using internal tables, metrics, and alerting strategies.
• Troubleshooting: Diagnosing and resolving cluster-wide performance issues.

Core philosophy

Effective performance tuning follows a systematic approach:

1. Measure: Use monitoring tools to establish a baseline and observe system behavior under load.
2. Identify bottleneck: Analyze the data to find the primary limiting factor (CPU, memory, or storage).
3. Tune: Make a single, targeted change to address the identified bottleneck.
4. Repeat: Measure again to validate the impact of the change and identify the next bottleneck.

This cycle is crucial because:

• Systems rarely have a single bottleneck — fixing one often reveals the next.
• Making multiple changes simultaneously makes it impossible to determine which change had what effect.
• Performance improvements in one area can shift the bottleneck to another area.

Performance pillars

Performance tuning in Deephaven focuses on the following key areas:

• CPU: The engine for computation. Optimize through proper resource allocation, JVM tuning, and efficient query patterns to maximize throughput.
• Memory: The workspace for data and operations. Tune heap and direct memory sizing, garbage collection, and caching strategies to minimize overhead.
• Storage I/O: The speed of reading from and writing to persistent data. Optimize through hardware selection, filesystem tuning, and storage architecture to maximize throughput.

Understanding which area offers the greatest optimization opportunity is the first step in any performance tuning effort.

Quick reference

Common bottleneck indicators

Tools for observation

Deephaven-specific tools

Deephaven provides several built-in tools for monitoring system performance:

Status dashboard

The Status Dashboard provides Prometheus-compatible metrics that can be visualized in Grafana or other monitoring systems. It monitors:

• Process health (Controller, Dispatchers, Workers).
• Persistent Query status and data latency.
• Certificate expiration.
• Custom metrics from your queries.

See Status Dashboard for access URLs and configuration (port 8112 by default, or via Envoy if configured).

Internal system tables

Deephaven's DbInternal namespace contains tables that record performance data, audit events, and system state. Key tables include:

Performance metrics:

• QueryPerformanceLogCoreV2 — Query-level performance data.
• QueryOperationPerformanceLogCoreV2 — Individual operation timings.
• UpdatePerformanceLogCoreV2 — Update cycle performance.

System events:

• ProcessEventLog — Process lifecycle and errors.
• AuditEventLog — Authentication and authorization events.
• PersistentQueryStateLog — PQ state transitions and errors.

Resource monitoring:

• ProcessMetricsLogCoreV2 — JVM metrics (heap, GC, threads). Disabled by default; enable with IrisLogDefaults.writeDatabaseProcessMetrics=true.
• ResourceUtilization — Dispatcher resource tracking (heap, worker count).

See the Internal tables documentation for complete details.

Example query to examine recent query performance:

Application logs

Application logs are written to /var/log/deephaven/ and include:

• Process-specific logs (e.g., db_query_server.log).
• Binary logs that feed the internal tables.
• Garbage collection logs (if enabled).

For details on log structure and location, see Log files.

System-level tools

Standard Linux utilities provide visibility into host-level resource usage:

• htop: Interactive process viewer showing CPU and memory usage per process/thread.
• iostat: Reports disk I/O statistics including utilization, throughput, and wait times.
• vmstat: System-wide view of CPU, memory, and I/O.
• netstat / ss: Network connection statistics, packet counts, and error rates.
• dstat: Versatile tool combining CPU, disk, network, and other stats in a single view.
• sar: Historical system activity reporting (requires sysstat package).

JVM profiling tools

Java-specific tools are necessary for diagnosing heap, GC, and thread behavior:

• Java Flight Recorder (JFR): Low-overhead profiler that captures detailed JVM and application events.
• jstack: Captures thread dumps to identify deadlocks or CPU-bound threads.
• jmap: Generates heap dumps for memory analysis.
• jstat: Monitors JVM statistics including GC activity.
• VisualVM: GUI tool for monitoring and profiling JVM applications (see Monitor with VisualVM).

Getting started with performance tuning

Where to start

When queries are slow or the cluster feels off, run through these steps before diving into a specific guide:

1. Check process health: Run dh_monit summary to confirm all processes are running and healthy.
2. Find slow queries: Query QueryPerformanceLogCoreV2 for recent operations with high UsageNanos.
3. Check resource usage: Query ResourceUtilization for dispatcher heap usage and worker counts. Query ProcessMetricsLogCoreV2 for JVM heap and GC metrics.
4. Match the symptom: Use the Common bottleneck indicators table to determine which guide to consult.

For unresponsive processes or active incidents, go directly to Troubleshooting performance.

Establish a baseline

Before making any changes, establish baseline metrics for normal operation:

1. System metrics:

   • CPU utilization (average and peak).
   • Memory usage (heap and direct).
   • Disk I/O rates and latency.
   • Network throughput and latency.

2. Deephaven metrics:

   • Query execution times (average, 95th percentile, max).
   • Update cycle durations.
   • GC pause times and frequency.
   • Worker startup times.

3. Business metrics:

   • Data ingestion lag.
   • Dashboard load times.
   • Query response times from user perspective.

Document these baselines and the conditions under which they were measured (time of day, data volumes, concurrent users).

Understand workload patterns

Understanding normal workload patterns helps identify optimization opportunities:

• Expected variations: CPU spikes during market open, increased memory during batch imports, network traffic bursts during data ingestion.
• Growth trends: Gradually increasing resource usage as data volumes or user counts grow.
• Performance characteristics: Typical query execution times, update cycle durations, GC pause patterns.

Prioritize optimization opportunities

Before diving into detailed tuning:

1. Review current resource usage: Identify which resources are most utilized and have the least headroom.
2. Analyze workload patterns: Understand peak vs. normal load, data volumes, and query characteristics.
3. Consider recent growth: Has data volume, user count, or query complexity increased?
4. Evaluate architecture: Are components properly sized for the current workload?

Related documentation

• Scaling to multiple servers
• Metrics and monitoring
• Status dashboard
• Internal tables
• Production log monitoring
• Monitor queries (user guide)