Garbage collection log files

Garbage collection (GC) logging provides detailed insights into JVM memory management, helping administrators diagnose performance issues, latency spikes, and memory pressure. Deephaven uses the G1 (Garbage First) garbage collector by default.

Enable GC logging

Services

For Deephaven services (Authentication Server, Data Import Server, etc.), add GC logging flags to the process launch configuration. See Java process launch configuration for instructions on modifying JVM arguments.

Add the following flag to enable GC logging:

-Xlog:gc*:file=<path>::filecount=<count>,filesize=<size>

path — The log file path (e.g., /var/log/deephaven/gc/auth-server-gc.log).
count — Number of rotated log files to retain.
size — Maximum size per log file before rotation (e.g., 10M for 10 megabytes).

Example:

-Xlog:gc*:file=/var/log/deephaven/gc/auth-server-gc.log::filecount=10,filesize=10M

Workers

For Persistent Query workers, GC logging is controlled by the Log GC Details checkbox in the Persistent Query configuration. When enabled (the default), GC events are written to the Process Event Log table.

To query GC log entries for a specific worker:

gcLogs = db.liveTable("DbInternal", "ProcessEventLog")
    .where("Date=today()", "Process=`worker_abc123`", "Level in `stdout`, `stderr`")
    .where("LogEntry.contains(`GC`)")
    .sort("Timestamp")

gc_logs = (
    db.live_table("DbInternal", "ProcessEventLog")
    .where(["Date=today()", "Process=`worker_abc123`", "Level in `stdout`, `stderr`"])
    .where("LogEntry.contains(`GC`)")
    .sort("Timestamp")
)

G1 GC log messages

The G1 garbage collector produces several types of log messages. Understanding these helps diagnose memory issues.

Normal collection events

Young generation pause:

[GC pause (G1 Evacuation Pause) (young) 512M->256M(1024M), 0.0234567 secs]

This is a normal young generation collection. The format shows heap usage before and after collection, total heap size, and pause duration.

Mixed collection:

[GC pause (G1 Evacuation Pause) (mixed) 768M->512M(1024M), 0.0345678 secs]

A mixed collection cleans both young and some old generation regions. These occur after concurrent marking identifies garbage in old regions.

Warning signs

Humongous allocation:

[GC pause (G1 Humongous Allocation) 896M->640M(1024M), 0.0456789 secs]

This indicates allocation of objects larger than half a G1 region. Frequent humongous allocations may indicate inefficient memory usage patterns.

Full GC (Allocation Failure):

[Full GC (Allocation Failure) 950M->512M(1024M), 2.3456789 secs]

A full GC is a stop-the-world event that pauses all application threads. This indicates the G1 collector could not keep up with allocation. If you see this frequently, consider:

Increasing heap size
Using the G1 MarkStackSize 128M profile (see below)

Concurrent mark overflow:

[GC concurrent-mark-reset-for-overflow]

The G1 mark stack overflowed during concurrent marking. This can lead to full GC events. The G1 MarkStackSize 128M profile addresses this issue.

JVM profiles

Deephaven provides predefined JVM profiles for worker GC configuration. Select profiles in the Persistent Query configuration under JVM Profile.

Profile	Description
Default	Uses G1 GC for Java 11+. Administrators can change this default.
G1 GC	Standard G1 configuration. Recommended starting point.
G1 MarkStackSize 128M	G1 with increased mark stack size. Use when seeing `Full GC (Allocation Failure)` or `concurrent-mark-reset-for-overflow` messages.
None	No predefined GC parameters. For advanced users who specify all JVM arguments manually.

Note

The CMS (Concurrent Mark Sweep) garbage collector was removed in Java 17. Deephaven Grizzly+ (2026.01) and later require Java 17 or newer, so CMS is no longer available.

For detailed profile configuration, see Remote processing profiles.

Analysis tools

Several tools can help analyze GC logs:

GCEasy — Web-based GC log analyzer with visualizations.
GCViewer — Open-source desktop application for GC log analysis.
Java Flight Recorder (JFR) — Built-in Java profiling tool that captures GC events along with other performance data.

For general Java troubleshooting techniques, see Troubleshoot Java processes.