Tailoring worker environments for Persistent Queries and consoles

Remote processing profiles are a key administrative feature in Deephaven for optimizing how worker processes operate. They allow fine-grained control over worker performance, stability, and resource utilization by enabling customized JVM parameters and environment variables.

What do they control?

These profiles apply to the worker processes that execute:

Persistent Queries (PQs)
Interactive consoles (e.g., Web IDE, Swing console)

When would you use them?

System administrators typically use remote processing profiles to:

Tailor environments for new query types with specific resource demands.
Troubleshoot and resolve performance bottlenecks.
Standardize worker configurations across a Deephaven deployment.

Deephaven offers several pre-defined profiles as a starting point, and you can create custom profiles to meet specific needs using controller properties.

Defining profile properties

Properties for remote processing profiles must be accessible to the Persistent Query Controller, the Deephaven Console, and the Web API service. If these properties are defined within a configuration stanza, ensure the stanza targets these services:

[service.name=iris_console|iris_controller|web_api_service] {
    # Your remote processing profile properties here
    # e.g., RemoteProcessingRequestProfile.custom.MyProfile...
}

Profiles

Deephaven provides several remote processing profiles out-of-the-box. These profiles offer tested configurations of JVM parameters tailored for common use cases and Java versions, simplifying setup and providing a solid baseline for performance and stability. Users can select these profiles directly or use them as a foundation for creating custom profiles.

Classic (CMS GC)

The name of this profile is "CMS GC".

The classic profile uses Java's CMS (Concurrent Mark Sweep) garbage collection. It includes a set of JVM parameters designed to limit the number of CPUs a single Remote Query Processor can use for garbage collection. You can override these default parameters by defining the RemoteQueryDispatcher.ClassicJVMParameters property. Additionally, if the RemoteQueryDispatcher.JVMParameters property is defined, it will add further parameters when the JVM starts. These additional parameters will also apply to the Garbage First profile.

Garbage First (G1 GC)

The name of this profile is "G1 GC".

The Garbage First profile uses Java's G1 (Garbage First) garbage collection. It has a default set of parameters that can be overridden with the RemoteQueryDispatcher.G1JVMParameters property. Additionally, if the RemoteQueryDispatcher.JVMParameters property is defined, it will add further parameters when the JVM starts. These parameters will also apply to the Classic profile.

If the RemoteQueryDispatcher.G1NumberHeapRegions property is defined, this profile will set the Java -XX:G1HeapRegionSize parameter based on standard G1 tuning recommendations.

Garbage First with custom MarkStackSize (G1 MarkStackSize 128M)

The name of this profile is "G1 MarkStackSize 128M".

For some queries, the default G1 MarkStackSize values may not be sufficient. This can result in full garbage collection cycles, indicated by messages like the following in the garbage collection log:

[Full GC (Allocation Failure) ...]

You may also see messages such as:

[GC concurrent-mark-reset-for-overflow]

In such cases, this GC profile can help reduce or eliminate full garbage collection cycles. This profile adds the following parameters to the JVM:

-XX:MarkStackSize=128M
-XX:MarkStackSizeMax=256M

None

The name of this profile is "None".

This profile indicates that no additional parameters will be sent to the JVM when it starts. However, you can still define parameters within the Persistent Query or at the console start screen.

Default

The name of this profile is "Default".

This profile allows administrators to specify a default behavior. It uses the RemoteQueryDispatcher.defaultJVMProfile property to define the name of the profile to be used. By default, it uses CMS GC for Java 8 JVMs and G1 GC for Java 11 and later. This setup is useful if an administrator wants to set a default profile, such as classic or G1, and change it for all queries using this default profile.

Choosing the right profile

Selecting the appropriate profile depends on your specific workload, Java version, and performance goals. Here's some guidance:

G1 GC ("Garbage First GC"): This is generally the recommended starting point for applications running on Java 11 or newer. G1 GC aims to provide a good balance between throughput and predictable pause times, and it typically handles larger heap sizes more effectively than CMS.
G1 MarkStackSize 128M ("Garbage First with custom MarkStackSize (G1 MarkStackSize 128M)"): Opt for this profile if you are using G1 GC and observe frequent Full GC (Allocation Failure) messages or [GC concurrent-mark-reset-for-overflow] entries in your worker logs. These indicate that the default G1 MarkStackSize is insufficient for your workload, and this profile provides increased values to mitigate such issues.
None: Choose this profile if you are an advanced user and prefer to specify all JVM parameters manually, either through controller properties for a custom profile that includes "None", or directly at the query or console startup. This gives maximum control but requires careful configuration to ensure stability and performance. It's also useful if you want a minimal JVM configuration without any Deephaven-provided defaults.
Default: Use this profile if you want your workers to automatically use the system-recommended default (CMS GC for Java 8, G1 GC for Java 11+) or if you, as an administrator, want to define a site-wide default profile via the RemoteQueryDispatcher.defaultJVMProfile property. This allows you to change the underlying default for many queries and consoles by modifying a single controller property.

Custom profiles

An administrator can define custom profiles by adding properties to the property file in the controller stanza. Each custom profile is assigned a name, which is used to define the properties. With the exception of the properties that allow setting -Xms for workers, custom properties follow this format:

RemoteProcessingRequestProfile.custom.<profile name>.<parameter type>.<parameter name>=value

<profile name>: The custom profile's name. A custom profile may have several properties, but all of them will start with RemoteProcessingRequestProfile.custom.<profile name>. All properties starting with the same profile name are applied to that profile. The name will be displayed to users in the profile drop-down box.
<parameter type>: The type of parameter, as detailed below. This indicates what is being defined for this profile.
<parameter name>: The parameter's name. Parameter names are arbitrary and are only used to distinguish different parameters of the same type.

Parameter types

The parameter type defines a specific thing that is being applied for a profile. The following parameter types are supported. All examples assume that the profile is called trialProfile.

`jvmParameter`

One or more JVM arguments to be passed to the JVM when it is started. For example, to define a profile with the name trialProfile and pass two JVM parameters:

RemoteProcessingRequestProfile.custom.trialProfile.jvmParameter.deDup=-XX:+UseStringDeduplication
RemoteProcessingRequestProfile.custom.trialProfile.jvmParameter.newProp=-DmyProp=value

`jvmAppendableParameter`

JVM parameters are combined into a single argument passed to the JVM when it starts. Each parameter must be in the form <parameter name>=<parameter value>. All values with the same name will be combined into a single JVM parameter using the system's path separator. For example, the following two properties could be defined for the profile called trialProfile in different property files, both of which are included by the controller:

RemoteProcessingRequestProfile.custom.trialProfile.jvmAppendableParameter.libPath1=-Djava.library.path=/plugins/plugin1/bin
RemoteProcessingRequestProfile.custom.trialProfile.jvmAppendableParameter.libPath2=-Djava.library.path=/plugins/plugin2/bin

When trialProfile is selected, they will be combined into the single parameter ``-Djava.library.path=/plugins/plugin1/bin:/plugins/plugin2/bin` for the workers.

These appendable parameters are included across profiles if one profile includes another. The string used to join them (by default ":" or ";" depending on the operating system) can be changed with the following property (note that this is specific to the property name, not the profile): - RemoteProcessingRequestProfile.joinString.<parameter name>

`environmentVariable`

Environment variables to be defined for the JVM. These are always defined in the format <environment variable name>=<value>.

RemoteProcessingRequestProfile.Custom.trialProfile.environmentVariable.example1=my_env_variable=someValue

`include`

Specifies one or more profiles (by name) to be included in this profile. Multiple profiles can be included either by using a comma-delimited list, or by using multiple .include directives, each with its own name. If a profile is included twice, this is a configuration error and causes an exception.

RemoteProcessingRequestProfile.custom.trialProfile.include=G1 GC,customBaseProfile
RemoteProcessingRequestProfile.custom.trialProfile.include.second=customProfile3

Examples

The following custom profile adds a listening debugger:

RemoteProcessingRequestProfile.custom.Debug_5005.include.1=Default
RemoteProcessingRequestProfile.custom.Debug_5005.jvmParameter.1=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005

The following custom profile is for use with JProfiler:

RemoteProcessingRequestProfile.custom.Profile_8849.include.1=Default
RemoteProcessingRequestProfile.custom.Profile_8849.jvmParameter.1=-agentpath:/opt/jprofiler10.1.2/bin/linux-x64/libjprofilerti.so=port=8849,nowait

Additional properties

The following property is not applied to one profile, but across the system:

RemoteProcessingRequestProfile.defaultProfile - this defines the profile name that is selected for new persistent queries and in the console connection screen (unless another is saved in the user's workspace). Unless a user specifically changes the profile (by checking the "Advanced Options" checkbox and changing the profile), any new persistent query will use this default profile. Changing this property does not change the profile for any existing queries, just for newly-created ones.

To disable thread profiling:

-DThreadProfiler.cpuProfilingEnabled=false causes the CPU time columns in the QueryPerformanceLog, QueryOperationPerformanceLog, and UpdatePerformanceLog tables to be null.
-DThreadProfiler.memoryProfilingEnabled=false causes the memory allocation columns to be null.

Setting -Xms for workers

The JVM parameters -Xmx and -Xms specify memory settings for Java processes, including Deephaven workers.

-Xmx sets the maximum amount of memory allowed for the process. For Deephaven workers, this is always determined by the requested heap size, which is configured by the Heap Usage or Heap Size parameter for each persistent query or console.
-Xms sets the initial memory usage. For Deephaven workers, this is not specified by default.

Remote processing profiles allow rules for specifying the -Xms value using the following syntax:

RemoteProcessingRequestProfile.Xms.<profile name>=<rule>

Profile names can be either predefined profiles (CMS GC, G1 GC, and None) or custom property-defined profiles. Two rules are allowed:

Specifying a number will use that value in megabytes for the -Xms setting, or the worker’s requested heap size if it is less.
Specifying the token $RequestedHeap will use the worker's requested heap size as the -Xms value.

Note that this property is not inherited by profiles that include the one where it is specified. For example, if it’s added to the G1 GC profile, any profiles that include the G1 GC profile must also specify the Xms property if needed.

The following example sets a specific value of 4096 for the G1 GC profile:

# Specify 4096m for the G1 profile's -Xms value
RemoteProcessingRequestProfile.Xms.G1 GC=4096

Note that the G1 GC profile uses the $RequestedHeap token to set -Xms to the same value as the -Xmx parameter.

The following example creates a new custom profile called CustomWithXms, using the token to set -Xms to the requested heap size:

# Create a custom profile which is based off the basic G1 profile but sets -Xms to the requested heap
RemoteProcessingRequestProfile.custom.CustomWithXms.include.1=G1 GC
RemoteProcessingRequestProfile.Xms.CustomWithXms=$RequestedHeap

Setting JIT compiler options for workers

The JVM parameter -XX:CICompilerCount specifies the maximum number of JIT compiler threads allowed for Java processes, including Deephaven workers. If many Persistent Queries start simultaneously, the number of allowed compiler threads can overwhelm the CPUs, leading to Persistent Query startup timeouts. Remote processing profiles allow you to specify the maximum number of JIT compiler threads per worker.

To set the -XX:CICompilerCount value for workers, add properties in the following format:

RemoteProcessingRequestProfile.JitCompilerCount.<profile name>=<number of threads>

If the property is not found for a specific profile name, then the property RemoteProcessingRequestProfile.JitCompilerCount (without a profile name) is used. If neither property is found, the -XX:CICompilerCount JVM argument will not be used for worker JVMs.

Profile names can be either predefined profiles (CMS GC, G1 GC, and None) or custom property-defined profiles.

In the following example, the CMS GC profile allows 4 JIT compiler threads per worker, while other profiles allow 2:

RemoteProcessingRequestProfile.JitCompilerCount.CMS GC=4
RemoteProcessingRequestProfile.JitCompilerCount=2

The default value for RemoteProcessingRequestProfile.JitCompilerCount is 2.

Applying profile changes

Changes to remote processing profiles are not applied instantaneously to all running processes. The method for applying changes depends on the type of worker:

For Persistent Queries: The Persistent Query Controller can dynamically reload profile configurations. Use the controller tool's reload configuration command. Changes will affect PQs started after the reload.
For Interactive Consoles (Swing/Web):
1. The web_api_service must be restarted.
2. Web client users (e.g., Web IDE) must log out and log back in.
3. Swing console users must restart their client application.

Best practices for custom profiles

When creating and managing custom remote processing profiles, consider the following best practices to ensure clarity, maintainability, and optimal performance:

Start with a pre-defined profile: Whenever possible, use a profile (like "G1 GC" or "CMS GC") as a base for your custom profile by using the include directive. This ensures you benefit from a tested set of default parameters and only need to specify your additions or overrides.
```
# Example: Custom profile based on G1 GC with an additional system property
RemoteProcessingRequestProfile.custom.MyG1BasedProfile.include.1=G1 GC
RemoteProcessingRequestProfile.custom.MyG1BasedProfile.jvmParameter.myProp=-Dcom.example.myfeature=true
```
Test thoroughly: Always test new or modified custom profiles in a non-production environment before deploying them to production. Monitor worker logs, garbage collection behavior, and application performance to ensure the changes have the desired effect and don't introduce instability.
Use descriptive names: Choose clear and descriptive names for your custom profiles. The name should give an indication of its purpose or key characteristics (e.g., HighMemoryAnalytics, LowLatencyStreamingWithJFR, DebugProfilePort5005). This makes it easier for users to select the correct profile and for administrators to manage them.

Document your profiles: Keep a record of your custom profiles, the specific settings they contain, and the rationale behind those settings. This can be done through comments in the properties file or separate documentation. This is especially important if multiple administrators manage the Deephaven environment.

# MyProfile: For large data import tasks, increases initial heap and enables string deduplication
# Based on G1 GC for better large heap management
RemoteProcessingRequestProfile.custom.MyProfile.include.1=G1 GC
RemoteProcessingRequestProfile.Xms.MyProfile=8192 # Set initial heap to 8GB, or requested heap if smaller
RemoteProcessingRequestProfile.custom.MyProfile.jvmParameter.deDup=-XX:+UseStringDeduplication

Minimize overrides: Only override or add parameters when necessary. Rely on the included base profile for common settings. Over-customization can make profiles harder to manage and troubleshoot.
Be mindful of inheritance and specificity: Understand how properties like -Xms and JIT compiler counts are applied (they are not always inherited from included profiles and may need to be specified directly for the custom profile). Refer to the specific documentation for these parameters.
Keep profiles focused: If a profile becomes too complex with many disparate settings, consider if it should be split into multiple, more focused profiles.
Regularly review profiles: Periodically review your custom profiles, especially after Deephaven upgrades or changes in workload patterns, to ensure they are still optimal and necessary.

Verification and troubleshooting

After configuring or modifying remote processing profiles, verify that they are being applied correctly using the following methods:

Worker Logs: The most direct way to verify JVM parameters is to inspect the startup logs of a worker process launched with the profile in question.
- For Persistent Queries, these logs can be found in the PEL (Process Event Log) and accessed via performanceOverview or queried directly at DbInternal.ProcessEventLog.
- Search for the command line that launched the Java process. It will list all JVM arguments (-Xmx, -XX:, -D, etc.).
- Confirm that the parameters from your selected profile (and any included profiles) are present.
- Check for environment variables if your profile sets them.
Controller Logs: The Persistent Query Controller logs may indicate which profiles are being loaded and if there are any errors in parsing profile definitions from the properties file. See the log files documentation for log locations.
Deephaven UI/Console:
- When starting a new console or query, the profile selection dropdown should list your custom profiles by the names you defined.
- Some JVM properties or environment variables might be introspectable from within a query console using Java system calls (e.g., System.getProperty("..."), System.getenv("...")), but this depends on the specific parameters.
JMX (Java Management Extensions): If JMX is enabled for your workers, you can use tools like JConsole or VisualVM to connect to a worker JVM and inspect its runtime parameters, memory usage, GC activity, and more. This provides a live view of the JVM's state.

Common issues and troubleshooting steps

Profile Not Appearing in UI:
- Controller Stanza: Ensure your profile properties are in the correct controller stanza (e.g., [service.name=iris_console|iris_controller|web_api_service]) in the controller's properties file.
- Property Syntax: Double-check the syntax: RemoteProcessingRequestProfile.custom.<profile name>.<type>.<param>=value. Typos are common.
- Service Restart/Reload:
  - For changes to affect new PQs: Reload the controller configuration (controller.sh reload configuration).
  - For changes to affect new Swing/Web consoles: Restart the web_api_service. Web clients also need to log out/in, and Swing clients need a restart.
Parameters Not Taking Effect:
- Verification: Use the methods in "Verifying profile application" above to confirm.
- Typos: Check for typos in parameter names or values within your profile definition.
- Inclusion Order/Overrides: If using include, be aware of how parameters from different profiles are combined or overridden. Parameters in the "outer" or more specific profile usually take precedence.
- -Xms and JIT Compiler Count Inheritance: Remember that RemoteProcessingRequestProfile.Xms.<profile name> and RemoteProcessingRequestProfile.JitCompilerCount.<profile name> are not automatically inherited by profiles that include the one where they are specified. They may need to be set directly for the custom profile.
Worker Fails to Start or Behaves Unexpectedly:
- Invalid JVM Parameters: An incorrect JVM option can prevent the JVM from starting. Check the PEL for errors like "Unrecognized VM option" or "Could not create the Java Virtual Machine."
- Resource Issues: Drastically changing memory settings (-Xms, -Xmx) or other resource-related parameters without understanding their impact can lead to instability or out-of-memory errors.
- Start Simple: If a complex custom profile isn't working, try a minimal version (e.g., based on "None" with just one or two parameters) and add settings incrementally, testing at each step.
Performance Degradation:
- GC Tuning: Incorrect garbage collection tuning can significantly impact performance. If you've customized GC parameters, monitor GC logs and metrics closely.
- Too Many/Few Resources: Setting -Xmx too low can lead to frequent GCs and OOMs. Setting it too high unnecessarily can waste memory. Similarly, JIT compiler thread counts need appropriate sizing.
- Revert and Test: If performance degrades after a profile change, revert to a known good profile to confirm the new profile is the cause.
Errors in Controller Logs:
- Check controller logs for any error messages related to parsing RemoteProcessingRequestProfile properties. These can point to syntax errors in your definitions.