Remote processing profiles

Remote processing profiles specify JVM parameters and environment variables for Persistent Queries and interactive consoles. Several predefined profiles are available, and you can also define new custom profiles using controller properties. To update remote processing profiles:

• The controller can dynamically reload remote processing profiles using the controller tool's reload command. These changes will apply to persistent queries started after the controller's reload command is issued.
• For Swing interactive consoles and web client code studios, the web_api_service must be restarted. Additionally, web clients must log out and log back in, and Swing clients must be restarted.

Properties related to remote processing profiles must be accessible to the persistent query controller, the Deephaven console, and the Web API service. If these properties are placed in a stanza, the stanza must include at least the following service names:

[service.name=iris_console|iris_controller|web_api_service] {
    <properties>
}

Pre-defined profiles

Several profiles are defined by Deephaven.

Classic (CMS GC)

The name of this profile is "CMS GC".

The classic profile uses Java's CMS (Concurrent Mark Sweep) garbage collection. It includes a set of JVM parameters designed to limit the number of CPUs a single Remote Query Processor can use for garbage collection. You can override these default parameters by defining the RemoteQueryDispatcher.ClassicJVMParameters property. Additionally, if the RemoteQueryDispatcher.JVMParameters property is defined, it will add further parameters when the JVM starts. These additional parameters will also apply to the Garbage First profile.

Garbage First (G1 GC)

The name of this profile is "G1 GC".

The Garbage First profile uses Java's G1 (Garbage First) garbage collection. It has a default set of parameters that can be overridden with the RemoteQueryDispatcher.G1JVMParameters property. Additionally, if the RemoteQueryDispatcher.JVMParameters property is defined, it will add further parameters when the JVM starts. These parameters will also apply to the Classic profile.

If the RemoteQueryDispatcher.G1NumberHeapRegions property is defined, this profile will set the Java -XX:G1HeapRegionSize parameter based on standard G1 tuning recommendations.

Garbage First with custom MarkStackSize (G1 MarkStackSize 128M)

The name of this profile is "G1 MarkStackSize 128M".

For some queries, the default G1 MarkStackSize values may not be sufficient. This can result in full garbage collection cycles, indicated by messages like the following in the garbage collection log:

[Full GC (Allocation Failure) ...]

You may also see messages such as:

[GC concurrent-mark-reset-for-overflow]

In such cases, this GC profile can help reduce or eliminate full garbage collection cycles. This profile adds the following parameters to the JVM:

• -XX:MarkStackSize=128M
• -XX:MarkStackSizeMax=256M

None

The name of this profile is "None".

This profile indicates that no additional parameters will be sent to the JVM when it starts. However, you can still define parameters within the Persistent Query or at the console start screen.

Default

The name of this profile is "Default".

This profile allows administrators to specify a default behavior. It uses the RemoteQueryDispatcher.defaultJVMProfile property to define the name of the profile to be used. By default, it uses CMS GC for Java 8 JVMs and G1 GC for Java 11 and later. This setup is useful if an administrator wants to set a default profile, such as classic or G1, and change it for all queries using this default profile.

Custom profiles

An administrator can define custom profiles by adding properties to the property file in the controller stanza. Each custom profile is assigned a name, which is used to define the properties. With the exception of the properties that allow setting -Xms for workers, custom properties follow this format:

RemoteProcessingRequestProfile.custom.<profile name>.<parameter type>.<parameter name>=value

• <profile name>: The custom profile's name. A custom profile may have several properties, but all of them will start with RemoteProcessingRequestProfile.custom.<profile name>. All properties starting with the same profile name are applied to that profile. The name will be displayed to users in the profile drop-down box.
• <parameter type>: The type of parameter, as detailed below. This indicates what is being defined for this profile.
• <parameter name>: The parameter's name. Parameter names are arbitrary and are only used to distinguish different parameters of the same type.

Parameter types

The parameter type defines a specific thing that is being applied for a profile. The following parameter types are supported. All examples assume that the profile is called trialProfile.

jvmParameter

One or more JVM arguments to be passed to the JVM when it is started. For example, to define a profile with the name trialProfile and pass two JVM parameters:

RemoteProcessingRequestProfile.custom.trialProfile.jvmParameter.deDup=-XX:+UseStringDeduplication
RemoteProcessingRequestProfile.custom.trialProfile.jvmParameter.newProp=-DmyProp=value

jvmAppendableParameter

JVM parameters are combined into a single argument passed to the JVM when it starts. Each parameter must be in the form <parameter name>=<parameter value>. All values with the same name will be combined into a single JVM parameter using the system's path separator. For example, the following two properties could be defined for the profile called trialProfile in different property files, both of which are included by the controller:

RemoteProcessingRequestProfile.custom.trialProfile.jvmAppendableParameter.libPath1=-Djava.library.path=/plugins/plugin1/bin
RemoteProcessingRequestProfile.custom.trialProfile.jvmAppendableParameter.libPath2=-Djava.library.path=/plugins/plugin2/bin

When trialProfile is selected, they will be combined into the single parameter ``-Djava.library.path=/plugins/plugin1/bin:/plugins/plugin2/bin` for the workers.

These appendable parameters are included across profiles if one profile includes another. The string used to join them (by default ":" or ";" depending on the operating system) can be changed with the following property (note that this is specific to the property name, not the profile): - RemoteProcessingRequestProfile.joinString.<parameter name>

environmentVariable

Environment variables to be defined for the JVM. These are always defined in the format <environment variable name>=<value>.

RemoteProcessingRequestProfile.Custom.trialProfile.environmentVariable.example1=my_env_variable=someValue

include

Specifies one or more profiles (by name) to be included in this profile. Multiple profiles can be included either by using a comma-delimited list, or by using multiple .include directives, each with its own name. If a profile is included twice, this is a configuration error and causes an exception.

RemoteProcessingRequestProfile.custom.trialProfile.include=G1 GC,customBaseProfile
RemoteProcessingRequestProfile.custom.trialProfile.include.second=customProfile3

Examples

The following custom profile adds a listening debugger:

RemoteProcessingRequestProfile.custom.Debug_5005.include.1=Default
RemoteProcessingRequestProfile.custom.Debug_5005.jvmParameter.1=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005

The following custom profile is for use with JProfiler:

RemoteProcessingRequestProfile.custom.Profile_8849.include.1=Default
RemoteProcessingRequestProfile.custom.Profile_8849.jvmParameter.1=-agentpath:/opt/jprofiler10.1.2/bin/linux-x64/libjprofilerti.so=port=8849,nowait

Additional properties

The following property is not applied to one profile, but across the system:

• RemoteProcessingRequestProfile.defaultProfile - this defines the profile name that is selected for new persistent queries and in the console connection screen (unless another is saved in the user's workspace). Unless a user specifically changes the profile (by checking the "Advanced Options" checkbox and changing the profile), any new persistent query will use this default profile. Changing this property does not change the profile for any existing queries, just for newly-created ones.

To disable thread profiling:

• -DThreadProfiler.cpuProfilingEnabled=false causes the CPU time columns in the QueryPerformanceLog, QueryOperationPerformanceLog, and UpdatePerformanceLog tables to be null.
• -DThreadProfiler.memoryProfilingEnabled=false causes the memory allocation columns to be null.

Setting -Xms for Workers

The JVM parameters -Xmx and -Xms specify memory settings for Java processes, including Deephaven workers.

• -Xmx sets the maximum amount of memory allowed for the process. For Deephaven workers, this is always determined by the requested heap size, which is configured by the Heap Usage or Heap Size parameter for each persistent query or console.
• -Xms sets the initial memory usage. For Deephaven workers, this is not specified by default.

Remote processing profiles allow rules for specifying the -Xms value using the following syntax:

RemoteProcessingRequestProfile.Xms.<profile name>=<rule>

Profile names can be either predefined profiles (CMS GC, G1 GC, and None) or custom property-defined profiles. Two rules are allowed:

1. Specifying a number will use that value in megabytes for the -Xms setting, or the worker’s requested heap size if it is less.
2. Specifying the token $RequestedHeap will use the worker's requested heap size as the -Xms value.

Note that this property is not inherited by profiles that include the one where it is specified. For example, if it’s added to the G1 GC profile, any profiles that include the G1 GC profile must also specify the Xms property if needed.

The following example sets a specific value of 4096 for the G1 GC profile:

# Specify 4096m for the G1 profile's -Xms value
RemoteProcessingRequestProfile.Xms.G1 GC=4096

Note that the G1 GC profile uses the $RequestedHeap token to set -Xms to the same value as the -Xmx parameter.

The following example creates a new custom profile called CustomWithXms, using the token to set -Xms to the requested heap size:

# Create a custom profile which is based off the basic G1 profile but sets -Xms to the requested heap
RemoteProcessingRequestProfile.custom.CustomWithXms.include.1=G1 GC
RemoteProcessingRequestProfile.Xms.CustomWithXms=$RequestedHeap

Setting JIT Compiler options for Workers

The JVM parameter -XX:CICompilerCount specifies the maximum number of JIT compiler threads allowed for Java processes, including Deephaven workers. If many Persistent Queries start simultaneously, the number of allowed compiler threads can overwhelm the CPUs, leading to Persistent Query startup timeouts. Remote processing profiles allow you to specify the maximum number of JIT compiler threads per worker.

To set the -XX:CICompilerCount value for workers, add properties in the following format:

RemoteProcessingRequestProfile.JitCompilerCount.<profile name>=<number of threads>

If the property is not found for a specific profile name, then the property RemoteProcessingRequestProfile.JitCompilerCount (without a profile name) is used. If neither property is found, the -XX:CICompilerCount JVM argument will not be used for worker JVMs.

Profile names can be either predefined profiles (CMS GC, G1 GC, and None) or custom property-defined profiles.

Note that this property is not inherited by profiles that include the one where it is specified. For example, if it’s added to the G1 GC profile, any profiles that include the G1 GC profile must also specify the JitCompilerCount property if needed.

In the following example, the CMS GC profile allows 4 JIT compiler threads per worker, while other profiles allow 2:

RemoteProcessingRequestProfile.JitCompilerCount.CMS GC=4
RemoteProcessingRequestProfile.JitCompilerCount=2

The default value for RemoteProcessingRequestProfile.JitCompilerCount is 2.

Related documentation

• Configuration tools