Controlling query worker heap size

Remote query dispatcher parameters

Each remote query dispatcher instance can be configured as to the resources it allows workers to consume. Dispatchers running on larger servers may allow more resources to be consumed, while smaller dispatchers (e.g., on test nodes) may have less resources available.

Some of these properties apply only to non-interactive workers. Most workers are considered interactive, including embedded consoles and persistent queries. Non-interactive workers are created through the use of specific Deephaven APIs.

• The following parameter defines the total available heap for all worker usage. The combined heap requests for all workers cannot exceed this value in MB:
  • RemoteQueryDispatcher.maxTotalQueryProcessorHeapMB=354304
• The following parameter specifies the total heap (in MB) that the dispatcher has available for active non-interactive query workers. A dispatcher can never run non-interactive workers with more total heap than the value configured by this parameter. If not defined, it uses the value defined by RemoteQueryDispatcher.maxTotalQueryProcessorHeapMB:
  • RemoteQueryDispatcher.maxActiveQueryProcessorHeapMB
• The following parameter defines how many non-interactive queries a dispatcher will allow to run at the same time:
  • RemoteQueryDispatcher.maxConcurrentQueries=20

Persistent Query Controller and Console

When creating a persistent query, a user cannot set its heap size to a larger value than allowed for the chosen server based on that server's configured active query processing heap. If that dispatcher is not running, then a basic maximum is based off the following property. If the property is not defined, a maximum of 1024 GB is enforced:

PersistentQueryController.defaultMaxHeapSizeGB