--- title: Remote Query Dispatcher configuration --- The Remote Query Dispatcher (simply called the dispatcher) is responsible for creating all Deephaven [workers](./overview.md#workers) in a Deephaven installation. Each installation will have at least two dispatchers: a query server and a merge server. Like all Deephaven processes, the base property file for any Deephaven process is specified by the `Configuration.rootFile` property, and property files are retrieved from the [configuration server](../../sys-admin/configuration/configuration-server-overview.md). - **Query servers** run most workers that do not require the ability to write data. There is at least one query server in a Deephaven installation. Additional query servers can be configured to increase capacity, typically by running an additional query server on each additional machine. - **Merge servers** run privileged workers that are allowed to write to (and delete) disk-backed tables, both [intraday and historical](../../legacy/importing-data/introduction.md#intraday-and-historical-data). Only users with specific privileges are allowed to start merge workers. Many installations only need one merge server, although more can be added if necessary. The following configuration options are likely of most interest to administrators: - [Worker heap size](./worker-heap-size.md) options allow administrators to configure and restrict worker heap sizes. - [Dispatcher usage restrictions](./dispatcher-usage-restrictions.md) allow administrators to restrict the usage of dispatchers. For example, it is possible to only allow certain users to start workers on a specific dispatcher. Merge servers automatically restrict usage to users with certain privileges. A description of the dispatcher's [communications protocols](../architecture/communications.md#dispatcher) may also be of interest. ## Updating properties See [viewing and changing configuration](../configuration/configuration-overview.md#managing-configuration) for details on how to change property files. By default, properties apply to all Deephaven processes. [Stanzas](../configuration/deephaven-properties-files.md#scoped-stanzas) restrict properties so that they only apply to certain processes, such as one specific dispatcher process. - Place properties that are required for all dispatchers (query servers and merge servers) in a property file outside any stanzas, and before any dispatcher-specific properties. - Following this, place properties that are overridden for specific dispatchers within a stanza. For instance, put merge server properties in a `[service.name=dbmerge]` stanza. - `iris_defaults.prop` file first defines [ports](#ports) (properties such as `RemoteQueryDispatcherParameters.queryPort`) outside any stanzas. It then adds the merge-server values within a `[service.name=dbmerge]` stanza. - `iris-endpoints.prop` uses a `host=` stanza to restrict properties to one specific dispatcher. Any property updates require the impacted dispatcher processes to be restarted. ## Properties Dispatcher properties can be broken into several areas: - [Ports](#ports) changes - [Web server](#web-server) - [Other properties](#other-dispatcher-properties) ### Ports The ports that the dispatchers use can be changed if necessary. Query servers and merge servers use different ports so that they can run on the same machine. | Property Name | Description | Default | | -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | | `RemoteQueryDispatcherParameters.queryPort` | The port on which the dispatcher listens for client connections, such as from the Persistent Query Controller (this also requires updating the `iris.db..port` in the [controller's server definitions](./pq-controller.md#server-definitions)). | 22013 (query server), 30002 (merge server) | | `RemoteQueryDispatcher.webserver.port` | The port on which the dispatcher's web service listens - workers connect to this port to establish run-time details. | 8084 (query servers), 8085 (merge servers) | | `RemoteQueryDispatcher.workerServerPorts` | The ports on which the workers started by this dispatcher will listen for client connections. | 23000-23999 (query servers), 25000-25999 (merge servers) | | `RemoteQueryDispatcher.workerServerWebsocketPorts` | The ports on which the workers started by this dispatcher will listen for web client connections. | 24000-24999 (query servers), 26000-26999 (merge servers) | ### Web server The dispatcher provides an optional [web server](../ops-guide/metrics-and-monitoring.md#remote-query-dispatcher) that can be queried for its current state. This is turned off by default, and can be enabled with the following property: ```properties RemoteQueryDispatcher.webserver.enabled=true ``` If the web server is enabled, it uses the following properties: | Property Name | Description | Default | | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | `RemoteQueryDispatcher.webserver.sslRequired` | If this is defined as `true`, an SSL certificate will be required and used for the Remote Query Dispatcher's Web services, and the connection will be over https. | `true` | | `RemoteQueryDispatcher.webserver.authenticationRequired` | If this is `true`, users will be required to authenticate to access the Remote Query Dispatcher's Web services. | `true` | | `RemoteQueryDispatcher.tls.keystore` | If SSL is being used, the PKCS12 keystore that contains the certificate for the encrypted https connection. By default this is the same `webServices-keystore.p12` generated for the Web API service. See [Deephaven public and private key configuration](../configuration/public-and-private-keys.md#web-api-service-keystore) for details on how this keystore is created and how to replace it. | `/etc/sysconfig/illumon.d/auth-user/webServices-keystore.p12` | | `RemoteQueryDispatcher.tls.passphrase.file` | If SSL is being used, a file containing the base64-encoded passphrase used to verify the integrity of the keystore. This should be a restricted file such as `.webapi_passphrase` with read access only for the Deephaven service user. See [Deephaven public and private key configuration](../configuration/public-and-private-keys.md#web-api-service-keystore) for how this file is generated. | `/etc/sysconfig/deephaven/auth-user/.webapi_passphrase` | | `RemoteQueryDispatcher.tls.passphrase` | If SSL is being used, the base64-encoded passphrase to be used to verify the integrity of the keystore. Using `RemoteQueryDispatcher.tls.passphrase.file` is recommended instead of this property. See [Deephaven public and private key configuration](../configuration/public-and-private-keys.md#web-api-service-keystore) for guidance on managing passphrases. | None | If authentication is required, then `sslRequired` must be `true` (i.e., credentials are not allowed to be sent over an insecure connection). Deephaven recommends using passphrase files rather than passphrases. The passphrase file must be protected against reading by non-privileged operating system users. > [!WARNING] > Do not change the following property, as it is essential for workers to function, even if the web server is not enabled. > > ```properties > RemoteQueryDispatcher.websocket.enabled=true > ``` ### Other dispatcher properties | Property Name | Description | Default | | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | | `RemoteQueryDispatcher.shutdownGracePeriodMS` | Optional; when set, the dispatcher should try to terminate all workers and wait up to a specified timeout to accumulate logs before going down when possible. If it is not set then the dispatcher tells the workers to terminate but does not wait for the results. | None | | `RemoteQueryDispatcherParameters.name` | An optional name used for display and log purposes. | | | `RemoteQueryDispatcherParameters.host` | The default dispatcher host to which clients will attempt to connect. | 8084 | | `RemoteQueryDispatcher.maxConcurrentStartups` | The number of workers that a dispatcher will start concurrently. | 10 | | `RemoteQueryDispatcher.pushClassGroups` | Defines the groups that are allowed to push classes to a server. | `*` | | `RemoteQueryDispatcher.adminGroups` | Defines the groups that are allowed to perform administrative dispatcher functions such as terminating workers. | `iris-superusers` | > [!CAUTION] > Be cautious when modifying `RemoteQueryDispatcher.maxConcurrentStartups`. Starting workers consumes significant system resources, so thoroughly test any increases to this value. On some servers, higher system utilization can lead to worker startup timeouts. The dispatcher can be configured to start workers either on a local (bare-metal or VM) server or with Kubernetes pods by using the `RemoteQueryDispatcher.workerControlType` property. This option is automatically configured by the installer so it should not be updated; it is shown here for reference only. The values allowed for this property are: - `LocalProcess` - The dispatcher starts a new process on its server using local scripts. This is the default value. - `Kubernetes` - The dispatcher uses Kubernetes control to start a new pod for each worker. > [!CAUTION] > On `LocalProcess` (bare metal or VM) deployments, the dispatcher shares server memory with all workers it manages. If workers consume more memory than available — due to undersized heap limits, Python native memory growth, or other factors — the Linux OOM killer may terminate any process on the server, including the dispatcher itself. If the dispatcher is killed, all of its workers are also terminated. > > To mitigate this risk, configure appropriate [worker heap limits](./worker-heap-size.md) and use the [Additional Memory](../../interfaces/web/code-studio.md#show-advanced) setting to account for non-heap memory usage. On Kubernetes, workers run in isolated pods with memory limits, so an out-of-memory condition affects only the individual worker pod. Here is the default on a Kubernetes installation. ```properties RemoteQueryDispatcher.workerControlType=Kubernetes ``` ## Related documentation - [Dispatcher communications protocols](../architecture/communications.md#dispatcher) - [Dispatcher usage restrictions](./dispatcher-usage-restrictions.md) - [Worker heap size](./worker-heap-size.md)