Kubernetes Configuration Settings

Configuration for a Kubernetes installation uses the same Deephaven customization and configuration processes for much of the installation. Because the Helm chart also specifies the equivalent of machine and Helm resources that would be handled by system administrators in a bare-metal (or VM-based) installation, the standard configuration process is augmented by Helm values.

How to apply configuration changes

Deephaven on Kubernetes is typically deployed using Helm. To apply configuration changes, you'll need to:

  1. Create or modify a values override file (e.g., my-values.yaml)
  2. Apply the changes using the Helm upgrade command
# Create or update your values override file
nano my-values.yaml

# Apply the configuration changes
helm upgrade deephaven deephaven/deephaven -f my-values.yaml

You can verify your changes have been applied using kubectl commands, which we'll cover in each section below.

Configuring worker heap overhead

Worker heap configuration ensures that Deephaven's JVM processes have enough memory to operate efficiently without wasting cluster resources or experiencing out-of-memory errors.

The Deephaven system creates a new container for each worker, which hosts a single JVM process. This provides isolation between workers, and the Linux kernel's container runtime automatically limits the resources that the container can use. The bulk of a JVM's memory usage is for the heap, which is where most user defined objects are allocated. However, the JVM also uses off-heap native memory for several reasons including direct memory allocations, garbage collection information, meta-space, classes, compiler caches, and more.

When creating the container for a worker, Deephaven provides two values to Kubernetes to size the container's memory:

  • Request: The minimum amount of heap that Kubernetes allocates to the process.
  • Limit: The maximum amount the process is permitted to use.

Deephaven sets both of these parameters to the same value. If the requested amount of heap is unavailable, then the worker cannot be scheduled by Kubernetes. If the JVM exceeds the limit, then the kernel terminates the process and the container is marked as "OOMKilled" (out-of-memory).

Configuration options

You can configure worker heap overhead in your my-values.yaml file by adding the following:

userEnv:
  common:
    # The percentage by which to increase container memory over Java heap size
    IRIS_PROCESS_HEAP_OVERHEAD_PCT: '30'
    # The fixed amount by which to increase container memory over Java heap size (in MB)
    IRIS_PROCESS_HEAP_OVERHEAD_MB: '512'

Example

If your workers need a 4GB Java heap with 30% overhead for off-heap memory:

userEnv:
  common:
    IRIS_PROCESS_HEAP_OVERHEAD_PCT: '30'

userProc:
  common:
    jvmArgsMemory: '-Xmx4g -Xms4g'

Verification

To verify your worker memory settings have been applied:

# Check the resources assigned to worker pods
kubectl get pods -l app.kubernetes.io/component=worker -o yaml | grep -A 10 resources

# Check if any workers are being OOMKilled
kubectl get pods -l app.kubernetes.io/component=worker | grep OOMKilled

Best practices

  • If workers are being unexpectedly OOM killed, increase memory allocation.
  • Don't set memory values higher than necessary as this reduces the number of workers that can be allocated on the cluster.
  • Monitor memory usage patterns to fine-tune these settings over time.
  • For detailed control over memory allocation, see Configuring Worker Heap Overhead and Deephaven properties.

Configuring process resources

Process resource configuration ensures that each Deephaven service has appropriate CPU, memory, and storage allocations to operate efficiently while making optimal use of cluster resources.

Each Deephaven service pod has two containers:

  1. The main container running the service (e.g., authserver)
  2. A second container running a tailer

The Helm chart provides values for memory, CPU, and a shared directory for binary logs produced by the main Deephaven processes.

Configuration options

The default resource values are:

resources:
  defaults:
    binlogsSize: 2Gi
    requests:
      cpu: 500m
      memory: 1Gi
      ephemeral-storage: 1Gi
    limits:
      cpu: 1
      memory: 4Gi
      ephemeral-storage: 1Gi
    tailer:
      requests:
        cpu: 250m
        memory: 1Gi
        ephemeral-storage: 1Gi
      limits:
        cpu: 1
        memory: 2Gi
        ephemeral-storage: 1Gi

Example

You can override defaults for specific services including "authserver", "configuration-server", "controller", "las", "merge-server", "query-server", "tdcp", and "webapi". For example, to increase the binary log directory size on the Log Aggregator service to 4GB:

resources:
  las:
    binlogsSize: 4Gi

Verification

To verify your resource settings have been applied:

# Check resources for a specific service (e.g., controller)
kubectl get pods -l app.kubernetes.io/component=controller -o yaml | grep -A 15 resources

# Check the persistent volume claims for binary logs
kubectl get pvc -l app.kubernetes.io/part-of=deephaven

Best practices

  • Monitor resource usage and adjust limits based on actual utilization patterns
  • For memory-intensive operations, increase limits on query and merge servers
  • Reserve more CPU for the controller service if managing many workers
  • Ensure binary log size is sufficient based on log verbosity and retention needs

Configuring Data Import Server

Data Import Server (DIS) configuration ensures persistent storage for Intraday data, allowing data to survive pod restarts and providing appropriate storage characteristics for your data volume and access patterns.

The DataImportServer uses Kubernetes PersistentVolumeClaims to store Intraday data. By default, even if the Helm chart is uninstalled, the DIS persistent volumes are preserved to prevent data loss.

Configuration options

The DIS storage defaults for the chart:

dis:
  keepPvs: true # Controls whether PVs are retained after helm uninstall
  storageClassName: 'standard-rwo' # Storage class for the PVs
  # How big /db/Intraday is
  intradaySize: 10Gi
  # How big /db/IntradayUser is
  intradayUserSize: 10Gi

Example

To use a faster storage class and increase volume sizes for a production environment:

dis:
  storageClassName: 'premium-ssd'
  intradaySize: 50Gi
  intradayUserSize: 25Gi

Verification

To verify your DIS storage settings:

# Check the PVCs used by the DIS
kubectl get pvc -l app.kubernetes.io/component=dis

# Examine the storage class and size
kubectl describe pvc -l app.kubernetes.io/component=dis

Best practices

  • Use SSD-backed storage classes for better performance
  • Size volumes based on expected data volume plus 20-30% headroom
  • Consider enabling volume snapshots for backup if available in your cluster
  • Set keepPvs: false only in development environments where data loss is acceptable

Configuring tolerations, node selectors and affinity

Kuberenetes provides several mechanisms to determine where pods are scheduled. Deephaven's Helm chart does not interpret any of these values, but can pass them through from your values.yaml to the various pods that are created by the installation (installation hooks, system-level processes, and workers). In particular, you can configure tolerations, node selectors, and affinity.

By default, no tolerations, selectors or affinity are added. To add tolerations to all created deployments, modify your values.yaml file to include a tolerations block, which is then copied into each pod. For example:

tolerations:
  - key: 'foo'
    operator: 'Exists'
    effect: 'NoSchedule'
  - key: 'bar'
    value: 'baz'
    operator: 'Equal'
    effect: 'NoSchedule'

Adds the following tolerations to each pod (in addition to the default tolerations provided by the Kubernetes system):

Tolerations:                 bar=baz:NoSchedule
                             foo:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Similarly, you can add a nodeSelector or affinity block:

nodeSelector:
  - key: 'foo'
    operator: 'Exists'
    effect: 'NoSchedule'
  - key: 'bar'
    value: 'baz'
    operator: 'Equal'
    effect: 'NoSchedule'

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
            - key: label
              operator: In
              values:
                - value1

Which result in pods containing node selectors like:

Node-Selectors:              key1=value1
                             key2=value2

And affinity as follows:

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
            - key: label
              operator: In
              values:
                - value1
        weight: 1

Configuring system process command arguments and environment variables

System process command arguments

Note

The hostconfig files described in configuring process resources are used to configure process parameters outside of Kubernetes environments, and changing them does not have an effect in a Kubernetes Deephaven deployment.

Deephaven processes in Kubernetes are started with arguments that are found in the helm chart's values.yaml file under the process key. To override a value, add a similar value to your override yaml file. For example, to change the default max memory for the controller, you would add this to your yaml file:

userProc:
  controller:
    jvmArgsMemory: '-Xmx6g -Xms4g -Xmn32m'

If the maximum JVM size for a process is increased with the -Xmx flag, you may also have to adjust the container resource limits. In this example the controller's max JVM memory is increased from the default value of 4G to 6G, so the controller container should be adjusted accordingly with this override for the resource limits in your my-values.yaml file:

# Note - resource limit set higher than the -Xmx6g max JVM size to account for any overhead.
resources:
  controller:
    limits:
      memory: 6.5Gi

There is a common section with values that apply to all processes. You may redefine this under process by copying the original set of values and changing or adding to it. For example, process.common.logRoot contains a JVM property denoting the directory home for process logs:

process:
  common:
    logRoot: '/var/log/deephaven'

You may redefine this value in your my-values.yaml file:

process:
  common:
    logRoot: '/new/logroot/directory'

In the event an arbitrary property should be provided to a system process, there is a process.jvmArgsUser key that may be populated in your my-values.yaml override file. Adding it under common will affect all processes, or it may be added only to specific processes similar to what is done for process arguments.

process:
  common:
    jvmArgsUser: '-Dproperty.common=set.on.all.processes'
  controller:
    jvmArgsUser: '-Dproperty.controller=set.on.controller.only'

Environment variables

Environment variables can be set under the userEnv key. The variable name will be the key, for example:

userEnv:
  common:
    MY_ENV_VAR: 'MY_ENV_VAR_VALUE'

Similar to the system arguments, they may be provided for all processes under the common key, or under a process key like controller to affect only that process environment. There is a systemEnv key in Deephaven's chart.yaml file also, and if there is a value defined in both, then the userEnv value takes precedence.

Worker parameters

The Query Dispatcher allows you to change certain parameters when creating a worker (as described in the Code Studio Advanced Settings page). By default, the query server allows the user to specify the CPU shares, but not other parameters, as that would allow the user to execute arbitrary code on the cluster. The merge server allows specification of all values, as it is by default restricted to users in the iris-schemamanagers group.

To change the permitted Kubernetes control parameters, you can set the Kubernetes.workerValidator property. The three built-in values are:

  • AllowCpu: CPU shares may be specified, but pod template and container image may not
  • AllowAll: All parameters are permitted
  • DenyAll: No changes are permitted

Additionally, if the validator property begins with class:, then a class name that implements the com.illumon.iris.db.tables.remotequery.process.K8SWorkerValidator will be instantiated using the zero-argument constructor.