Kubernetes process launch configuration

This document applies to Deephaven deployments on Kubernetes. For traditional bare-metal or VM-based installations, see Java process launch configuration.

Deephaven on Kubernetes uses containerized deployments where processes run as pods managed by Kubernetes. Configuration is primarily handled through Helm values, ConfigMaps, and environment variables rather than direct file system access.

This document discusses:

The various processes in a Kubernetes Deephaven deployment and how they're deployed as pods.
Configuration methods using Helm values, environment variables, and ConfigMaps.
How to modify process launch parameters such as memory, JVM arguments, and environment variables.
Classpath and custom libraries in containerized environments.
Differences from traditional installations.

Processes

In a Kubernetes deployment, Deephaven processes run as containerized pods. The following deployments are created:

Core services

Deephaven infrastructure services: Multiple deployments run core services:
- configuration-server - Configuration Server provides configuration to other components.
- authserver - Authentication Server handles user authentication.
- controller - Persistent Query Controller manages PQ scheduling and execution.
- webapi - Web API Service serves the web IDE.
- las - Log Aggregator Service combines binary log entries from multiple processes.
etcd StatefulSet: Runs the etcd cluster (typically 3 or 5 replicas for high availability).
query-server deployment: Runs Remote Query Dispatchers that create worker pods for queries and Code Studios.
merge-server deployment: Runs dispatchers for privileged workers that can write to historical data.
dis deployment: Data Import Server handles data ingestion from external sources.

Worker pods

Worker pods: Dynamically created by dispatchers for Persistent Queries and Code Studios. Each worker runs in its own isolated pod.

Command line utilities

Command line tools like dhconfig and dhctl are accessed by executing commands inside the management shell pod:

# Access the management shell
kubectl exec -it deployment/management-shell -n <namespace> -- bash

# Run dhconfig inside the management shell
/usr/illumon/latest/bin/dhconfig properties list

Configuration methods

Kubernetes deployments use different configuration approaches than traditional installations.

Helm values

The primary method for configuring Deephaven on Kubernetes is through Helm values files. Create a values override file to customize your deployment:

# my-values.yaml
userEnv:
  common:
    # Environment variables applied to all pods
    MY_CUSTOM_VAR: "value"

Apply the configuration:

helm upgrade <release-name> deephaven/deephaven -f my-values.yaml -n <namespace>

Environment variables

Environment variables are set through Helm values in the userEnv section. These are injected into pod containers at startup:

userEnv:
  common:
    JAVA_TOOL_OPTIONS: "-XX:+UseG1GC -XX:MaxGCPauseMillis=200"

Worker volumes and secrets

For complex configurations requiring additional volumes, use workerExtraVolumes to mount pre-existing PersistentVolumeClaims to worker pods:

# In Helm values - mount existing PVCs to workers
workerExtraVolumes:
  - name: "custom-data"
    claimName: "my-custom-data-pvc"
    mountPath: "/data/custom"

For secrets, use workerExtraSecrets:

workerExtraSecrets:
  - name: "api-credentials"
    secretName: "my-api-secret"
    mountPath: "/secrets/api"

Deephaven properties

Deephaven properties files are still used in Kubernetes deployments but are managed through the configuration service. Modify properties using dhconfig within the management shell:

# Access the management shell
kubectl exec -it deployment/management-shell -n <namespace> -- bash

# Inside the management shell - export properties to a directory
/usr/illumon/latest/bin/dhconfig properties export --directory /tmp iris-environment.prop

# Edit the file
vi /tmp/iris-environment.prop

# Import the modified file (requires authentication)
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import /tmp/iris-environment.prop

See How do I run commands on the command line for more details.

Modifying launch parameters

Memory configuration

Configure memory limits and requests for each deployment type:

# Service JVM memory configuration (example for a specific service)
# Services use container-aware memory settings by default
process:
  controller:
    jvmArgsMemory: "-XX:+UseContainerSupport -XX:InitialRAMPercentage=70 -XX:MaxRAMPercentage=70"

# Worker memory overhead configuration for dispatchers
dispatchers:
  memoryOverheadMB: 800
  memoryOverheadMultiplier: 0.08

# Resource allocation
# defaults applies to all services; override specific services as needed
resources:
  defaults:
    binlogsSize: 2Gi
    requests:
      cpu: 500m
      memory: 1Gi
      ephemeral-storage: 1Gi
    limits:
      memory: 1Gi
      ephemeral-storage: 1Gi
    tailer:
      requests:
        cpu: 250m
        memory: 1Gi
      limits:
        memory: 1Gi
  # Override for specific services (including binlogsSize)
  query-server:
    binlogsSize: 4Gi
    requests:
      cpu: 2
      memory: 8Gi
    limits:
      cpu: 4
      memory: 16Gi

See Kubernetes configuration settings for detailed memory configuration guidance.

JVM arguments

Add JVM arguments for specific services using the process.<service>.jvmArgsUser Helm value:

# Add JVM arguments for a specific service
process:
  webapi:
    jvmArgsUser: "-Dmy.property=value"

For worker-specific JVM arguments, configure them in the Persistent Query or Code Studio UI when creating the session.

Service-specific configuration

Configure individual deployments:

# Resource allocation for services
# Services include: aclwriter, authserver, configuration-server, controller,
# dis, las, merge-server, query-server, statusdashboard, tdcp, webapi
resources:
  query-server:
    requests:
      cpu: 2
      memory: 8Gi
    limits:
      cpu: 4
      memory: 16Gi
  merge-server:
    binlogsSize: 4Gi
    requests:
      cpu: 2
      memory: 8Gi
    limits:
      cpu: 4
      memory: 16Gi

# etcd connection configuration (etcd is installed separately)
etcd:
  release: "my-etcd-release" # Name of your etcd Helm release
  clientPort: "2379"

Classpath and custom libraries

In Kubernetes deployments, the default classpath includes /customer/deployed/java_lib/ and /customer/deployed/plugin/. Custom libraries are typically added through custom container images.

For details on adding custom libraries, see Kubernetes installation. For general classpath information, see Java process launch configuration.

Differences from traditional installations

Aspect	Traditional Installation	Kubernetes Installation
Process management	`dh_monit`, `systemd`	Kubernetes deployments, StatefulSets
Configuration files	Direct file system access in `/etc/sysconfig`	Helm values, ConfigMaps, exec into pods
Worker creation	Local process spawning	Pod creation via Kubernetes API
Custom libraries	File system directories (`CUSTOMER_JAR_DIR`)	Custom container images
Log access	Direct file access in `/var/log/deephaven`	`kubectl logs`, log aggregator services
Service restart	`dh_monit restart <service>`	`kubectl rollout restart deployment/<name>`
Scaling	Add physical/virtual machines	Scale deployments: `kubectl scale` or Helm
Updates	Installer scripts	Helm upgrade with new image versions

Accessing logs

View logs for pods:

# List pods
kubectl get pods -n <namespace>

# View logs for a specific pod
kubectl logs <pod-name> -n <namespace>

# Follow logs in real-time
kubectl logs -f <pod-name> -n <namespace>

# View logs for a deployment (using configuration-server as example)
kubectl logs deployment/configuration-server -n <namespace>

Restarting services

Restart deployments to pick up configuration changes:

# Restart configuration server
kubectl rollout restart deployment/configuration-server -n <namespace>

# Restart query servers
kubectl rollout restart deployment/query-server -n <namespace>

# Check rollout status
kubectl rollout status deployment/configuration-server -n <namespace>

Scaling services

Scale deployments horizontally:

# Scale query servers using kubectl
kubectl scale deployment/query-server --replicas=3 -n <namespace>

Troubleshooting

For Kubernetes-specific troubleshooting, see Troubleshooting Kubernetes.

Common debugging steps:

# Check pod status
kubectl get pods -n <namespace>

# Describe a pod to see events
kubectl describe pod <pod-name> -n <namespace>

# Check resource usage
kubectl top pods -n <namespace>

# Access a shell in a running pod
kubectl exec -it <pod-name> -n <namespace> -- bash

Kubernetes installation
Kubernetes configuration settings
Troubleshooting Kubernetes
Java process launch configuration (for traditional installations)
Configuration overview