Configuration file locations

Deephaven processes obtain their configuration from two primary sources: the Configuration Server (which stores data in etcd) and files stored on disk. Understanding where configuration is stored and how to access it is essential for system administration, troubleshooting, and customization.

In this documentation, "file" refers to both actual files on the filesystem and chunks of configuration data stored in etcd that can be represented as files.

Configuration in etcd

Much of a Deephaven system's configuration is stored in etcd and accessed via the Configuration Server. This centralized approach provides several benefits:

• Consistency: All nodes in a cluster access the same configuration data.
• High availability: etcd's distributed nature ensures configuration remains available even if some nodes fail.
• Dynamic updates: Some configuration changes can be applied without restarting services.

The following configuration types are stored in etcd:

• Properties files - System and application properties that control Deephaven behavior.
• Schemas - Data structure definitions for tables and queries.
• Data routing - Configuration that determines how data flows through the system.
• Permissions (ACLs) - Access control lists defining user permissions.
• User authentication keys - Public keys for user authentication.

Use the dhconfig tool to manage configuration stored in etcd. When the Configuration Server is unavailable, you can access etcd directly using the --etcd flag with dhconfig.

Configuration in the filesystem

While etcd stores centralized configuration, many configuration files reside directly on the filesystem of each server. These files control local server behavior, security credentials, and system-level settings.

Key characteristics of filesystem configuration:

• Local to each server: Each machine has its own copy of filesystem configuration files.
• Requires manual synchronization: Changes must be applied to each relevant server individually.
• Upgrade considerations: Some files persist through upgrades, while others are overwritten by the installer.
• Symbolic link structure: The installer uses symbolic links to separate customer-editable files from system-managed files.

Types of configuration stored in the filesystem:

• Security files - X.509 certificates, private keys, truststores, and password files for secure communication.
• Host configuration - Process startup parameters, memory allocation, and Java system properties (hostconfig).
• Cluster topology - Server roles and network configuration (cluster.cnf).
• Tailer configuration - Data ingestion rules and file parsing specifications.
• Custom resources - JAR files, plugins, calendars, chart themes, and other extensions.
• Service monitoring - M/Monit process supervisor configurations.
• Local authentication - File-based user credentials and superuser permissions (optional).

To edit filesystem configuration, modify files directly on each server and restart the affected services. Use the directories under /etc/sysconfig/illumon.d for customer-editable files to ensure your changes persist through upgrades.

Filesystem directory structure

The following sections describe the key directories and files used for filesystem-based configuration.

Symbolic links

Deephaven uses symbolic links to organize configuration files and simplify upgrades. This design allows the installer to update system files while preserving customer customizations.

How it works:

• /etc/sysconfig/illumon.d - This is the primary directory for customer-editable configuration files. It's actually a symbolic link to a versioned directory under /etc/sysconfig/deephaven/. Many subdirectories within illumon.d are also symbolic links pointing to locations in /etc/sysconfig/deephaven/.

• /etc/sysconfig/deephaven - This directory is managed by the Deephaven installer and contains the actual configuration files. During upgrades, the installer updates this directory and adjusts the symbolic links accordingly.

Why this matters:

When you upgrade Deephaven, the installer creates a new versioned directory with updated default files, then updates the symbolic links to point to the new version. Your customizations in the appropriate directories are preserved because they're stored separately from the default files.

/etc/sysconfig/illumon.d

This is the primary directory for customer-specific configuration files. Files and subdirectories here are preserved during upgrades, making it the safe location for customizations. The directory contains both files you can modify and files managed by the installer.

Key subdirectories and files:

• auth

  Contains security-sensitive files including X.509 certificates, private keys, and password files. This directory has restricted filesystem permissions (typically readable only by root and specific service accounts) to protect sensitive data. The installer and utility scripts manage most files here automatically.

  Common use cases:

  • SSL/TLS certificates for secure communication.
  • Private keys for server authentication.
  • Encrypted password files for service accounts.

• auth-user

  Similar to auth, but with less restrictive permissions that allow Deephaven processes to read the files. Use this directory for certificates and keys that need to be accessible to multiple Deephaven services but should still be protected from general user access.

• calendars

  The default location for custom business calendars. Business calendars define trading days, holidays, and business hours for financial applications. Place custom calendar XML files here to make them available to Deephaven queries and applications.

• chartthemes

  The default location for custom plotting themes for the legacy Swing UI. Chart themes define colors, fonts, and styling for Deephaven visualizations. Add custom theme files here to provide consistent branding or specialized visualization styles for your organization.

  Note

  Chart themes are only used by the legacy Swing UI. The modern web UI uses different theming mechanisms.

• client_update_service

  Contains files to customize the Client Update Service. This service automatically distributes configuration files, JARs, and other resources to remote clients, ensuring they have the correct versions of dependencies and configurations.

  The service prepares files for download by the Swing UI and other remote clients, maintaining version consistency across your deployment.

• dh-config

  Contains connection configuration for the Configuration Server(s). These files specify:

  • Configuration Server host addresses and ports.
  • SSL/TLS certificates for secure connections.
  • Connection timeout settings.
  • Authority information for certificate validation.

  The installer and configuration packager manage these files. In multi-server deployments, this directory may contain subdirectories for each Configuration Server, providing automatic failover capabilities.

• etcd

  Contains connection configuration for direct access to the etcd service. These files are used when accessing etcd directly (bypassing the Configuration Server), such as when using dhconfig --etcd or troubleshooting.

  Note

  These etcd connection files are not present on all Deephaven nodes — only on nodes that require direct etcd access based on their configured roles.

  The files specify etcd endpoints, certificates, and authentication credentials. The installer and configuration packager manage these files automatically.

• integrations

  Contains files for optional integrations with external tools and libraries (such as R and NumPy). Files placed here are automatically synchronized to remote clients by the Client Update Service.

  The Integrations JAR from /usr/illumon/latest/java_lib is also extracted and synchronized to clients. Note that this directory is not on the server classpath—it's specifically for client-side integrations.

• monit

  Preserves customizations to M/Monit process supervisor configurations across upgrades. If you've modified which processes should automatically start or changed monitoring thresholds, those customizations are stored here to survive upgrades.

  See Metrics and monitoring for details on using M/Monit to manage Deephaven processes.

• plugins

  The location for Deephaven plugins that extend system functionality. Plugins can add custom table operations, data sources, authentication providers, or UI components. Place plugin JAR files here to make them available to the Deephaven server.

• resources

  This directory is on the classpath for all Deephaven processes, making files here accessible to Java code via resource loading mechanisms. Use this directory for:

  • Configuration files needed by custom code.
  • Data files that should be accessible to queries.
  • Template files for report generation.
  • Any other resources that Java processes need to load.

  The truststore and password files for Deephaven processes are linked here from /etc/sysconfig/deephaven/trust. Truststore files contain public certificates (not secrets), and the password is used only for file validation.

  Warning

  Do not place Deephaven properties files here. Properties files are loaded from etcd via the Configuration Server, not from the filesystem.

• java_lib

  Contains custom JAR files that should be added to the Deephaven classpath. JAR files placed here take precedence over those in the default Deephaven installation directory (/usr/illumon/latest/java_lib).

  See CLASSPATH for the complete classpath precedence order.

• override

  Contains individual class files (not JARs) that override classes from java_lib and hotfix JARs. This directory has the highest precedence in the classpath, allowing you to replace specific classes without rebuilding entire JARs.

  Caution

  Use this directory sparingly and only for temporary fixes or testing. Overriding core Deephaven classes can cause unexpected behavior and may be overwritten during upgrades.

  See CLASSPATH for the complete precedence order.

• hotfixes

  Contains JAR files that override those in all java_lib directories (both custom and default). Deephaven support may provide hotfix JARs to address critical issues between releases.

  Caution

  Hotfix JARs will be removed during upgrades. Always check with Deephaven support before upgrading if you have active hotfixes, as the fix may be included in the new version or require a new hotfix JAR.

  See CLASSPATH for the complete precedence order.

• schema

  The initial staging location for Deephaven schema files during installation. The installer loads schemas from this directory into etcd, after which the files here are no longer used.

  After installation, schemas are stored in the Configuration Server (backed by etcd). Use dhconfig schemas to view, modify, or add schemas. Changes to files in this directory after installation have no effect.

/etc/sysconfig/deephaven

This directory is managed by the Deephaven installer and contains versioned configuration files. Many files and directories in /etc/sysconfig/illumon.d are symbolic links pointing to files under this directory.

• monit

  Contains M/Monit .conf files that define which services run on this server and their monitoring parameters. The Deephaven installer generates these files based on cluster.cnf during installation and upgrades.

  Any direct changes to these files will be lost during upgrades. To customize M/Monit configurations, use the /etc/sysconfig/illumon.d/monit directory instead. M/Monit loads configurations from /etc/monit.d/deephaven, which links to files here.

/usr/illumon/latest

Contains the Deephaven installation files, including binaries, libraries, and default configurations. This is actually a symbolic link to the current version directory (e.g., /usr/illumon/<version>), which allows multiple versions to coexist and simplifies upgrades.

Key subdirectories:

• bin/ - Deephaven executables and scripts
• java_lib/ - Default Deephaven JAR files
• etc/ - Default configuration files

Descriptions of selected files

Beyond the directory structure, several individual configuration files play critical roles in Deephaven deployments. This section provides detailed information about the most important configuration files, including their purpose, location, format, and how to customize them safely.

hostconfig

The hostconfig file is the central configuration file for Deephaven process startup parameters. It controls critical settings for all Deephaven services and processes started with iris_exec or launch.

Location: /etc/sysconfig/illumon

What it controls:

• Memory allocation (heap size, garbage collection settings)
• User accounts that processes run under
• Java system properties
• Environment variables
• Process-specific startup parameters

How it works:

The hostconfig file uses a two-layer configuration approach:

1. System defaults (/etc/sysconfig/illumon.confs/hostconfig.system) - Contains baseline settings managed by the Deephaven installer. This file is overwritten during upgrades and should never be edited directly.

2. Customer overrides (/etc/sysconfig/illumon) - This is where you make your customizations. The file sources (loads) the system defaults first, then applies your custom settings on top. Your changes here persist through upgrades.

When a Deephaven process starts, it reads the customer hostconfig file, which automatically includes the system defaults and then applies any overrides you've specified.

Common customizations:

• Increasing heap memory for data-intensive workloads (via EXTRA_ARGS with -Xms and -Xmx flags).
• Adding custom Java system properties (via EXTRA_ARGS with -D flags).
• Setting environment variables (using export statements, as hostconfig is a bash script).
• Defining process-specific startup parameters in case statements for each service.

See the Deephaven Operations Guide for detailed examples and best practices.

Tailer config file (tailerConfigDbInternal.xml)

The tailer configuration file defines how Deephaven ingests data from log files and other streaming sources. It specifies file patterns, parsing rules, and table schemas for tailed data.

Default location: /usr/illumon/latest/etc/tailerConfigDbInternal.xml

The file is loaded from the classpath using the property log.tailer.configs=tailerConfigDbInternal.xml. See the Deephaven data tailer documentation for the complete XML schema and configuration options.

To customize the tailer configuration:

1. Copy the default file to /etc/sysconfig/illumon.d/resources/tailerConfigDbInternal.xml.
2. Edit the copied file with your changes.
3. The file in resources/ will take precedence due to classpath ordering.

Alternatively, create a new configuration file and update the log.tailer.configs property to point to it.

cluster.cnf

The cluster configuration file defines the topology and settings for your Deephaven deployment. It specifies which services run on which servers, network addresses, and cluster-wide settings.

Location: /etc/sysconfig/deephaven/cluster.cnf

What it contains:

• Server hostnames and IP addresses
• Service assignments (which servers run which Deephaven components)
• Port configurations
• Cluster-wide settings

Important considerations:

Many Deephaven processes read this file at startup to determine their role in the cluster and how to communicate with other services. After editing cluster.cnf, you typically need to re-run the installer to apply changes across the cluster.

customer_jars.txt

This file lists JAR files that should be automatically distributed to remote clients by the Client Update Service.

Location: /etc/sysconfig/illumon.d/resources/customer_jars.txt

Format: One JAR file path per line.

Use cases:

• Adding custom JAR files to the set of files synchronized by the Client Update Service.
• Distributing custom libraries needed by remote clients.

Example:

authusers.txt

Provides an optional file-based authentication mechanism as an alternative to etcd, MySQL, LDAP, or SAML authentication. This is a legacy feature that may be useful for specific scenarios.

Default location: /etc/sysconfig/illumon.d/resources/authusers.txt

When to use it:

• Local administrative accounts that don't depend on external systems.
• Development and testing environments.
• Backup authentication method.

To enable:

Add the following properties to your Deephaven configuration:

File format:

The file uses htpasswd format (username:hashed_password). Create entries using:

Example entry:

superusers.txt

Provides an optional file-based method for defining superusers and user impersonation permissions. This is a legacy feature that allows designated users to operate as other users.

Default location: /etc/sysconfig/deephaven/superusers.txt

File format: <username>:<operate_as_user>

Special syntax:

• Use * in the operate-as-user field to allow the user to impersonate any user.
• One entry per line.

To enable:

Add the following properties to your Deephaven configuration:

Example entries:

In this example, admin can operate as any user, support can operate as testuser or developer1, and debugger can only operate as developer1.

Use cases:

• Administrators troubleshooting user-specific issues.
• Support staff investigating problems without knowing user passwords.
• Automated processes that need to act on behalf of users.

Related documentation

• Schema overview
• Java process launch configuration
• Deephaven data tailer
• dhconfig configuration tool
• Configuration packager
• Configuration server overview
• Custom calendars and plotting themes
• Custom plugins
• Data routing overview
• etcd
• Deephaven properties files
• How to automate configuration of clients and remote servers
• User authentication keys
• Configuration files