Custom libraries, overrides, and hotfixes

Custom or third-party libraries and resources — such as JARs, class files, and property files — can be added to designated directories to override and extend your Deephaven installation.

Override directories

Add custom or third-party resources to the following directories, which are placed at the front of the CLASSPATH for Deephaven processes:

• /etc/sysconfig/illumon.d/java_lib — For user and third-party Java libraries (JARs). Any JARs placed in this directory are available on the classpath of Deephaven processes.
• /etc/sysconfig/illumon.d/override — For class files or other Java resources that should override those contained in the java_lib JARs.
• /etc/sysconfig/illumon.d/hotfixes — For JAR files provided by Deephaven support that override classes in the Deephaven installation (/usr/illumon/latest/java_lib). Customer JARs in java_lib take precedence over hotfixes.

See CLASSPATH precedence for the full precedence order.

Restart affected processes

After placing custom libraries, overrides, or hotfixes in the designated directories, restart the affected Deephaven processes.

To restart all processes on a server:

To restart a specific process:

See Startup/Shutdown of Deephaven Processes for detailed restart instructions.

Troubleshooting custom extensions

Custom libraries, extensions, or hotfixes can cause conflicts with existing Deephaven code. After restarting processes with custom extensions, check the log files for errors or warnings.

Applying Deephaven hotfixes

Deephaven support provides hotfixes to address critical issues between releases. A hotfix is a JAR file that overrides specific classes in the Deephaven installation without requiring a full upgrade.

When you receive a hotfix

Deephaven support delivers hotfixes via support ticket when a critical issue requires immediate resolution. The hotfix package includes:

• One or more JAR files.
• Instructions specifying which processes require the hotfix.
• Expected behavior changes or log messages to verify the fix.

Prerequisites

Before applying a hotfix:

• Access: You need SSH access to each Deephaven server and permission to run commands as the irisadmin user (or equivalent).
• Backup: While hotfixes are low-risk, consider backing up your current configuration if you're applying multiple changes.
• Downtime: Applying a hotfix requires restarting affected processes. Plan for brief service interruption.

Step 1: Copy the hotfix JAR file to the hotfixes directory on each affected server:

Step 2: Ensure the file is readable by Deephaven processes:

Step 3: If the hotfix applies to all processes, restart everything:

If the hotfix applies to specific processes only (as indicated in the support instructions), restart only those processes:

See Startup/Shutdown of Deephaven Processes for detailed restart instructions.

Step 4: Check that the hotfix JAR appears in the classpath of the restarted process. You can verify this by checking the process startup logs or running:

Review the process logs for any messages indicating the hotfix is active, as specified in the support instructions.

Multi-node clusters

In a multi-node cluster, you must apply the hotfix to every node where the affected processes run.

Option 1: Copy to each node

Manually copy the hotfix JAR to /etc/sysconfig/illumon.d/hotfixes/ on each node, then restart processes on each node.

Option 2: Use shared storage

If your cluster uses shared storage, you can configure Deephaven to use a shared hotfixes directory. Set the DH_LINK_HOTFIXES property in your cluster configuration:

This creates a symlink from /etc/sysconfig/illumon.d/hotfixes to the shared path. See DH_LINK_HOTFIXES for details.

Rolling restarts

For high-availability deployments, consider a rolling restart strategy:

1. Apply the hotfix to all nodes first.
2. Restart processes on one node at a time.
3. Verify each node is healthy before proceeding to the next.

Removing a hotfix

Remove a hotfix when Deephaven support confirms it's no longer needed — typically after upgrading to a release that includes the fix.

Step 1: Remove the JAR file:

Step 2: Restart affected processes:

Step 3: Check process logs to confirm the system is running correctly without the hotfix.

Hotfixes and upgrades

When upgrading Deephaven with active hotfixes:

1. Contact support — Confirm whether the hotfix is included in the new release.
2. If included — Proceed with the upgrade normally. The hotfix is no longer needed.
3. If not included — Support provides an updated hotfix JAR compatible with the new version. Apply it after the upgrade completes.

Troubleshooting hotfixes

Hotfix not taking effect:

• Verify file location: The JAR must be in /etc/sysconfig/illumon.d/hotfixes/, not /etc/sysconfig/illumon.d/java_lib/.
• Check permissions: The file must be readable by all service accounts running affected processes. Use chmod 644 to ensure world-readable permissions.
• Confirm restart: The affected processes must be restarted after placing the hotfix.
• Check classpath order: Hotfixes take precedence over Deephaven installation JARs but not customer java_lib. See CLASSPATH for the full precedence order.

Process fails to start after applying hotfix:

1. Check the process logs for error messages.
2. Verify the hotfix JAR is not corrupted (compare checksums if provided).
3. Remove the hotfix and restart to confirm the issue is hotfix-related.
4. Contact Deephaven support with the error logs.

Rollback procedure

To revert to the pre-hotfix state:

1. Remove the hotfix JAR from /etc/sysconfig/illumon.d/hotfixes/.
2. Restart affected processes.
3. Contact support to report the issue.

Related documentation

• CLASSPATH precedence
• Startup/Shutdown of Deephaven processes
• Configuration file locations
• Cluster configuration
• Upgrading Deephaven