Detailed Version Log Deephaven 1.202601

Certified versions

No Grizzly Plus versions are certified at this time.

Detailed Release Version Log: Deephaven v2026.01

Detailed Release Candidate Version Log: Deephaven v1.20250219beta

Added support for Java 21 and 25

Deephaven deployments now support Java versions 21 and 25

TimeSeriesFilterGenerator

Core+ workers can now apply a TimeSeriesFilter as part of table ACLs. The io.deephaven.enterprise.acl.generator.TimeSeriesFilterGenerator is constructed with three arguments:

1. The period as a String (e.g. PT15m).
2. The column name to filter.
3. Whether to exclude recent values or past values. For example, with a period of PT15m and excludeRecent=true, only data that is 15 minutes or more old is displayed.

For convenience, the AclFilterGenerator.outOfWindow and AclFilterGenerator.inWindow static methods are provided to include only rows that are less recent than the specified period or more recent the specified period, respectively.

If a TimeSeriesFilterGenerator is applied for a user, then they cannot access the table from Legacy workers.

Add Core+ CSV import job

CSV import persistent queries can now be executed in a Core+ worker. Previously, all CSV import job workers were Legacy.

Since legacy workers will not be supported after Grizzly Plus, users are encouraged to migrate their CSV import jobs to Core+ as soon as possible.

Input Table specification updates

io.deephaven.enterprise.inputtables.ColumnSpec

dataType

ColumnSpec has been updated to properly enforce the expected type restrictions. Previously, callers who used ColumnSpec.Builder.dataType could construct a ColumnSpec with any dataType. Now, ColumnSpec explicitly checks that dataType is one of:

• byte
• char
• short
• int
• long
• float
• double
• Boolean
• String
• Instant

The helper method ColumnSpec.Builder.setDataType, which applies a data type conversion (converts the incoming dataType from boolean to Boolean, and from non-Boolean boxed primitives to primitives) has been deprecated; callers are encouraged to migrate to the more explicit method ColumnSpec.Builder.dataType.

enumValues

The requirements for enumValues remains the same: if dataType is a primitive, the enum values must be an instance of the boxed version of that primitive (for example, if dataType is int, the enumValues instances must be Integers); if dataType is not a primitive, the enum values must be an instance of the same dataType.

The helper method ColumnSpec.Builder.addToEnumValues, which applies an automatic conversion of incoming values from BigDecimal to Double, has been deprecated; callers are encouraged to migrate to the more explicit method ColumnSpec.Builder.addEnumValues or ColumnSpec.Builder.addAllEnumValues.

Core+ Input Table Validation

Core+ Input Tables now validate additions and deletions. In particular, columns created with an enum reject updates containing values that do not belong to the permitted set of values.

Input Table updates and deletions are now validated both for gRPC initiated updates and programatically initiated updates within a worker. Previously, updates initiated within a worker were not validated. The most noticeable change is that in-worker deletion operations now reject invalid key columns. The delete method requires a table argument that contains only key columns; a table with additional columns results in an ArgumentException.

ObjectCodec deprecations

The com.illumon.util.codec.ObjectCodec interface is being deprecated in favor of io.deephaven.util.codec.ObjectCodec. Any custom codecs that have been written against com.illumon.util.codec.ObjectCodec should be updated to implement io.deephaven.util.codec.ObjectCodec instead. The interfaces are otherwise identical.

Built-in codec updates

The following codecs have been re-implemented against io.deephaven.util.codec.ObjectCodec, and should otherwise be identical (see the Notes column for exceptions). While these deprecated codec classes still exist, customers are encouraged to use the replacement codecs going forward.

Future release

While the deprecated classes will be removed in a future release, any data written using the deprecated codecs will continue to be readable indefinitely, and new data written from Schemas that reference the deprecated codecs will inherit the replacement codecs automatically.

Schema interface in Core+

Core+ workers are now able to access Schema and SchemaService implementations without the need to directly access shadowed classes.

Class packages changed

A number of classes have changed packages in order to allow non-shadow package access from Core+.

1. Sub-interfaces and helper-classes of the Legacy Schema are moved to io.deephaven.enterprise.schema.internal, which are included in the shadow jar, but not in a shadowed package. This allows the Core+ Schema interface to use the interfaces/classes "as-is", without using shadow packages. The .internal package is likely to change and is intended for Deephaven internal use only.
2. XML implementations of some interfaces are moved to io.deephaven.enterprise.schema.xml, which are included in the shadow jar, but not in a shadowed package. This allows the Core+ Schema implementation(s) to use the objects directly without using shadow packages. The .xml package is likely to change and is intended for Deephaven internal use only.
3. Schema related exceptions have been moved to io.deephaven.enterprise.schema.exception, which matches the target package in Sanluis.
4. NamespaceSet has been copied to io.deephaven.enterprise.database, which matches the target package in Sanluis.
5. SchemaConstants has been safely moved from com.illumon.iris.db.tables.dataimport to io.deephaven.enterprise.schema. A simple wrapper-class has been left at com.illumon.iris.db.tables.dataimport to prevent Legacy workers from breaking. Core+ workers should prefer io.deephaven.enterprise.schema.SchemaConstants over the shadowed version, which will be removed in Sanluis.
6. A new StorageType enum is added to Core+, which wraps the DefaultTableDefinition.STORAGETYPE_* constants. This enum should be preferred over using the shadowed DefaultTableDefinition.STORAGETYPE_* constants because the package for the constants will change (no longer be shadowed) in Sanluis.

Changes to data routing filtering

This corrects several errors and ambiguities in how filters were applied to sources in Table Data Services (TDS) and failover groups.

Sources with single elements not honoring filter

If the sources of a TDS had only one value, the filter (if any) on that source was not honored. When multiple sources are listed (as in source_composite), the filters on the sources were checked. This creates inconsistent processing, although in most cases the result was the same because the source (incorrectly) queried for table locations would just return an empty set.

Sources without a filter

Sources that are not Data Import Servers or failover groups are now assigned the null filter, instead of the if not claimed filter.

When a source is a Data Import Server or failover group, the filter for that DIS or group is used. Otherwise, the source is some other delegate TDS, and it was assigned an if not claimed filter. This was an incorrect assumption, because the delegate TDS might actually handle a claimed table.

Failover groups

Ad hoc failover groups are now assigned the correct filter for the group members (which must all be identical), instead of the if not claimed filter.

A failover group defines multiple DISes that process the same data and can serve as backups to each other. You can create a failover group using the failoverGroup keyword in each member DIS configuration. This creates a TDS for the group that you can use in the data routing configuration.

You can also define a failover group inline in the sources of a TDS by listing the member names in a list. Deephaven recommends explicit configuration using the failoverGroup keyword to define the group and create a TDS with the given name.

See The dataImportServers keyword for some important caveats when using ad hoc failover groups.

Simplify of default tailer config file, tailerConfigDbInternal.xml

The Tailer process is configured with an XML file described in Tailer XML configuration files. The default tailer configuration file is /usr/illumon/latest/etc/tailerConfigDbInternal.xml. In current installations, binary log files are named in a standard way, and the StandardBinaryLogFileManager class is sufficient for managing them.

This change simplifies the default tailer configuration file and removes a complex configuration section that is largely obsolete.

The new file contains:

The original file contained:

Custom gRPC-web transport for Deephaven JS API

The createClient function in the clientUtils module in auth-nodejs now supports a ConnectOptions object that allows customization of the gRPC-web transport used by the Deephaven JS API client. See TypeScript definitions for details on how to create a custom transport.

Additionally, the clientUtils module in auth-nodejs no longer exports a getWsUrl function. This function should no longer be necessary when using dh.Client, and URL will always be rewritten to use the correct protocol and path as needed.

One Artifact Supports All Java Versions

In previous versions of Deephaven, there was a different set of product tar files for each supported JDK.

Starting in versions 1.20231218.550 and 1.20240517.527, there is only one set of artifacts to support all JDK versions.

The mechanisms for selecting your JDK, such as DH_JAVA_VERSION in cluster.cnf or --jdk* flags passed to containerized deployments, remain unchanged.

The Enterprise installer and Core+ tar files no longer have a -jdkN classifier. For example, deephaven-enterprise-1.20231212.123-jdk17.tar.gz is now deephaven-enterprise-1.20231212.123.tar.gz.

The Enterprise rpm no longer has the jdk major version as deephaven-enterprise minor version; the minor version is always 1. For example, deephaven-enterprise-1.20231212.123-17-1.rpm is now deephaven-enterprise-1.20231212.123-1-1.rpm.

The Core+ tar file has removed its jdkN classifier.
For example, deephaven-coreplus-0.32.0-1.20231212.123-jdk17.tgz is now deephaven-coreplus-0.32.0-1.20231212.123.tgz.

Note that ONLY the filenames and RPM package name have changed. All paths on the filesystem still reflect previous locations.

Rename core+ artifacts to use io.deephaven.coreplus maven groupId

Previously, Core+ artifacts were published as io.deephaven.dnd in Deephaven maven repositories.

These artifacts now use io.deephaven.coreplus for their groupId.

Fix Core+ mis-decoding of certain String and StringSet values

Previously, Core+ reading of Deephaven-format Table data would assume all String and StringSet columns were encoded using ISO-8859-1; this is an issue for encoded characters that aren't the same as in ISO-8859-1. All encodings should now decode correctly.

The issue would manifest differently depending on the character set / encoding:

ASCII

Corresponds to Schema Column with attribute encoding="US_ASCII".

No issues. (ASCII is a subset of ISO-8859-1.)

ISO-8559-1 (also known as Latin-1)

Corresponds to Schema Column with attribute encoding="ISO_8559_1", or a Schema column without the encoding attribute.

No issues.

UTF-8

Corresponds to Schema Column with attribute encoding="UTF_8".

All UTF-8 single-byte characters (ASCII) will decode correctly. All other characters would have decoded incorrectly.

UTF-16

Corresponds to Schema Column with attribute encoding="UTF_16", encoding="UTF_16LE", or encoding="UTF_16BE".

All UTF-16 characters would have decoded incorrectly.

Column Expression Validation

Column expression validation now uses the parsed validator by default, which provides more precise control over which methods users may invoke.

See Formula validation configuration for more details.

HTTP/2 support in Deephaven

The Grizzly Plus version of Deephaven now supports HTTP/2 for all web connections. This requires TLS, and Envoy requires a configuration change to allow WebSocket connections to continue to work.

HTTP+TLS required for all Deephaven web connections

All Deephaven servers now require TLS for web connections. This was already enabled by default in previous versions, except for the DIS.

Change to Envoy configuration to allow HTTP/2 CONNECT method

Deephaven now uses HTTP2 instead of HTTP/1.1 for its web connections. In order to support WebSocket connections over HTTP/2, the Envoy configuration needs to support the CONNECT method.

This change can be applied to the default Envoy configuration of earlier Deephaven versions:

Exception thrown if FileHandle key changes

A safety check has been added for whenever a FileHandle is refreshed to ensure it represents the same physical file as it did previously. When this is not the case, it can lead to very hard to debug downstream issues.

This check is enabled by default. In order to revert to previous behavior, the following flag may be added to a query's configuration:

Core+ Validation Queries

Validation Jobs may now be executed in a Core+ worker. Previously, all Validation Job workers were Legacy. Schema-based Validation queries should migrate seamlessly.

There are a number of steps required to convert custom a Legacy Validation class to a Core+ Validation class.

• A number of supporting classes have changed package-path from com.illumon.iris.validation to io.deephaven.enterprise.validation, including:
  • DataQualityTestCase
  • ValidationTableDescription
  • ImportDetailsUtils
  • ...
• The following Deephaven classes have changed from Legacy to Core+, which may require a number of changes each. See Converting Legacy scripts to Core+ Scripts Cheat Sheet for a more in-depth analysis.
  • Database
    • db.i(...) -> db.liveTable(...)
    • db.t(...) -> db.historicalTable(...)
  • Table
    • Numerous changes
  • TableDefinition
    • Please note that the Core+ TableDefinition does not include grouping information. Grouping information may still be accessed from the Schema or from the shadowed TableDefinition provided by theSchema.
• com.fishlib.io.logger.DelegatingLoggerImpl may be replaced with io.deephaven.internal.log.LoggerFactory.getLogger()
• The ValidationTableDescription has been modified to remove FullTableLocationKey implementation details. The getLocation() method has been replaced with a self-reference to the ValidationTableDescription instance in order to simplify the porting process. The following new methods have been added to the ValidationTableDescription interface:
  • String getTableName() - the Table Name for the associated Table
  • String getNamespace() - the Namespace for the associated Table
  • String getColumnPartition() - the Partition Value (for the Partition Column) for the associated Table
  • String getInternalPartition() - the Internal Partition, if applicable, for the associated Table

Note that RunDataQualityTests#main is not ported. In effect, this disables the RunDataQualityTests command-line utility for Core+ validations. All Core+ validations should be executed in a Data Validation PQ .

On-Demand Asynchronous Cleanup of the DataBufferPool

The AutoReclaimingObjectPool (DataBufferPool) now triggers asynchronous cleanup as soon as the pool occupancy exceeds the configured threshold, rather than waiting for the next scheduled interval. Consequently, the synchronous cleanup path—logged as "Failed to take() an item for thread=<thread name>, initiating synchronous cleanup"—is invoked far less often.

Because of this, the DataBufferConfiguration.cleanupIntervalMillis setting has become largely irrelevant and may be removed in a future release.

Improved support for Iceberg tables in Core+

Core+ has better support for reading Iceberg tables as historical tables. This includes correctly handling Iceberg Schema evolution, nested Iceberg Structs, and proper field id mapping into Parquet files.

Configuration

Iceberg-specific configuration options is out-of-scope for release notes, as they are highly dependent on the specifics of the Iceberg Catalog and data storage setup. In general, Deephaven has good support for the "rest" and "glue" catalog types, but should support other Catalog implementations as well. The Iceberg-specific configurations options that work with Deephaven will be similar to the configuration options needed for Spark (given that Spark is also a JVM-based application). See the Iceberg Documentation, or the documentation for your specific Catalog, for configuration property details.

Verification:

Deployment

Verification:

Tailer cleanup of old binary log files

A new property, log.tailer.fileCleanup.startupDeletionCutoffTime, allows the tailer to find and delete old binary log files when it starts. Further information can be found in the tailer configuration page.

Improved Binary Logger and Listener Generation

Previously, to generate Java binary loggers from Deephaven schema, the IntradayLoggerFactory was the preferred tool. In this release, we have introduced a new binary logger factory that generates much improved logging classes that can operate more independently of the Deephaven system and with far fewer Java-level dependencies. The existing IntradayLoggerFactory continues to generate loggers and listeners for schema elements that are labelled Logger or LoggerListener. The new factory uses LogFormat elements.

See binary Log format configuration for more details.

Deprecated Features and Java Changes

The com.illumon.iris.binarystore.RollingFileManagerBase.ExistingFilePolicy values for DELETE and APPEND, which were deprecated in 2018, have been removed from this release.

The com.illumon.iris.binarystore.RollingFileManagerBase provides the same functionality, but custom extensions must now provide template parameters for the expiration type.

New API to format the log-format suffix of internal partitions

A new builder method IntradayLoggerBuilder#setSuffixInternalPartitionWithLogFormat(String) has been added that lets caller provide a single-argument String.format pattern. The formatted log-format value is appended to the internal-partition name. This overloads the existing IntradayLoggerBuilder#setSuffixInternalPartitionWithLogFormat(boolean) which, when true, appends the suffix using the default %d pattern.

Example:

For internal partition "ABC" and log-format version 4:

• For setSuffixInternalPartitionWithLogFormat(true), the actual partition used would be ABC-4
• For setSuffixInternalPartitionWithLogFormat("%02d"), the actual partition used would be ABC-04

Enabled several HTTP security headers by default

Previously, when used with Envoy, Deephaven had several optional HTTP security headers configured, but left disabled. These defaults are now also available without Envoy, and additionally, some of them are on by default.

Configuration properties prefixed with envoy.add.header. will be set on outgoing responses from Envoy, while http.add.header. will be set on outgoing responses from the server itself, regardless of whether Envoy is used or not.

The following headers are now enabled by default:

• X-XSS-Protection: 1; mode=block - Enables the Cross-Site Scripting (XSS) filter built into some browsers.
• X-Content-Type-Options: nosniff - Prevents the browser from MIME-sniffing a response away from the declared content-type, but instead requires that the server deliberately set a content type.
• Refer-Policy: no-referrer - Disables sending any Referer headers.
• Cross-Origin-Resource-Policy: same-origin - Restricts the ability of a document to load 'no-cors' resources from other origins.
• Cross-Origin-Embedder-Policy: require-corp - Limits embedding cross origin resources into the document.
• Cross-Origin-Opener-Policy: same-origin - Prevents cross-origin documents from being able to interact with pages.

Optional limit on appendLiveTable table size

WritableDatabase.appendLiveTable(...) sends the given table to the Log Aggregator Service as an atomic update. A large enough table can cause the LAS to run out of memory.

You can now set a maximum table size (number of rows) that will be accepted by the appendLiveTable call, by setting the optional property LogAggregatorService.transactionLimit.rows. This check only looks at the number of rows and does not take the number of columns or actual data size into account.

Combined Jars for Core+ clients

Combined jars are now available for Core+ clients allowing you to use a single jar for client applications. The Legacy Core+ client jar is deephaven-ent-client-combined-<version>.jar and the Core+ Client combined jar is deephaven-coreplus-client-combined-<version>.jar

Data Import Server aggressively releases unused resources

The Data Import Server (DIS) now aggressively releases resources that are no longer in use. This change eliminates the slow accumulation of resources that could result from ingesting data over many days.

The DIS serves two functions: ingesting data from tailers, and serving intraday data to queries. When a tailer disconnects (e.g. when it finishes a day's data) and when the last subscriber to a location unsubscribes or disconnects, the relevant table locations are dropped when unused.

New information added to the DIS Web Server

The Data Import Server's web server now includes summary information about the table locations in the active set.

Backport Core+ Merge to Grizzly Plus

Merge Jobs may now be executed in a Core+ worker. Previously, all Merge Job workers were Legacy. The following tickets were backported;

• DH-17611: Write to DH format from Core+ with Core chunks
• DH-17680: Port Merge PQ to Core+ Engine (builder)
• DH-18060: Add Core+ option to Engine drop-down for Merge PQs
• DH-18377: Specify Merge_1 instead of AutoMerge for Core+ Kafka integration tests
• DH-18531: Performance updates for Core+ Merge Jobs
• DH-18463: Add sort formula handling to Core+ MergeJob
• DH-18763: Fix Kafka integration-test failure

Directly create Core AclFilterGenerator for Database ACLs

Previously, the row and column ACLs applied to tables retrieved from a Database must have implemented the FilterGenerator interface, which produces Legacy filters. Optionally, the FilterGenerator can provide a Core filter generator using the getCommunityFilterGenerator method. This means that to build a Core+ AclFilterGenerator for source tables, you must create a corresponding Legacy FilterGenerator.

You can now provide a Core+ AclFilterGenerator directly in the ACL editor. In Core+ workers, the generated Core WhereFilter is applied to the returned table without the need to instantiate a Legacy FilterGenerator.

In a Legacy worker, an error occurs applying the ACL (typically a compilation error). If the user has no other ACLs defined, the table will not accessible. The tables with Core+ filters are not visible in the catalog table if the acl.filter.namespaces or acl.filter.tables Configuration properties are set to true.

The order that ACLs for different groups are generated is not guaranteed, and the user receives a union of the permissions across all their groups with defined ACLs. Therefore, you cannot depend on the existence of a Core+ filter generator to deny access to a table. If an all-access ACL for another of the user's group is processed first, then no other ACLs are processed and the table will be accessible. When evaluating catalog filtering, the system can short circuit evaluation after the first successful ACL application. Similarly, the existence of an another ACL does not ensure access to the table -- if the Core+ AclFilterGenerator fails to compile before another ACL is processed the table is inaccessible.

Removed Jupyter Notebook integration

Server side Jupyter Notebook integration has been removed from Deephaven. The Legacy worker Jupyter Notebook is no longer supported and will not be updated. Use the Deephaven Core+ Python client from Jupyter notebooks beginning in Deephaven 1.20231218 and later.

Optional limit on appendCentral table size (client side)

Database.appendCentral(...) sends the given table to the Log Aggregator Service as an atomic update. A large enough table can cause the LAS to run out of memory.

You can now set a maximum table size (number of rows) that will be accepted by the appendCentral call, by setting the optional property LogAggregatorService.transactionLimit.rows. This check only looks at the number of rows and does not take the number of columns into account. Zero or unset means no limit is enforced.

To make updates larger than the configured limit, either break the table into smaller pieces, or use the RemoteTableAppender directly to make a non-atomic update:

See also Optional server-side limit on appendCentral table size for related server-side changes.

Optional limit on appendCentral table size (server side)

Database.appendCentral(...) and RemoteTableAppender.appendAtomic(...) calls send the given table to the Log Aggregator Service as an atomic update. A large enough table can cause the LAS to run out of memory.

You can now set a maximum table size (number of rows or number of bytes) that will be accepted by the Log Aggregator, by setting the optional properties LogAggregatorService.transactionLimit.rows or LogAggregatorService.transactionLimit.bytes. Zero or unset means no limit is enforced. When the Log Aggregator accumulates more rows or more bytes in a transaction than the configured limit, it will abort the transaction and release the accumulated memory. The client will get an error.

To make updates larger than the configured limit, either break the table into smaller pieces, or use the RemoteTableAppender directly to make a non-atomic update:

See also Optional client-side limit on appendCentral table size for related client-side changes.

Ensure partitioned user table additions and deletions are reflected in future historical table calls

The following code block documents the expected behavior for partitioned user table with additions and deletions:

This reflection of this change is specific to the worker where these deleteTablePartition, addTablePartition calls were made. Other processes that have previously read this historical table may retain the previous behavior (that is, calls to historicalTable may not reflect the newly added partitions).

Note, the previous empty location caching behavior could also be exhibited without using db.deleteTablePartition, as illustrated by the following script:

Automatic Server Selection

Automated server-selection is now turned on by default on new non-Kubernetes installations. To turn it off or change the defaults, edit your iris-environment.prop file and remove or edit the properties from the iris_controller stanza, as described in automated server selection.

Changes to tailer process memory requirements

Tailer now uses constrained and pre-allocated buffer pools for both User and System tables

With default configuration, the tailer now uses constrained and pre-allocated buffer pools for sending data to Data Import Servers. This makes the tailer's memory consumption more predictable, and avoids potential out-of-memory conditions. There are separate pools for User and System tables, governed by the following properties:

The tailer allocates two pools of buffers, one for user tables and one for system tables. Each item in that pool requires two buffers for concurrency, so the memory required will be double the buffer size times the pool capacity.

Total direct memory required for the tailer is approximately 2 * (DataContent.producerBufferSize * DataContent.systemPoolCapacity + DataContent.producerBufferSize.user * DataContent.userPoolCapacity).

Tailer now adjusts to DIS buffer size

When the tailer establishes a connection to a DIS, the processes now exchange configuration information. If the tailer is configured with a larger maximum message size than the DIS can accept, it will reduce it's maximum message size to match the DIS.

Change configuration to handle older tailers sending data to newer Data Import Servers

This section applies only to environments using default configuration values, and where tailers are outside the Deephaven system, either tailing from external systems or from different Deephaven installations. In this scenario, the tailer may send messages that are too large for the DIS to accept.

Older tailers do not accommodate to DIS settings, and may have default configuration settings that allow them to send larger messages than a DIS with default configuration can accept. This typically only happens when the tailer gets behind, or otherwise has file to process with more than 327,710 bytes.

When this happens, the DIS will reject the tailer with a message like:

A similar message might appear in the tailer log.

When an old tailer sends data to a new DIS, the DIS will log a warning like this:

If your environment includes older tailers that send data to current DISes, you should make one of the following configuration changes:

1. update the tailers to the current Deephaven version
2. update the tailer configurations to set the max message size, by setting the following properties in the tailer configuration or properties file:

3. update the DIS max message size to match the tailer max message size by setting the following properties in the DIS configuration or properties file:

Database.inputTableUpdater improvements

The Database.inputTableUpdater method has been updated to return a new, more general interface io.deephaven.enterprise.database.inputtables.InputTableUpdater, which allows callers to have more control over table lineage with input tables. The InputTableUpdater can also be explicitly managed with a liveness scope, and is also now more efficient in caching intermediate operations.

When sourcing both an InputTableUpdater and an input Table view from db, the resulting objects may not share a common lineage:

With the new interface, callers can explicitly derive the corresponding input Table view from the InputTableUpdater:

Note, the python layer does not currently expose a stand-alone equivalent of InputTableUpdater; an input Table view must be created, and in that context, table lineage is already guaranteed.