Deephaven Release Notes: Version 1.20231218

If you are upgrading from a previous Deephaven release, please refer to the additional Upgrade documentation.

Vermilion+ improves upon the Deephaven Vermilion release. Notably:

• the introduction of Deephaven.ui
• Core version 0.31
• improvements to the Web UI
• Core+ data ingestion and system administration

Overall, Vermilion+ provides more functionality for Core+ workers and makes it simpler to configure and monitor your Deephaven installation.

Web

deephaven.ui

Vermilion+ includes a preview release of Deephaven.ui. Deephaven.ui is a toolkit for developing interactive query experiences. By defining components and callbacks, you can construct a flexible and custom experience for your users. In addition to defining reactive callbacks, you can programatically determine the layout of your results using appropriate containers and widgets. Deephaven.ui subsumes the functionality of parameterized queries from Legacy workers and provides much of the functionality that necessitated developing Swing widgets or Javascript plugins in prior versions of Deephaven. Using simple Python building blocks, Deephaven.ui enables you to easily and efficiently produce a flexible user experience for your application. Executing server-side Python allows you to update the presentation of your UI components or interact with other systems.

Without any front-end development, Deephaven.ui enables you to:

• Produce tables, charts, and custom text laid out in simple rows, columns and tabs.
• Provide richer user input choices including text boxes, checkboxes, and sliders.
• Define table-based actions like right-click context menus, recomputing another table when a row is selected, or making a REST API call.
• Update the UI in response to ticking tables.

Web ACL Editor

Authorized users can now edit ACLs using the Deephaven Web UI. To open the ACL editor select the "+New" tab, then select "Advanced", and "New ACL Editor". You can create and manage users and groups in the first tab and create and manage table ACLs in the second tab.

More Query Types

The Deephaven Web UI can now manage more types of Persistent Queries, eliminating the need for Swing for many types of data management queries. In particular, the following are now supported:

• Import - CSV
• Import - JDBC
• Data merge
• Data validation
• Replay Queries

Community Core 0.31

Deephaven 1.20231218 includes Community Core 0.31. Expanded Python type hints allow more complex return types from Python user-defined functions. Under the hood improvements enable deephaven.ui and Kafka function transformations. Improved query performance tables enable matching operations made as part of the same RPC.

Core+ Data Ingestion

User Tables

Core+ workers can now read and write user tables. Compared to Legacy workers, the interface has been simplified. Centrally appended user tables are always live and accessed with db.live_table. Direct user tables are always historical and accessed with db.historical_table. More operations are now synchronous — for example, the delete call now returns only after observing the deletion (as opposed to Legacy workers, which return after sending the delete request).

lastBy Ingesters

You can now configure a Data Import Server inside of a Core+ worker to serve as a lastBy Ingester. This enables you to serve lastBy data through Barrage to other Core+ or Legacy workers. Additionally, you can now combine a lastBy ingestion with Kafka, enabling queries that rely on Kafka ingestion to initialize more quickly and with fewer resources.

Kafka Function Transformations

Core+ Kafka ingesters use the Community Core specifications for ingesting Kafka payloads. If you need more flexibility than an ingestion spec provides, you can use an update statement to transform the table before writing it to disk.

System Administration

Dynamic DIS Configuration

Instead of defining Data Import Servers in the routing YAML configuration file, you can now use dhconfig dis to dynamically add a DIS without editing the underlying routing file. Together with claims and the service registry, this simplifies adding a Data Import Server to your Deephaven cluster. In this release, after adding or removing a dynamic DIS, you still must restart the TDCP and Tailers as if you had edited the routing file.

Automated Core+ Python Installation

When installing Core+, the Python virtual environment for the worker is automatically created. This eliminates a tedious step that was required per-query-server to enable Core+ workers to use Python.

Status Dashboard

The status dashboard provides data to Prometheus for monitoring a Deephaven installation. Used in conjunction with a Prometheus node exporter that provides Unix-level information, the status dashboard adds Deephaven application level monitoring of the controller, dispatchers, persistent queries, and table data.

Controller Restart

When the persistent query controller restarts, Core+ workers no longer terminate immediately. If the controller process is restored within 60 seconds, it reconnects to Running persistent queries, verifies connectivity, and allows those queries to continue without needing to reinitialize. Legacy workers, as well as Core+ workers that have not yet reached a running state, are still terminated when the controller exits (as in prior releases).

Python 3.10

Vermilion+ adds support for Python 3.10, which is now the default Python version for workers.

Kubernetes

Kafka

The configuration of a Kafka ingester on Kubernetes has been greatly simplified. You can now configure an ingester without manually creating any Kubernetes objects. You can automatically attach a persistent volume to merge workers, making it simple to provision storage of the appropriate size and class for your ingester. All merge workers now automatically expose a Table Data and Tailer port service, which are automatically inserted into the Deephaven service registry. Together with dynamic DIS configuration, this eliminates many of the manual steps that were previously required to setup an ingester.

Process Configuration

Kubernetes installations no longer use "hostconfig" files; instead, additional Java arguments and environment variables are now set directly in the Helm chart.