Deephaven Release Notes: Version 1.20230131

Deephaven v1.20230131 (Silverheels) delivers two primary enhancements to users:

1. Deephaven Community in Deephaven Enterprise, which brings the benefits of the Community Core engine into the Enterprise environment, and
2. Kubernetes deployment via Helm chart, which improves the overall deployment process.

In addition, this release brings users:

1. A Shared Swing Console REPL experience
2. Table Data Protocol authentication
3. Dynamic DIS addresses in etcd
4. Pushdown predicates for Match Filter
5. Legacy workers consuming Barrage
6. Query language improvements
7. New web features

These release notes provide an overview of significant changes. See also the accompanying Version Log for a detailed record of each update. The Version Log includes links to change-specific release notes that supplement this page's release notes.

Deephaven Core+

Deephaven created the Community Core product to grow the capabilities of the engine. In Silverheels, we have added the ability to launch Community Workers, running the improved Community engine, within the Deephaven Enterprise ecosystem, known as Core+ (Deephaven Community Core plus Enterprise features). This feature is a Beta feature for customers to experiment with while we work on integrating the Community Core workers more completely.

Kubernetes containerization

Deephaven can now be installed on a Kubernetes cluster using a Helm chart. Each Deephaven system process runs in a Kubernetes pod; when starting a query or merge worker, a new pod is created for that worker. Having distinct containers for each worker improves performance isolation and elasticity. Multiple Deephaven clusters within Kubernetes can also share hardware. The accelerated provisioning process enables development environments with different Deephaven instances to be spun up and down quickly. Deephaven has tested on Amazon EKS and Google GKE, but will work with customers to support other Kubernetes engines of their choice.

Shared Swing Console (Code Studio)

The Deephaven PQ controller now supports dynamically created queries, which enable rich sharing experiences. When a Shared Console is created, its Dynamic Queries can have new objects (tables, plots, etc.) added and/or removed dynamically from their scope. When users or groups are granted access, these objects are available from the Show Table and Show Widget menus in Swing or the Panels menu in Web.

Shared Consoles can be created in Deephaven Classic by selecting the Shared checkbox next to Session Type.

This will open a dropdown to create a Dynamic Query and a dialog to configure its settings:

Select the users who can edit or view the query. Additionally, these consoles can be configured to auto-delete on disconnect. Unlike Persistent Queries, these scripts will not be saved in the Query Config panel to be reopened once a session is closed.

Any changes made by admin users are immediately available to all users. Previously, changes to the PQ's scope could not be shared with users on-the-fly. Note that web users can access and interact with Dynamic Queries, but cannot create Shared Consoles.

Table Data Protocol authentication

Previously, ACL checking was performed by the query worker. The Table Data wire protocol is now authenticated, which adds more robust control of data access, and eliminates all data access without authentication.

Enforcement is at the table level. In practice, this affects Remote Table Data Services (RTDS):

1. Data Import Server (DIS) - requires membership in a set of allowed groups, or * for unrestricted data access.
2. Table Data Cache Proxy (TDCP) - matches the authenticated effective user to the table ACLs to determine access. This is at the table level, row level enforcement is still performed by the worker.

TableDataService now has an authenticate() method. This will use AuthenticationClientManager.getDefault().defaultAuthentication() ("default authentication"). The only time authenticate() needs to be called explicitly is when default authentication changes after a Table Data Service is created.

Dynamic DIS addresses in etcd

Creating additional DIS processes to handle specific tables can be painful. For each DIS, a static address (host and port) is required for data routing. This requires system-wide planning, and is further complicated by in-worker DIS processes and containerization.

To alleviate these issues, we have provided a mechanism to allow these processes to dynamically determine their addresses, and register this dynamic configuration in etcd.

See the version log for more details.

Pushdown Predicates

The Deephaven system goes through great effort to organize data on disk efficiently for processing within queries. The engine now has the capability to evaluate match predicates at the finest grained level within each table. This means that the engine can now take advantage of grouping and indexing wherever it is present, instead of having to treat an entire table as a single unit. Pushdown Predicates also allow for much more efficient filtering of dictionary-backed columns and greatly improve performance when filtering sorted data.

Barrage in Enterprise

​​Deephaven Enterprise now supports subscribing to Barrage tables from Deephaven Community Core workers; anonymously to regular stock community workers, and authenticated via three-way-handshake with an authentication server token for Core+ (Deephaven Community Core plus Enterprise) workers. See the version log for more information.

Query language improvements

withColumnDescription modified Parent Table

The withColumnDescription method would modify the parent table if the parent table already had column descriptions. If the parent table had no column descriptions, it was unchanged. This change will cause the method always to return a new table when the description is change. If your query depended on modifying the parent table when setting column descriptions, those will no longer be present in the UI, and you must update your query to set the description on the proper table.

Parameterized Queries

Deephaven Persistent Queries are essentially applications and analytics running on the platform that generate tables and/or plots that you can add to workspaces or consume via APIs. When computations are performed on tables or chains of tables within a Persistent Query, the engine must visit and consider each and every row in the source tables. Most of the time, this is precisely what you need to do. There are many cases, however, when the vast majority of the result rows will never be used or viewed by a user. In these cases the Persistent Query is doing much more work than is strictly necessary to fill the business needs.

Parameterized Queries are a concept that bridges the gap between the traditional Persistent Query and a query that can be executed “on demand” to produce tailored results without wasting valuable computational time and resources on data that will never be used. Previously available in the Swing console, paramterized queries are now available on the Web UI.

New web features

• Columns can now be formatted into groups using the .layoutHints method or in the UI.
• The Web UI now allows the creation of batch and live merge queries.
• The Web UI now supports viewing multiple charts in a figure.
• "Go to" row and "Go to" value support has been added to tables.
• Tables now show a preview of your selection in the scrollbar, helping you to find your place in large tables.
• Dashboards can be imported and exported from the UI for moving between server environments.
• The Enterprise client has been migrated to TypeScript, resulting in numerous small bug fixes. Customer plugins can now use TypeScript for type safety.