Deephaven Release Notes: Version 1.20230511

Community Integration

Deephaven created the Community Core product to grow the capabilities of the engine and ecosystem in an open environment. Over the last two years, the Deephaven Community team has improved usability, increased performance, added features, and focused on interoperability:

• For communication, Deephaven Core uses gRPC rather than custom protocols or Java serialization. Deephaven has extended the standard Arrow Flight protocol to support ticking updates. This Apache 2.0 licensed protocol and implementation is called Barrage. This enables rapid development of client applications across languages (e.g., Java, C++, Python, Javascript, R, and Go).
• While the core engine remains highly-optimized Java, the Community Core Python environment is first-class, manually wrapping each class and method in a Pythonic manner. For Java users, Groovy is still available.
• Compared to the legacy Enterprise engine, the Community Core engine includes new operations such as rangeJoin, higher single-threaded performance for many operations (e.g., select, joins and aggregations), window-based aggregations, and improved concurrency.
• The Community Core API and web interface support several standard plotting packages: matplotlib, seaborn, and plotly express. The Community Web UI embraces a plugin architecture, so customer front-end JS teams can extend the Deephaven UX to meet their specific needs.

The 1.20230511 release brings Deephaven Community Core and Enterprise back together. We refer to this package as Core+ (Deephaven Community Core plus Enterprise features). When creating a Code Studio or Persistent Query, you may select the "Core+" or "Legacy" engine. The results of Core+ and Legacy Persistent Queries can be mixed and matched when creating and sharing Dashboards. (Note that Core+ workers do not support Java 8 or the Swing UI). Core+ workers can access the same ticking and historical tables as Legacy workers. Legacy workers can subscribe to Barrage data and integrate ticking Core+ results with existing Legacy workloads.

From a deployment perspective, Core+ workers have much less coupling between the Deephaven system components (e.g., the Controller and Dispatcher) than the existing Legacy worker. This enables a new version of the Community Core engine to be deployed without redeploying the Legacy system. This allows an updated engine to be employed for a small number of workloads before rolling it out widely, or alternatively if one query has a performance or other regression after updating the Core+ engine, it can be rolled back to an existing version. The 1.20230511 initially supports Community Core 0.25.0, but may support future versions. When updating to the next major Deephaven version, the highest Community release supported by 1.20230511 will be supported by the next Deephaven version,thus ensuring continuity.

The Deephaven Enterprise product has evolved over the last decade to integrate with myriad data formats, technologies, and user experiences in response to customer use cases. Core+ does not yet cover all of those use cases, but we are working to close those gaps as efficiently as possible. As always, Deephaven remains committed to building features based on customer demands -- in Community Core or Core+. Core+ provides the mechanism for us to deliver greater value more quickly. Additionally, by developing the Deephaven engine in the open, customer engineering teams can collaborate with Deephaven to provide new Engine functionality. Deephaven will continue to support the Legacy engine with bug fixes, but new innovation will take place in the context of Deephaven Community Core.

Please see the Community Core Documentation for more information.

Data Routing Improvements

One of the challenges of adding new data sources is providing suitable filters for Data Import Servers and Table Data Services. In 1.20230511, instead of writing filters, you can claim a table (or namespace) in the routing file. Combined with the service registry's dynamic endpoints, claims make configuring and managing DISs that handle particular tables or namespaces far simpler than in prior versions.

Internal Partition Indexing

The Data Import Server imports each independent data source into an internal partition. Sometimes, it is known which partition maps to a particular set of data (e.g., based on region, hash codes, or another well-defined scheme). Using this knowledge can improve the performance of intraday queries that must filter large tables by only reading the internal partitions of interest (this is why the performanceOverview provides an option to filter to internal partition).

The 1.20230511 release introduces a new ImportState that causes the DIS's listener to produce a corresponding index table of internal partitions for the keys in the primary table. The Deephaven DbInternal schemas have been updated to produce these indices, and customers can add similar index tables to their Listener definitions.

Web UI Improvements

The Community Core engine is now supported across consoles, dashboards, and persistent queries. The Web UI allows mixing Enterprise engine and Community Core engine outputs in the same dashboard. The Community Core engine also adds support for several new plotting libraries. Support for Matplotlib and Seaborn, and Plotly (static snapshot data only) is now available. Available as an alpha, "Deephaven Express" enhances Plotly Express with support for real-time Deephaven tables and performs aggregations and partitioning automatically in the engine.