Contributors at Deephaven are focused on doing everything possible to make its engine and APIs easy to use. We want Deephaven to be deployable via various patterns, as it is an empowering solution for a large range of data-driven applications and analytical experiences.
The complete 0.16.0 and 0.17.0 release notes can be found on GitHub.
Highlights
Simplified Docker
Deephaven has been reorganized to reduce complexity. Previously, the services itemized below each respectively had their own image. They’ve now been centralized in a single container. In addition to serving the deephaven-core engine, this single image is able to:
- serve the Web-UI directly,
- provide a proxy for gRPC-web, so browsers can communicate via gRPC without an additional, external proxy,
- supply a Java implementation that will be able to support SAML, OpenID Connect, and OAuth as a foundation for (work-in-progress) authentication, without the need for a separate server.
This has been enabled by Deephaven’s transition from Netty to Jetty.
The image ghcr.io/deephaven/server:0.17.0 is now Jetty-based, simplifying your orchestration efforts and enhancing the feature set the server itself can provide.
For example, a minimal setup via Docker is now as easy as:
docker run –-rm -p 10000:10000 ghcr.io/deephaven/server:0.17.0
As part of this transition, the Web UI’s IDE notebook support was rewired to rely on a gRPC service that is indistinguishable to external middleware. Please be aware, that in doing so, we have placed information to support the UI experience in a directory now called storage. (It was previously named data).
The official release notes provide more detail about Jetty, and an explanatory, technical blog post on the topic is currently being crafted by Deephaven developers.
Real-time plotting widgets available in Jupyter
Users of our Web IDE know the benefits of having real-time, ticking tables available interactively in their GUI. In the release notes for version 0.15.0, we highlighted those same dynamic table experiences being available as real-time, updating widgets in Jupyter.
With the current release, that paradigm has now been extended to real-time plots. Launching Deephaven in Jupyter remains the same, but deephaven_ipywidgets now includes plots. The example below uses standard Deephaven plotting libraries on an updating table source. Interacting with these ticking plots in Jupyter should empower modern collaboration.
Using Python variables and functions in queries is now even easier
Many use cases rely on Python functions and variables being used directly in Deephaven queries. An update() statement is a good example here.
With this release, Deephaven now allows all Python variables that are within scope to be used in queries without any additional coding. Here is some brief exposition describing a few usage patterns, but by quick example, the pattern used below will work for variables, functions, and even more complex Python objects, like set comprehensions.
#an example of a function’s scope being automatically handled
def query_logic(table, b):
return table.where("X > b").update("B = b + 9")
t1 = query_logic(source, 1)
t2 = query_logic(source, 2)
The C++ Client API now supports Streaming Table updates
The C++ client API has long supported receiving static tables and snapshots, as well as sending dynamic data from the client to the server. Much of Deephaven’s special sauce, however, relates to real-time analytics and applications, and users will often want to feed these derived, dynamic data to the client.
The client API now supports such patterns. Deephaven’s subscription model allows client code to maintain a local replica of a server-side table. Having a client-side replica enables fast local access to the data and allows for custom calculations to be performed, e.g., by third-party C++ libraries. The replica is kept up to date in real time by our efficient update protocol as the table undergoes arbitrary mutations (adds, removes, modifies). Furthermore a novel versioning technique allows client code to work on a consistent snapshot of the table, even while further updates are concurrently arriving.
These are powerful capabilities. Two of our developers introduce the pertinent concepts in a three-part video series. Here are the first and second installments. (The final episode will drop soon.) Reference documentation and examples are also on their way.
General improvements
Server-side java app-mode applications are now configurable using Java’s ServiceLoader.
Snapshot performance for large tables was meaningfully improved. Messages are now broken into smaller pieces based on the client’s request or appropriate heuristics, allowing for pipelining of snapshot transmission. This is relevant both for worker-to-worker communications, client application workflows.
The engine is now using a different strategy for row set construction in
partition_by()andgroup_by, thereby greatly improving performance for large cardinality aggregations.
Behind-the-scenes work to support soon-to-arrive tools
An integration of Deephaven's gRPC API's session management with the Arrow Flight's Authorization specification. This is an important predecessor to authentication and authorization interfaces that are on their way.
Encoding stats are now being properly written with their corresponding data. This is fundamental to support predicate pushdown, which will soon be ported to deephaven-core. That is an important capability for efficiently interacting with Parquet (for example) to source narrowly filtered data.
Current priorities
Contributors are focusing current efforts on:
- support of roll-ups and tree tables and other aggregation views that employ partitioned tables.
- further window-related use cases to extend
update_by(). - authentication and authorization hooks.
- research related to data persistence and cloud deployment.
Further communication
As usual, we itemized changes in the docs since the last release in a blog piece, and we welcome your engagement on Deephaven’s Slack or its GitHub Discussions.