Releases 0.25 and 0.26 delivered some heavyweight features for Deephaven users. A new SQL integration and R client, as well as a final polish on the bidirectional stream support in the Python client, extend the engine’s real-time capabilities to more access patterns. Authentication integrations and a new time library address important needs.
Full release notes for 0.25.0 and 0.26.0 can be found on GitHub.
SQL on real-time tables
Users can now use SQL to interact with Deephaven tables – both static and dynamic ones. Further innovations will follow, but today, users can embed SQL inside a simple Python script. Under the covers, the popular Apache Calcite parser and Deephaven’s query syntax tree (its declarative query structure) collaborate. As Deephaven is not a traditional database, some SQL concepts won’t apply, but analytics akin to “SQL select” will work. Capabilities will continue to evolve, and a SQL editor in the WebIDE Code Studio experience is exciting to consider.
Type SQL...get ticking tables!
from deephaven.experimental import sql
from deephaven import time_table
fake_ticking = time_table("PT0.5S").update_view(
["I=ii%10==0?null:ii", "B=ii%11==0?null:ii%3==0"]
)
# Access the table with SQL! (And watch it update in real-time!)
t = sql.evaluate("SELECT *, \
I + 2 * I - I / 2 as math_i, \
RAND() as rand, \
CURRENT_TIMESTAMP as the_timestamp, \
LN(I) as ln_i \
FROM fake_ticking").reverse()
A new R Client
Deephaven’s processing and derived data can now be accessed from R. This new R client API supports connecting to Deephaven servers, interfacing with source and derivative tables, and ultimately porting scripts to run server-side.
The R client relies on the Deephaven C++ client, and the R Arrow library provides a slick mechanism for data transfer between the two languages.
We look forward to hearing about use cases like:
- snapshotting ticking tables on a cycle to deliver them to R to update business calculations,
- co-mingling R Data Frame data with Deephaven data server-side,
- or simply using RStudio as a front end for interrogating deephaven-core.
Currently, the R client is supported only on Ubuntu 20.04 or 22.04, so please open a GitHub issue or discussion if you’d like to vote for prioritizing a particular OS beyond those.
Data Streaming to Python Clients
Final polish has been put on Deephaven’s powerful Python client API, importantly adding the ability to subscribe to a ticking table. Your Python client can now inherit the full complexion of table changes over the wire. This facilitates development patterns where Deephaven is used as a remote engine while providing you significant capabilities in your local client.
The API is designed for consistency of use for Python users. In this case, "consistency" arrives from the intentional design of the client API’s real-time data consumption to be consistent with the table_listener module used server-side. The client makes heavy use of Arrow libraries and data structures, so development patterns should feel familiar. The Python client is currently available for Linux; other OS’s will be prioritized based on community requests.
Authentication Hooks
Deephaven now supports authentication integrations. To illuminate your path, the team has provided examples for Pre-Shared Key, Mutual TLS, PostgreSQL username/password, and OpenID Connect via Keycloak. These examples should help guide your efforts to integrate your own authentication technology.
As of 0.26.0, deephaven-core’s default configuration uses pre-shared key. Discovering and managing your key is straightforward. Please reach out via Slack for support.
A Modernized Time Library
Handling and manipulating time- and date-related data has always been fundamental to Deephaven’s value proposition. As its core technology was being invented a decade ago, the team developed time libraries to fill the void. Since that time (pun intended?), Java’s approach to date-time has changed, and in v0.25, we released a big-bang change to the library to coordinate Deephaven with modern standards.
Please note: Breaking change!
This provides many benefits, including:
- direct use of Java data-time types (Instant, LocalDate, ZoneId, Period, Duration, etc.)
- new functions
- ISO formats
- more complete parsing
- 90% unit test coverage
It goes without saying that this embrace of the industry standard allows the Deephaven community to inherit future innovations to the Java library and to treat time within Deephaven in familiar ways. You should find the library’s PyDocs and Javadocs to be strong resources.
Unfortunately, this was a breaking change. Your data is certainly safe, but as you upgrade to v0.25 or later, some of your queries may need to be edited. A stable API is a priority for the contributing team, so the decision to make this change received significant consideration and debate. We recognize the friction that breaking changes present, intend to minimize their occurrence, and will foreshadow their arrival when possible.
We look forward to interacting with you via Deephaven’s Slack or GitHub Discussions.