Unleashing the power of Multiple Update Graphs in Deephaven

As data science and software development continue to evolve, the necessity for tools that can handle complex, real-time operations becomes ever more important. In this context, we are thrilled to announce a significant enhancement to Deephaven - the ability to utilize multiple update graphs simultaneously.

For the uninitiated, Deephaven is an open-source platform for real-time computation on large amounts of data. Update graphs form a critical part of Deephaven's architecture, managing tables and operations on them in real-time.

A single update graph can handle a wide variety of use cases. However, it does impose limitations for problems where complex solutions are necessary - for instance, in use cases where operations must be completely isolated from one another, or in cases where updates must be applied at different rates.

Now, Deephaven users can harness the power of multiple update graphs simultaneously. This enables a wider array of operations and more flexibility in how data is manipulated and viewed.

This new feature significantly enhances the flexibility and productivity of Deephaven. By allowing multiple update graphs, users can now run separate operations concurrently and in an isolated manner. This can be beneficial for handling larger datasets and complex operations, improving efficiency by reducing interdependencies and potential bottlenecks. Moreover, it paves the way for improved modularity in the application architecture, as different parts of the system can operate independently under different update graphs. This innovation sets the stage for more complex, yet maintainable, real-time data processing in Deephaven.

This feature not only positions Deephaven as a versatile tool for modern data science tasks but also underscores our commitment to enhancing user experience and data processing capabilities.

Example

Here's a simple example of creating a table under a different update graph:

Python

from deephaven import time_table
outer = time_table('PT1s')

import jpy
_JPUG = jpy.get_type('io.deephaven.engine.updategraph.impl.PeriodicUpdateGraph')
update_graph = _JPUG.newBuilder("TestUG").existingOrBuild()

from deephaven import execution_context
_JEC = jpy.get_type('io.deephaven.engine.context.ExecutionContext')
ug_ctx = execution_context.ExecutionContext(j_exec_ctx=_JEC.newBuilder()
    .emptyQueryScope()
    .newQueryLibrary()
    .captureQueryCompiler()
    .setUpdateGraph(update_graph)
    .build())

with ug_ctx:
    inner = time_table("PT1s")

Groovy

import io.deephaven.engine.util.TableTools
outer = TableTools.timeTable("PT1s")

import io.deephaven.engine.updategraph.impl.PeriodicUpdateGraph
update_graph = PeriodicUpdateGraph.newBuilder("TestUG").existingOrBuild()

import io.deephaven.engine.context.ExecutionContext
execution_context = ExecutionContext.newBuilder()
    .emptyQueryScope()
    .newQueryLibrary()
    .captureQueryCompiler()
    .setUpdateGraph(update_graph)
    .build()

import io.deephaven.util.SafeCloseable
try (SafeCloseable ignored = execution_context.open()) {
    inner = TableTools.timeTable("PT1s")
}

Here, outer and inner are tables belonging to different update graphs. The outer table uses the default update graph, while inner is explicitly associated with the "TestUG" update graph through a custom ExecutionContext.

The ExecutionContext class is an essential part of this feature. It's an object that encapsulates all the context needed for a query's execution - including the update graph. With this class, we have a flexible and powerful way to manage and manipulate the update graph used for specific operations.

note

Currently, only tables that belong to the same update graph can be used as source tables for new operations. Development is underway for tooling that will provide the ability to replicate a table from one update graph to another.

This new capability to use multiple update graphs unlocks new avenues of exploration and efficiency in handling complex data operations in Deephaven. We are excited to see how our community will leverage this feature, and as always, we welcome your feedback and contributions!

Reach out to us on Deephaven’s Slack or GitHub Discussions.