Deephaven overview
What is Deephaven?
Deephaven is a query engine and software stack centered on "live dataframes," a groundbreaking abstraction: tables whose rows update as their source data changes. You write queries with familiar patterns from batch tools like SQL, pandas, Polars, and R. Like a query plan in SQL or a lazy query in Polars, a Deephaven query forms a directed acyclic graph (DAG). Unlike a batch system, Deephaven moves each change ("delta") through that DAG, so downstream results update incrementally without any change-tracking code from you.
Live dataframes give Deephaven two properties that set it apart: composability and range.
Composability
In Deephaven, a query's output tables stay live, so another query can use them as inputs and keep updating as they change. That query can run in the same engine or in another engine across the network. This "building block" approach lets you connect real-time microservices, unlocking countless new use cases for real-time data.
Range
Deephaven bridges real-time and batch data, letting you apply the same operations to both and combine them in one query. Its range extends across the stack:
- First-class interfaces in Python, Groovy, and Java, plus experimental SQL support.
- Calls from queries to your own functions and to popular Python libraries such as PyTorch and TensorFlow. These calls run on live tables, and their results update as the data changes, so you don't have to export a static copy first.
- Wire protocols, such as Barrage, that let clients and other engines subscribe to a query's live tables and build on them across the network.
- gRPC-based client APIs in Python, Java, C++, C#, JavaScript, R, and Go (see the Community subsystems list for each) for building and running queries on a server across the network.
- Front-end tools for real-time data:
- The JavaScript API, for building your own real-time web front ends.
- Browser-based grid and plotting components that let you scroll through and manipulate tables with billions of records.
- Deephaven Express, for real-time plotting.
- deephaven.ui, a Python library for building real-time dashboards.
- Excel and Jupyter integrations that bring Deephaven's server-side capabilities and live updates to these tools.
Because the stack covers ingestion from many data sources, computation, and display, it empowers teams to focus on business problems instead of getting bogged down in utility engineering.
What can you use Deephaven for?
Deephaven fits streaming, batch, and hybrid analytics and applications. People turn to Deephaven when their work involves:
- Integrating real-time and dynamic data into applications or analytics.
- Performing time-series and relational operations on large Iceberg or Parquet datasets, Kafka streams, or combinations of these.
- Running SQL queries on live tables, with results that update as the data changes.
- Running custom Python or Java code alongside table operations.
- Building client applications in C++ and other languages.
- Linking multiple real-time query engines for parallel processing, pipelining, or microservice-driven analytical tasks.
Deephaven's engine supports everything from simple transformations to machine learning and autonomous applications. The stack also supports large-scale data distribution, connected applications, and advanced real-time visualizations and dashboards.
Deephaven is the perfect fit for workloads in IoT, personalization, capital markets, social media, blockchain, crypto, gaming, e-commerce, industrial telemetry, power and energy, and healthcare. The examples below show what Deephaven's capabilities look like in practice.
Example use cases
- Streaming analytics: Connect to Kafka streams and run relational and time-series joins, aggregations, and calculations that call your own functions and classes. Publish the derived, updating data downstream.
- Device telemetry: Link IoT devices to Deephaven for real-time summaries, aggregations, visualizations, interactions, and alerts.
- Batch data exploration: Analyze Parquet files in S3 or other cloud storage with responsive tools on the Deephaven web UI or embedded real-time Jupyter widgets.
- Data science at scale: Use Python libraries directly on updating tables or through snapshots mapped to NumPy and pandas.
- Real-time alerts: Integrate with the JavaScript API to send alerts to mobile apps.
- Web scraping and integration: Write web scrapers in Python or Java to ingest XML, store records persistently, and perform real-time aggregations alongside historical data.
- Wall Street expertise: Wall Street professionals trust Deephaven for trading-signal research, quantitative modeling, simulations, risk management, monitoring, reporting, and automated actions on real-time and historical market data.
What are the data sources?
Deephaven aligns with modern trends in persisted, in-memory, and streaming data formats, so it works across platforms. It reads partitioned, columnar data in place from Parquet files, reads and writes Iceberg tables, and consumes event streams from Kafka and Kafka-compatible platforms such as Redpanda. It also exchanges data with other systems over Arrow Flight.
Deephaven integrates with many other data sources by bringing them into memory as ordinary Deephaven tables. These sources include pandas dataframes, NumPy arrays, CSVs, JSON, external SQL databases, change data capture (CDC) streams, and R dataframes. You can join, manipulate, persist, or consume them as needed.
For many scenarios, direct integration with upstream applications is ideal. Upstream applications can push data into Deephaven through its client APIs, which are listed under Range. Deephaven can also poll an upstream source on a schedule you choose, with a function-generated table.
You can store unstructured data as blobs in Deephaven table columns. Machine learning tools can then extract structured values from those blobs, and you can analyze the results with the same table operations as any other data.
Isn't Deephaven like...?
"Yes." Probably. There are plenty of elephants in this room.
It's natural to map new tools against familiar ones. Deephaven operates in a landscape shaped by Flink, Spark Streaming, KX, Influx, Tinybird, RisingWave, and Materialize, so it certainly shares some capabilities and concepts with such technologies. It also exchanges data with many of them through shared formats and protocols such as Kafka, Parquet, Iceberg, and Arrow Flight.
What sets Deephaven apart is its live dataframes. Together with the composability and range they provide, live dataframes let Deephaven excel in:
- Ease of use: Deephaven simplifies complex workflows for developers and data scientists.
- Real-time Python integrations: Python libraries for analytics and data science run directly on real-time data.
- Rich joins: Relational and time-series joins, such as as-of joins, update incrementally on live data. Range joins are the exception: they currently require static tables.
- Dynamic UI/UX: User interfaces and dashboards respond and update as the data changes.
What's under the hood?
Deephaven has delivered software components that are both unique and necessary for the streaming, batch, and hybrid use cases listed in What can you use Deephaven for?. Some represent full solutions to what other industry players consider open problems.
- An incremental update model, driven by the update graph, that propagates each cycle's changes through the DAG of dependent tables, so most operations process only the rows a change affects.
- A high-performance Java engine bound tightly with native CPython and NumPy through jpy, a Python-Java bridge that Deephaven helps maintain.
- An array-oriented architecture that allows for vectorized operations, amortized costs, and columnar data transfer.
- An architecture that runs user code in-process as part of engine evaluation, rather than shipping data to the client.
- Ticking-table subscriptions. A ticking table (a live dataframe) is one whose rows change as new data arrives. A subscription streams the table's changes, batched at a configurable update interval, from the server's DAG to clients, dataframe-based tools, and other engine processes over Barrage, an Arrow Flight extension.
- Subscription support in the Java, C++, C#, and JavaScript clients, and in the Python client with the
pydeephaven-tickingpackage.
What's the difference between the Community and Enterprise stacks?
Deephaven's Enterprise product is like an F1 car, with its Community products as the subsystems. The analogy offers a clear perspective on Deephaven's software development and delivery strategy. A racecar's success relies on its subsystems, such as the engine, transmission, chassis, suspension, aerodynamics, and tires. Each one plays a critical and often differentiating role.
Deephaven develops and releases these "subsystems" openly to the community. Examples include:
- deephaven-core: The engine driving real-time data operations.
- jpy: The Python-Java bridge that binds the engine to CPython.
- Barrage: The Arrow Flight extension that streams ticking data to clients.
- Polyglot client APIs: gRPC-based client libraries in Python, Java, C++, JavaScript, R, Go, and C#.
- Web UI: A real-time grid browser application framework.
- deephaven.ui: A Python library for building real-time dashboards, with programmatic layouts and live parameterized queries.
These subsystems are sufficient for smaller projects that don't require advanced clustering, large-scale fanout of live data to many consumers, sophisticated access controls, or high-grade reliability. Community users and their organizations can embed these tools into their workflows and infrastructure to effectively meet their needs.
People want gear that "just works." Community users benefit from Enterprise reliability work. Deephaven fixes the edge cases its Enterprise customers find in production, and when an edge case is in a Community component, the fix ships in Community too.
For larger, mission-critical workloads, the metaphor evolves. Teams typically require a complete car, not just individual components. These teams have three primary options:
- Build their own car using Deephaven's subsystems.
- Upgrade their existing car by integrating selected Deephaven subsystems.
- Leverage the Deephaven Enterprise product, a customizable, high-performance, fully packaged solution delivering the capabilities of an elite F1 car.
This dual-stack approach gives flexibility and accessibility for a wide range of use cases, from community projects to enterprise-scale solutions.
For the technical differences between Community and Enterprise, see the Deephaven Enterprise documentation.
Next steps
- Install Deephaven and run your first queries with the Quickstart.
- Learn the core concepts step by step in the Crash Course.
- Bring in data from Kafka, Parquet, or Iceberg.
- See how changes flow through queries in the incremental update model.
- Follow new features and integrations in the release notes and the deephaven-core GitHub repository.