Deephaven Community Core Quickstart for Jupyter

Deephaven Community Core and Jupyter together make a powerful workflow for real-time data science. You can start and use a Deephaven server directly from Jupyter with pip-installed Deephaven. Alternatively, you can use the Deephaven Python client from Jupyter to connect to an already-running Deephaven server. This guide covers pip-installed Deephaven. For the Python client, see Use Deephaven in Jupyter.

1. Install Jupyter

Deephaven can operate in JupyterLab or Jupyter Notebook. The choice is yours! Install either one with pip:

2. Install and launch Deephaven

The deephaven-server package enables you to use Deephaven directly from Jupyter. The deephaven-ipywidgets package renders Deephaven tables and plots in Jupyter. Install them both in the same environment as your Jupyter installation:

If you plan to plot with Deephaven Express after this guide, also run pip install deephaven-plugin-plotly-express now. Plugin packages must be installed before you start the server.

Now, start an instance of Jupyter:

The following code block starts a Deephaven server on port 10000 with a 4 GB maximum heap and anonymous authentication. Run it in your Jupyter instance:

For more advanced configuration options, see the pip installation guide.

3. Import static and streaming data

Deephaven empowers you to wrangle static and streaming data with ease. It can ingest data from CSV files, Parquet files, and Kafka streams.

Load a CSV

Run the code below in a Jupyter cell to ingest a million-row CSV of crypto trades with read_csv. All you need is a path or URL for the data:

Display the resulting table with DeephavenWidget from deephaven-ipywidgets, which renders Deephaven tables and plots right inside Jupyter:

The table widget now in view is highly interactive:

• Click on a table and press Ctrl + F (Windows/Linux) or ⌘ + F (Mac) to open the Quick Filters bar, a filter row above the column headers.
• Click the funnel icon in a Quick Filters field to open the Advanced Filters panel and build more detailed filters.
• Hover over column headers to see data types.
• Right-click headers to access more options, like adding or changing sorts.
• Click the Table Options hamburger menu at right to plot from the UI, create and manage columns, and download CSVs.

Replay historical data

Ingesting real-time data is one of Deephaven's core strengths. However, streaming pipelines such as Kafka streams can be complicated to set up and are outside the scope of this guide. For a streaming data example, this guide uses Deephaven's TableReplayer to replay historical cryptocurrency data in real time.

The following code takes fake historical crypto trade data from a CSV file and replays it in real time based on timestamps. Replaying a table is one of several ways to create real-time data in just a few lines of code. Replaying historical data is a great way to test real-time algorithms before deployment into production.

4. Work with Deephaven tables

Deephaven represents both static and streaming data as tables. You derive new tables from parent tables, and data efficiently flows from parents to their dependents. See the concept guide on the table update model if you're interested in what's under the hood.

Deephaven represents data transformations as operations on tables. This is a familiar paradigm for data scientists using pandas, Polars, R, MATLAB, and more.

Deephaven's table operations are special — they work the same way whether the underlying data is static or streaming! This means that code written for static data generally works seamlessly on live data. A few formulas are exceptions, such as those that use the special variables i, ii, and k.

There are many table operations to cover, so this guide keeps it short and gives you the highlights.

Manipulate data

First, reverse the ticking (live-updating) table with reverse so that the newest data appears at the top:

Add a column with update:

Use view to pick out particular columns. select does the same but computes the columns and stores them in memory, while view computes them on demand:

Remove columns with drop_columns:

Next, filter the table. Deephaven offers many filtering operations, including where, where_one_of, where_in, and where_not_in. See the filtering guide for the full set.

The following code uses where and where_one_of to filter for only Bitcoin transactions, and then for Bitcoin and Ethereum transactions:

Aggregate data

Deephaven's dedicated aggregations suite provides several table operations that enable efficient column-wise aggregations and support aggregations by group.

Use count_by to count the number of transactions from each exchange:

Then, get the average price for each instrument with avg_by:

Find the largest transaction per instrument with max_by. This example starts from crypto_streaming, so it recomputes the TransactionTotal column first:

While dedicated aggregations are powerful, they only perform one aggregation at a time. However, you often need to perform multiple aggregations on the same data. For this, Deephaven provides the agg_by table operation and the deephaven.agg Python module.

First, use agg_by to compute the mean and standard deviation of the price, grouped by instrument and exchange, and sort the result:

Then, add a column containing the coefficient of variation for each instrument and exchange, measuring the relative risk of each:

Finally, create a minute-by-minute Open-High-Low-Close table. First, bin each timestamp to the minute with lowerBin, one of Deephaven's built-in time functions. Then, aggregate each bin with agg_by and the first, max_, min_, and last aggregations:

Window calculations

You may want to perform window-based calculations, compute rolling or cumulative statistics, or look at pairwise differences. Deephaven's update_by table operation and the deephaven.updateby Python module are the right tools for the job.

Compute the rolling average and standard deviation of each instrument's price using rolling_avg_time and rolling_std_time:

Use these statistics to find "extreme" instrument prices, where the price is more than 1.645 standard deviations above or below the rolling average for the window that ends at that row's timestamp:

There's a lot more to update_by. See the guide on rolling aggregations for more information.

Combine tables

Combining datasets can often yield powerful insights. Deephaven offers two primary ways to combine tables: merges and joins. Deephaven has two kinds of joins. Exact and relational joins match rows on equal key values. Time-series and range joins match rows on the closest value or a range.

The merge operation stacks tables on top of one another. The tables must have the same columns with the same types. The tables you merge can be static, ticking, or a mix of both. For example, combine the static September 2021 trades with the ticking February 2023 replay:

The ubiquitous join operation combines tables based on columns they have in common. Exact and relational joins include join, natural_join, and exact_join. The exact and relational joins guide covers each of them.

For example, summarize the older September 2021 trades in crypto_from_csv, the table you loaded at the start of this guide. Then, use rename_columns and join to combine that summary with summary_prices, the summary of the replayed February 2023 trades from Aggregate data. The result shows both periods' price statistics side by side for the instruments and exchanges that appear in both datasets. A join drops rows that have no match, so PAXG/USD and YFI/USD aren't in the result:

Many real-time data applications combine data based on timestamps. Exact and relational joins can't do this, because they require exact matches in both datasets. To remedy this, Deephaven provides time-series joins, such as aj and raj, that join tables on timestamps with approximate matches.

The following example uses aj to find the Ethereum price at or immediately before the timestamp of each Bitcoin trade:

To learn more about our join methods, see the guides on exact and relational joins and time-series and range joins.

5. Plot data

Deephaven has rich plotting libraries that support updating, real-time plots. The following example uses plot_xy from Deephaven's built-in plotting API. The code plots the Bitcoin price and its 30-second rolling average, and DeephavenWidget renders the resulting figure:

For most new work, use Deephaven Express, built on top of the popular Plotly Express library, instead of the built-in plotting API. Deephaven Express needs the deephaven-plugin-plotly-express package, which you must install before you start the server, as shown in Install and launch Deephaven. To compare Deephaven's plotting options, see the plotting overview.

6. Export data to popular formats

It's easy to export your data out of Deephaven to popular open formats such as CSV and Parquet, or to convert tables to pandas DataFrames.

To export a table to a CSV file, use write_csv with the table and the path where you want to save the file.

A ticking table can change while you write it, and code in a Jupyter cell doesn't hold the lock that keeps it consistent. Take a static snapshot of a ticking table first. The following code writes the CSV to the current working directory:

Similarly, use write for Parquet:

To create a static pandas DataFrame, use the to_pandas function. It takes the snapshot for you, so you can pass the ticking table directly:

See the pandas guide for more details.

7. What to do next

Now that you've imported data, created tables, and manipulated static and real-time data, we suggest heading to A Crash Course in Deephaven to learn more about Deephaven's real-time data platform.