Deephaven Community Core Quickstart

This guide shows you how to install and launch Deephaven Community Core, then takes you on a short tour of the Deephaven IDE: importing static and streaming data, transforming and aggregating tables, plotting, and exporting results.

You can install Deephaven Community Core with Docker or with the production application, which runs Deephaven natively on your machine. If you don't have a preference, we recommend starting with Docker.

1. Install and launch Deephaven

With Docker

Docker-installed Deephaven runs in a Docker container, so you need Docker installed on your machine. Install and launch Deephaven with a one-line command:

The -v option mounts the data folder in your current directory at /data inside the container, so files that Deephaven writes to /data appear in that folder on your machine. Docker creates the folder if it doesn't exist.

For additional configuration options, see the install guide for Docker.

With the production application

Download the Deephaven server-jetty-<version>.tar file from the assets of the latest release using your browser or the command line, unpack the tar file, and start Deephaven. The commands below use a DH_VERSION environment variable. Replace LATEST_VERSION_HERE with the version number of the latest release, without the leading v.

The -Ddeephaven.console.type=groovy option starts the console in Groovy instead of the default Python.

For more details, see the production application guide.

2. The Deephaven IDE

Navigate to http://localhost:10000/ and enter your password in the token field:

You're ready to go. The Deephaven IDE is a full scripting environment. Here's a brief overview of its basic features.

1. Write and execute commands

   Use this console to write and execute Groovy and Deephaven commands.

2. Create new notebooks

   Click this button to create new notebooks where you can write scripts.

3. Edit active notebook

   Edit the currently active notebook.

4. Run entire notebook

   Click this button to execute all of the code in the active notebook, from top to bottom.

5. Run selected code

   Click this button to run only the selected code in the active notebook.

6. Save your work

   Save your work in the active notebook. Do this often!

Now that you have Deephaven installed and open, you can explore some of its key features in the rest of this guide.

3. Import static and streaming data

Deephaven works with both static and streaming data, and it can ingest data from CSV files, Parquet files, Kafka streams, and more. This section loads a static CSV file, then replays historical data to simulate a stream.

Load a CSV

Run the command below inside a Deephaven console to ingest a million-row CSV of crypto trades with readCsv. All you need is a path or URL for the data:

The table widget now in view is highly interactive:

• Click on a table and press Ctrl + F (Windows/Linux) or ⌘ + F (Mac) to open the Quick Filters bar, a filter row above the column headers.
• Click the funnel icon in a quick filter field to open the Advanced Filters panel and build more detailed filters.
• Hover over column headers to see data types.
• Right-click headers to access more options, like adding or changing sorts.
• Click the Table Options hamburger menu at right to plot from the UI, create and manage columns, and download CSVs.

Replay historical data

Ingesting real-time data is one of Deephaven's core strengths. However, streaming pipelines can be complicated to set up and are outside the scope of this guide. For a streaming data example, this guide uses Deephaven's Replayer to replay historical cryptocurrency data in real time.

The following code takes fake historical crypto trade data from a CSV file and replays it in real time based on timestamps. This is only one of multiple ways to create real-time data in just a few lines of code. Replaying historical data is a great way to test real-time algorithms before deployment into production.

4. Work with Deephaven tables

Deephaven represents both static and streaming data as tables. You can derive new tables from parent tables, and data flows efficiently from parents to their dependents. See the concept guide on the table update model if you're interested in what's under the hood.

Deephaven represents data transformations as operations on tables. This is a familiar paradigm for data scientists using pandas, Polars, R, MATLAB, and more. Deephaven's table operations have one key difference — they work the same way whether the underlying data is static or streaming. This means that code written for static data generally also works on streaming data. A few formulas are exceptions, such as those that use the special variables i, ii, and k.

There are many table operations to cover, so this guide keeps it short and covers the highlights.

Manipulate data

First, reverse the ticking (live-updating) cryptoStreaming table with reverse so that the newest data appears at the top:

Add a column with update:

Use view to pick out particular columns. select does the same but computes the columns and stores them in memory, while view computes them on demand:

Remove columns with dropColumns:

Deephaven offers many operations for filtering tables, including where, whereIn, whereNotIn, and others. See the filtering guide for the full set.

The following code uses where to filter for only Bitcoin transactions, and then uses where with the in match operator to filter for Bitcoin and Ethereum transactions:

Aggregate data

Deephaven's dedicated aggregations suite provides a number of table operations that enable efficient column-wise aggregations. These operations also support aggregations by group.

Use countBy to count the number of transactions from each exchange:

Then, get the average price for each instrument with avgBy:

Find the largest transaction per instrument with maxBy:

Each dedicated aggregation performs one aggregation at a time. You often need to perform multiple aggregations on the same data. For this, Deephaven provides the aggBy table operation and the io.deephaven.api.agg.Aggregation Java API.

First, use aggBy to compute the mean and standard deviation of the price, grouped by instrument and exchange:

Then, add a column containing the coefficient of variation for each instrument and exchange, measuring the relative risk of each:

Finally, create a minute-by-minute Open-High-Low-Close table using lowerBin, one of Deephaven's built-in time functions, along with AggFirst, AggMax, AggMin, and AggLast:

Window calculations

You may want to perform window-based calculations, compute moving or cumulative statistics, or look at pairwise differences. Deephaven's updateBy table operation is the right tool for the job.

Compute the moving average and standard deviation of each instrument's price using RollingAvg and RollingStd:

Use these statistics to find "extreme" instrument prices, where the price is more than 1.645 standard deviations above or below the rolling average for the window that ends at that row's timestamp:

There's a lot more to updateBy. See the guide on rolling aggregations for more information.

Combine tables

Combining datasets can often yield powerful insights. Deephaven offers two primary ways to combine tables — the merge and join operations.

The merge operation stacks tables on top of one another. The tables must have the same columns with the same types. The tables you merge can be static, ticking, or a mix of both. For example, combine the static September 2021 trades with the ticking February 2023 replay:

The ubiquitous join operation combines tables based on columns that they have in common. Deephaven offers many variants of this operation, such as join, naturalJoin, and exactJoin. The exact and relational joins guide covers the rest.

For example, summarize the older September 2021 trades in cryptoFromCsv, the table you loaded at the start of this guide. Then, use join to combine that summary with the February 2023 summaryPrices table from Aggregate data to compare February 2023 prices with September 2021 prices:

Many real-time data applications combine data based on timestamps. Traditional join operations often fail this task, as they require exact matches in both datasets. To remedy this, Deephaven provides time-series joins, such as aj and raj, that can join tables on timestamps with approximate matches.

The following example uses aj to find the Ethereum price at or immediately before the timestamp of each Bitcoin trade:

5. Plot data via query or the UI

Deephaven has a rich plotting API that supports updating, real-time plots. You can call it programmatically, for example with plot:

You can also create plots from the web UI:

6. Export data

You can export your data from Deephaven to popular open file formats.

The examples below write to /data, the data directory in the Deephaven Docker image. If you started Deephaven with the Docker command above, the files appear in the data folder of the directory where you ran it. To mount other directories, see Add a second volume. If you run the production application, replace /data with a directory on your machine that you can write to.

To export a table to a CSV file, use the writeCsv method with the table and the path where you want to save the file:

Similarly, use writeTable to export a Parquet file:

If a table is ticking, each exported file captures the table's state at the moment the script runs.

7. What to do next

Now that you've imported data, created tables, and manipulated static and real-time data, we suggest heading to the Crash Course in Deephaven for a more in-depth introduction to Deephaven's design and APIs.

To go further with the topics in this guide, see:

• Read CSV files
• Read Parquet files
• Connect to a Kafka stream
• Filter table data
• Plot with the built-in plotting API
• Navigate the Deephaven UI