Deephaven Release Notes: Version 1.20211129

See also the accompanying Version Log for this release. The Version Log includes links to change-specific release notes which supplement the release notes on this page.

If you are upgrading from a previous Deephaven release, please refer to the additional Upgrade documentation.

Web Features

Open-Source Web UI

Deephaven's Web UI is now almost entirely open-source. You can find the packages at github.com/deephaven/web-client-ui. Additionally, the Web UI is modularized into consumable packages, which makes it easier to extend and add plugins. The Deephaven grid exists as a standalone component that can be used by others.

New Navigation

You'll notice that the New+ screen has been restyled with favicons and a Shortcut grid. The New Code Studio and New Query Monitor options can now be found under Advanced, next to New Dashboard.

img

Superuser Mode

In the updated navigation above, you can see that we've added Superuser Mode button at the top of the New+ screen. When Superuser Mode is enabled, the dashboards list includes all dashboards, even the ones that belong to other users and are not shared with the current user. When this mode is disabled, the user will only see the dashboards that are owned by them or shared with them.

Filter Set Manager

Filter Sets capture the current state of all filters applied across the current dashboard (including plots, tables, etc.) into one "set" that can then be reapplied at a later time. From the Controls menu, choose Filter Sets. Capture the current state as a new filter set and click Save. This opens a drop-down list of all available sets:

img

File Explorer improvements

It is now possible to import Deephaven notebooks from the File Explorer menu:

img

Select a file from your computer, or simply drag-and-drop files into the window.

You can export Deephaven notebooks from the Notebook Toolbar:

img

See: Notebook

Layout Hints

Deephaven's Layout Hints query methods are now available for use on the web. These include:

  • .atFront and .atEnd - These methods lock one or more columns to the beginning or end of the table. Column names are used as the argument(s).
  • .freeze - This method designates columns in Deephaven tables to be "frozen" in place on the left side of the table. The columns in the argument will remain in view even when horizontally scrolling through the table. Similar to the "freeze panes" option in spreadsheet software, this feature enables users to keep key columns in view as they read table data.

See: Layout

Deephaven Query Language Improvements

This release introduces some notable changes to the Deephaven Query Language:

  • db.timeTable is now simply timeTable.
  • Time convenience utilities: we've added support to DateTime and TimeUtils for standard Java types (Instant, ZonedDateTime, LocalDate, and LocalTime). Users can now use the TableTimeConversions class to efficiently convert between the various time types.
  • The new AutoTuning Clock Release Filter (AutotuningIncrementalReleaseFilter) can be used to benchmark how many rows of data a query can process. In its simplest form, we can measure how many rows a .lastBy statement can process.

Zip Joins

Zip Join allows users to “zip” the columns of one table onto another table, which results in tables stitched together side-by-side. This feature supports static tables and Append-Only ticking tables.

New MultiJoin Class

The multiJoin operation collects the set of distinct keys from the input tables, and then joins one row from each of the input tables onto the result. The operation can be thought of as a merge of the key columns, followed by a selectDistinct, and then a series of iterative naturalJoin operations. The multiJoin operation is more efficient than an iterative sequence of naturalJoins, because the Deephaven engine must only keep a single hash table for all of the results as opposed to one for the selectDistinct and another for each table.

Deephaven Classic / Legacy Features

MultiViewWidget

img

The MultiViewWidget is a new component that enables users to build powerful, flexible, but well-defined views of their data. Each widget instance is intended to be a self contained view of specific data, with shared filters and the ability to link between views.

Within a MultiViewWidget instance, the user may define a number of "views" that can include ticking data and column-level formatting. Each view includes a summary-row, which aggregates each column in the table. There are five primary types:

  • Leaf View: A basic flat table view that is generally the target of an aggregated-view link, used to view underlying data of a specific aggregation.
  • Aggregation View: Defines a table with a number of aggregations applied.
  • Pivot View: Similar to Deephaven's pivot tables, these are used to summarize complex underlying data by applying aggregations on a column using groupings to define both the resultant row and column values. Unlike other Deephaven tables, formatting applied to a pivot view is not defined by column, but by region within the table; columns are either "Grouping Column(s)" or "Pivoted Columns", where all "Pivoted Columns" are the result of an aggregation of a single column in the underlying table.
  • Plot View: Used to display visual charts of underlying data. This is intended as a link-target of other views.
  • Roll-up View: Defines a hierarchical table of rolled-up aggregations.

Parameterized queries

Parameterized Queries are a concept that bridges the gap between the traditional Persistent Query, and a query that can be executed “on demand” to produce tailored results without wasting valuable computational time and resources on data that will never be used. In many cases within Persistent Queries, the vast majority of the result rows will never be used or viewed by a user. The Persistent Query ends up doing much more work than is strictly necessary to fill the business needs. Parameterized queries allow you to set parameters that must be met before computing values.

Import/Export PQs from the UI

It is now possible to export and import Deephaven Persistent Queries from the Query Config panel:

img

Note that the Export button is available only when selecting a specific query or queries. A file system dialog will open, allowing you to save the script(s) as an .xml file.

Array Support in Copy Selection

It is already possible to copy selected cell values from Deephaven tables to the clipboard, from the table data context menu. This feature now includes two additional options to copy array values (when array values are present in the cell):

img

  • Copy w/ Array Previews only copies the first few values with an ellipsis to indicate there are more: [1, 2, 3, 4, 5, …]
  • Copy w/ Array Expansion copies all the array values: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].

See: Table Data Context Menu

Historical sort for large tables

Historical tables with a single sort column can now be sorted even if they are over a billion rows. This feature is limited by available worker memory but not the size of an integer.

Programmatic Sorting for Roll-up tables

Users now have the ability to add a .sort() operation to their roll-up queries. Previously, we only allowed .formatColumns(). Note that Deephaven is not creating a new table that is sorted, but is hinting to the UI that it should sort the table on fetch.

SysAdmin / Under-the-Hood

Mixed Java Versions

Deephaven's Launcher now supports manually setting the Java path per instance, by selecting the instance in the combo box and right-clicking. The menu allows you to set or clear the Java path to be used when connecting to that instance.

Additionally, the Launcher now reads the property server_java_version = xx and requires that Deephaven be started with a matching version. If the version used to run the launcher does not match, the launcher will present dialogs allowing the user to select a new JDK.

Note that this functionality requires Deephaven Launcher v8.14.

See: Launcher configuration

Envoy Upgrades

Previously, Envoy could be used only for the Web API. Now, you can use both Web and Swing interfaces.

Worker Heap Allocations

In this release, we provide an option to set Xms = Xmx when launching workers. When set, this will ensure that workers reserve their specified heap during startup, meaning that they should not have issues allocating additional memory from the operating system after startup, helping to eliminate memory contention between workers on a heavily-utilized server.

Set Default Time Zone

Users can now configure a default time zone for use with Deephaven.

This is dictated by two properties:

  • Webapi.user.defaults.timeZone=[TZ]; e.g., America/New_York
  • Webapi.user.defaults.timeZone.editable=true

When set to false, users cannot alter the time zone from the UI.

If these properties are unspecified, the time zone will default to the user’s time zone on first login.

DIS Cleanup

A new command line tool, dhctl, allows users to delete Intraday partitions via the Data Import Server.

dhctl is designed to resemble dhconfig, with built-in guidance:

dhctl [intraday] [truncate|delete|help] [arguments]

Deleting intraday data happens as follows:

  • The first step is to truncate a location. This removes the data and marks the partition as permanently truncated. Any tailers for that partition will be disconnected, and any future attempt to tail data for that partition will be rejected.
  • Next, you may prepare to accept new data with the delete command. This is permitted only when all the target partitions have been truncated.

If you want to append new data to the same location, you will need to: 1) truncate the partition, 2) remove any bin files the tailer would be picking up for that partition, and 3) delete the partition. At that point, the system is prepared to accept new data for the partition.

Indexing and Grouping Improvements

Deephaven now stores grouping information for historical tables in adjacent tables, instead of metadata within column files. This has a few advantages:

  • Less disk storage is required.
  • Grouping can take advantage of deferred and partial reads.
  • Deephaven can now support complex, multi-column, and non-contiguous groupings.

Other Changes

  • Support for the Python-based import/merge scripts is being removed, and the scripts will no longer be supplied with the Deephaven installation. This includes the following scripts:
    • IllumonThreadedBatchQueue.py
    • Db_ingest_daily.py
    • Db_ingest_utils.py
    • Db_ingester_v2.py
    • db_merge_import_base.sh
  • Deephaven no longer supports 3D plotting.
  • Deephaven no longer supports rollbacks (reverting to previous versions).

Groovy Upgrade

The version of Groovy used within Deephaven has been upgraded to 3.0.9. The following behavioral difference has been noticed.

Casting Precedence

In some cases Groovy 3.0.9 is casting in a more Java-like way than Groovy 2. The following example illustrates the difference:

result = (int)(1 + 1) / 2
println result.getClass()

Groovy 2 applied the cast to the whole expression, resulting in an Integer result. Groovy 3 applies the cast only to the addition, so the result is Groovy’s default division result of BigDecimal.