Deephaven Release Notes: Version 1.20221001

Note

See also the accompanying Version Log for this release. The Version Log includes links to change-specific release notes which supplement the release notes on this page.

If you are upgrading from a previous Deephaven release, please refer to the additional Upgrade documentation and specifically instructions for configuring the dispatcher to use etcd and updating Python venvs for new Python versions.

Web Features

Apply conditional formatting from the UI

The Conditional Formatting feature in the Table Options menu allows you to define conditional formatting of cells or rows by creating new rules.

In the example below, all cells in the Exchange column containing the value "Arca" will be colored in the "Positive" style:

img

Support for parameterized queries

Deephaven on the web now supports parameterized queries. Please refer to the user guide for more detail.

Improved performance of new tab screen

The Shortcuts panel provides easy access to your existing Code Studios and Dashboards, which keeps your list of workspaces streamlined and avoids duplicates.

Optional ###,### Excel style number truncation

By default, values in number columns are truncated with ellipsis (e.g., 123...). Users can now choose to truncate numbers with #.

A new checkbox is available in User Settings, under Integer formatting:

img

For example, the following column values use the ellipsis:

img

After the selecting the Truncate numbers with # checkbox, they become:

img

Customizable keyboard shortcuts in Settings

Expanding the Keyboard Shortcuts option displays the current defaults and allows you to customize keyboard shortcuts for use within the Deephaven IDE. Use caution when assigning shortcuts to avoid conflicts with existing shortcuts within your operating system or browser.

img

Visually distinguish between a null value, blank string, and char

In String columns, a null and an empty string previously looked identical, leading to potential mistakes. String and Char columns will now display a gray italic “null” for null values, and String columns will display a gray italic “empty” for empty strings. In String and Char columns, only characters with no visible output (such as space) will display as a blank cell. Number columns will continue to display as blank for null.

img

Column freezing via context menu

The Freeze Columns feature is now avaiable in the UI, from the Column Header context menu:

img

This allows you to freeze the selected in place at the left of the table.

Display cell overflow

When the contents of a cell are too big to fit within its boundaries, a button will appear to its right that opens a hover panel with all the data.

Login credentials saved

A user’s session will be saved until they log out. If they refresh the page, close/re-open the tab, etc. within a sufficiently short enough period (10 minutes or less), they will remain logged in.

Deephaven Classic / Legacy Features

A "dry run" mode is available when importing persistent queries, which can be used to identify some errors before performing an actual import. For more information see the system administration guide.

SysAdmin / Under-the-Hood

Python 3.9

This release discontinues support for Python 3.6, which is no longer receiving security updates, with Python 3.9. The new default Python version is now 3.8. Any Python 3.6 VEnvs in /db/VEnvs will no longer be supported. You should instead create new Python 3.8 VEnvs before upgrading to the latest version, or create new Python 3.9 VEnvs after upgrading.

Refer to the official documentation for Python Installation to learn how to add new VEnvs or install newer versions of Python.See our Installing Python guide.

Remove deephaven-jpy and use official jpy from PyPi

In this release, Deephaven Enterprise will no longer build or ship deephaven-jpy wheels. We now use the official jpy wheels, version 0.11.0. Any VEnv managed using deephaven auto-provisioning will automatically remove deephaven-jpy and install jpy. If you have any custom VEnvs or client Python environments, you will need to pip uninstall deephaven-jpy before you run pip install deephaven, or you will have conflicting jpy artifacts in your environment.

Remote Query Dispatcher etcd Use

The RemoteQueryDispatcher and workers now uses a combination of etcd and a REST endpoint for worker registration. The db_query_server and db_merge_server processes (RemoteQueryDispatcher) must have read-write access to etcd with the prefix /dispatcher, workers (RemoteQueryProcessor)must have read-only access to the /dispatcher prefix. The dispatcher also always listens to the port defined in the RemoteQueryDispatcher.webserver.port property even if RemoteQueryDispatcher.webserver.enabled is set to false. The RemoteQueryDispatcher.webserver.enabled property only controls whether information about the dispatcher usage and queue is made available. Regardless of the enabled property value, the dispatcher creates a /status end point for workers to complete their registration and send heartbeats to the dispatcher. This mechanism replaces the Java serialization based protocol that was used in earlier versions of Deephaven.

For new installations, the etcd connection information and users are created as part of the Deephaven installation. To upgrade an existing installation, they must be created manually. On the infrastructure node, the following scripts will create the keys and etcd users:

# Create keys and endpoint information for connecting to etcd
sudo -u irisadmin /usr/illumon/latest/install/migrate/create_dispatcher_etcd_clients.sh
# Create users and grant privileges within etcd
sudo /usr/illumon/latest/install/migrate/create_dispatcher_etcd_users.sh

The etcd client information is stored in /etc/sysconfig/deephaven/etcd/client/dispatcher and /etc/sysconfig/deephaven/etcd/client/dispatcher-ro these directories must be copied to all other nodes in the cluster.

sudo -u irisadmin tar czf /tmp/dispatcher-etcd.tar.gz /etc/sysconfig/illumon.d/etcd/client/dispatcher /etc/sysconfig/illumon.d/etcd/client/dispatcher-ro/

The resulting archive should be copied to each other node in the cluster, and then unpacked:

# Unpack the archive
sudo tar xf /tmp/dispatcher-etcd.tar.gz -C /
# Ensure that the keys are owned by the dbquery user and dbmergegrp
sudo chown -R irisadmin:dbmergegrp  /etc/sysconfig/deephaven/etcd/client/dispatcher
sudo chown -R irisadmin:dbquerygrp  /etc/sysconfig/deephaven/etcd/client/dispatcher-ro

Drop singleUpdateQueue from LTM

We will no longer refresh a single table ahead of all other tables. This feature was previously used by replicated tables (the client side of a preemptive updates table) to accelerate processing from network updates. As of the Bard release, the LiveTableMonitor no longer refreshes one table at a time but rather refreshes all source tables at the start of the cycle (since the Summit release, this was enabled by the LiveTableMonitor.refreshAllTables property, which defaulted to true in Powell). When a refresh is requested, the LiveTableMonitor simply starts its cycle early, without the normal delay between cycles.

Additionally, ReplicatedTable.requestLiveTableMonitorRefresh has been set to false by default for query workers. This means that query workers will no longer accelerate the start of their LTM cycle after receiving an update from a PreemptiveTable.

Remove LTDS from default routing files

The Local Table Data Service (LTDS) can be used to access static intraday data. This service is typically used in conjunction with a Data Import Server (DIS), where the data routing configuration requests online data for "today" from the DIS and for "yesterday" from the LTDS.

In most environments, this added complexity is of limited benefit, and can cause problems at the midnight transition. As of release 20211129, the default data routing configuration does not use the LTDS. Existing installations will continue to use the LTDS, but we recommend that the changes be manually applied.

The old configuration will look something like this:

    db_tdcp:
      ...
      sources:
        - name: db_dis
          filters: {whereTableKey: "NamespaceSet = `System` && Online", whereLocationKey: "ColumnPartition >= consistentDateNy()"}
        - name: db_ltds
          filters: {whereTableKey: "NamespaceSet = `System` && Online", whereLocationKey: "ColumnPartition < consistentDateNy()"}

In that snippet, the filters specify that data where the ColumnPartition (assumed to be a date) is the current date should come from the DIS, and if it is in the past it should come from the LTDS.

To remove LTDS from data routing, remove the db_ltds section, and remove the filter on the ColumnPartition. For example:

    db_tdcp:
      ...
      sources:
        - name: db_dis
          filters: {whereTableKey: "NamespaceSet = `System` && Online"}

Edit the data routing configuration using dhconfig. See dhconfig routing for help.

Present Grouping tables directly to the engine

The Deephaven engine now takes full advantage of the new Table based grouping format introduced in previous versions. The engine no longer needs to waste time and memory converting the table to a HashMap and can take advantage of its own advanced features when using grouping data.

Run Workers as Specific System Users

A new table has been added to the ACL database, which is optionally used to identify system-user mappings for worker processes. When updating from a previous version, a system administrator must update the table definitions by connecting to the MySQL database with R/W privileges and running the command:

source /usr/illumon/latest/etc/sql/init-dbacl-and-tables.sql

To enable this feature, a system property must be set, and several commands must be enabled in sudoers. See Per-user workers.

Holiday calendars deprecated

Only the USNYSE is updated by Deephaven. Any other calendars must be maintained by by your own enterprise. If you choose to maintain a calendar that we have removed, you can reference the path to the calendar via Calendar.resourcePath property. See Installing custom calendars.

Pre-generation for binary store listeners discontinued

The generate_loggers_listeners script is now generate_loggers. Listeners are no longer included, and listener/schema changes no longer require the DIS to be restarted. Note: this does not obviate the need to log to a new partition when the schema changes. Each intraday partition may have only a single schema.

MySQL dependencies changed

This is a breaking change. We have updated to the latest mysql-connector-java, v8.0.28.

Since MySQL jars are not distributed with Deephaven, customers will need to download it from upstream from https://dev.mysql.com/downloads/connector/j/8.0.html. A free account is required to download the jars.

After placing the downloaded mysql-connector-java-8.0.28.jar into the /tmp/ directory, perform these steps on the target machines:


sudo rm /var/lib/illumon/mysql-connector-java-5.1.44.jar  # Confirm no other version exists

sudo rm /etc/sysconfig/illumon.d/java_lib/mysql-connector-java-5.1.44.jar  # Confirm no other version exists

sudo cp /tmp/mysql-connector-java-8.0.28.jar /etc/sysconfig/illumon.d/java_lib/

sudo chown irisadmin:irisadmin /etc/sysconfig/illumon.d/java_lib/mysql-connector-java-8.0.28.jar

sudo -u irisadmin monit restart all

Remove Dispatcher's Interactive Worker Logic

Decommissioned properties:

  • RemoteQueryDispatcher.failNonInteractiveWorkerWhenResourceError

  • RemoteQueryDispatcher.pagingRateMBPS

  • RemoteQueryDispatcher.maxConcurrentQueries