Detailed Version Log Deephaven 1.20211129

Detailed Version Log: Deephaven v1.20211129

Detailed Release Candidate Version Log: Deephaven # Detailed Version Log: Deephaven v1.20210201beta

Removed Jupyter Notebook integration

Server side Jupyter Notebook integration has been removed from Deephaven. The Legacy worker Jupyter Notebook is no longer supported and will not be updated. Use the Deephaven Core+ Python client from Jupyter notebooks beginning in Deephaven 1.20231218 and later.

Python 3.8 is the oldest supported Python version

Even though Python 3.8 has already reached EOL, on some versions of Deephaven, this is the newest built + tested version of Python.

As of Bard version 1.20211129.426, Python 3.8 is the only Python version built, and iris-defaults.prop changes the default from Python 3.6 to 3.8.

If you still have virtual environments setup with Python 3.6 or 3.7, you should replace them with Python 3.8 venvs. To use newer versions of Python, upgrade to a newer version of Deephaven.

For legacy systems, you can change the default back to Python 3.6 by updating your iris-environment.prop to set the various jpy.* props to the values found in iris-defaults.prop, inside the jpy.env=python36 stanza:

# Legacy python3.6 locations:
jpy.programName=/db/VEnvs/python36/bin/python3.6
jpy.pythonLib=/usr/lib64/libpython3.6m.so.1.0
jpy.jpyLib=/db/VEnvs/python36/lib/python3.6/site-packages/jpy.cpython-36m-x86_64-linux-gnu.so
jpy.jdlLib=/db/VEnvs/python36/lib/python3.6/site-packages/jdl.cpython-36m-x86_64-linux-gnu.so

The new iris-defaults.prop python props are now:

# New iris-defaults.prop python3.8 locations:
jpy.programName=/db/VEnvs/python38/bin/python3.8
jpy.pythonLib=/usr/lib/libpython3.8.so
jpy.jpyLib=/db/VEnvs/python38/lib/python3.8/site-packages/jpy.cpython-38-x86_64-linux-gnu.so
jpy.jdlLib=/db/VEnvs/python38/lib/python3.8/site-packages/jdl.cpython-38-x86_64-linux-gnu.so

Tailer configuration changes to isolate user actions

The tailer allocates resources for each connection to a Data Import Server for each destination (namespace, table name, internal partition, and column partition). System table characteristics are predictable and fairly consistent, and can be used to configure the tailer with appropriate memory.

User tables are controlled by system users, so their characteristics are subject to unpredictable variations. It is possible for a user to cause the tailer to consume large amounts of tailer resources, which can impact System data processing or crash the process.

This change adds more properties for configuration, and adds constraints on User table processing separate from System tables.

User table isolation

Resources for User table locations are taken from a new resource pool. The buffers are smaller by default, and the pool has a constrained size. This puts an upper limit on memory consumption when users flood the system with changed locations, which can happen with closeAndDeleteCentral or when back filling data. The resources for this pool are pre-allocated at startup. The pool size should be large enough to handle expected concurrent user table writes.

Tailer/DIS configuration options

The following properties configure the memory consumption of the Tailer and Data Import Server processes.

Revert to previous behavior

To disable the new behavior in the tailer, set the following property:

DataContent.disableUserPool = true

Logger overhead

The default Logger creates a fixed pool of buffers. Certain processes are fine with a smaller size.

The following properties can be used to override the default configuration of the standard process Logger. Every log message uses an entry from the entry pool, and at least one buffer from the buffer pool. Additional buffers are taken from the buffer pool as needed. Both pools will expand as needed, so the values below dictate the minimum memory that will be consumed.

The default value for IrisLogCreator.entryPoolCapacity has been reduced to 16384 for Tailer processes.

Modified Bessel correction formula for weighted variance

The weighted variance computation formula has been changed to match that used in the Deephaven Community engine. We now use the standard formula for "reliability weights" instead of the previous "frequency weights" interpretation. This will affect statistics based on variance such as standard deviation.

Allow removal of "Help / Contact Support ..." via property

A new property, IrisConsole.contactSupportEnabled has been added, which may be used to remove the "Help / Contact Support ..." button from the swing front-end.

By default, this property is set to true in order to preserve current behavior. Setting this to false in properties will remove the menu-option.

Reduce default max table display size

The maximum number of rows that may be displayed in the swing front-end before the red "warning bar" is displayed is now configurable. A new default maximum has been defined as 67,108,864 (64 x 1024 x 1024). Technical limitations cause rows beyond this limit to not properly update. When necessary, the Web UI is capable of displaying much larger tables than Swing.

The previous default max may be configured with the following property:

DBTableModel.defaultMaxRows=100000000

Note that the property-defined maximum may be programmatically reduced based on technical limits.

Add support for worker-scope plugin classpaths

Server processes now search in /etc/sysconfig/illumon.d/plugins/*/worker for server-only plugin jars and classpath entries in addition to searching for path items from /etc/sysconfig/illumon.d/plugins/*/global.

While global dependencies are included on both server and client classpaths, worker dependencies are only be added to server processes (any process using monit or the iris launch script, as well as any jvm started from a server python session). In particular, the client update service does not make JARs in the worker directory available to the Swing console.

Fix concurrency error in Table deserialization

When multiple Tables containing the same type of SparseArrayColumnSource were concurrently deserialized, the reader (e.g., a query worker) had a static object that was incorrectly used across threads. This change corrects that error, which could result in corrupted tables or exceptions during the deserialization process.

Parquet Files Properly Use Codecs

A bug was fixed where ParquetTools.writeTable() would not respect the codec settings set by TableDefinitions.
Data written before this fix would have been written with the ExternalizableCodec if the object was Externalizable, or SerializableCodec if the object was Serializable. For this reason the data will still be readable.
Data written after this fix will use the properly assigned codec.

Fixed Errors in TableMap Liveness management

When TableMaps were exported, they were always managed (via liveness) regardless of if they are refreshing or not.

When a derived object is created from a LivenessObjects (e.g., a TableMap created from a Table), clients must invoke LivenessManager#manage(LivenessReferent) on the parent object if and only if the parent object is itself, live. This can be checked via DynamicNode#isDynamicAndIsRefreshing(Object).

Persistent Query Export Storage Format

Persistent Queries are exported as an XML version 1.0 file, which does not allow for certain characters that may be found in script code. These include the control characters with a numeric value below 0x20 except 0x09 (Horizontal Tab), 0x0A (Line Feed) 0x0D (Carriage Return).

With this change, when a persistent query containing any invalid characters is exported, the script code will be base64 encoded in the resulting XML file to prevent the controller from crashing by attempting to create an invalid XML document.

Persistent Query Export Storage Format

Persistent Queries are exported as an XML version 1.0 file, which does not allow for certain characters that may be found in script code. These include the control characters with a numeric value below 0x20 except 0x09 (Horizontal Tab), 0x0A (Line Feed) 0x0D (Carriage Return).

With this change, when a persistent query containing any invalid characters is exported, the script code will be base64 encoded in the resulting XML file to prevent the controller from crashing by attempting to create an invalid XML document.

withColumnDescription modified Parent Table

The withColumnDescription method would modify the parent table if the parent table already had column descriptions. If the parent table had no column descriptions it was unchanged. This change will cause the method to always return a new table when the description is change. If your query depended on modifying the parent table when setting column descriptions, those will no longer be present in the UI, and you must update your query to set the description on the proper table.

Installer improvements and fixes

There no impact from this change. It improves the robustness of some aspects of the install process.

Etcd install enhancements

If the installer process requires an etcd download and no JFrog artifactory API key is provided, the installer will download it from a public Deephaven repository hosted on Google Cloud Platform. If the download fails, the installer will stop at that point in the process making it easier to determine the cause of the failure. If an etcd rpm is present in the installer directory then that will be distributed to the cluster's target nodes in lieu of a download.

Memoize Query ACL Application

When users fetch a table from a persistent query that defines ACLs, a series of complex query operations may be applied. These operations include filtering to valid rows, applying wouldMatch and updateView to provide per-cell access control, and performing a treeTable or rollup operation when a tree or rollup is filtered.

These operations can be quite expensive and many workspaces include the same table with a different combination of filters, sorts, and other customizations; therefore performance may be improved by applying the ACL operations only once per user per exported table. These operations are now memoized, which prevents re-application of the same ACL multiple times.

Schema import now compiles listeners

There are possible schema errors that only become apparent when the listeners are compiled. This compilation can now be performed when importing a schema, as part of the validation. dhconfig schemas import now includes options to control whether to compile listeners when validating a schema.

    --compile-listeners      compile listeners when importing
    --no-compile-listeners   do not compile listeners when importing (default)

--lenient-validation changes listener compilation errors into warnings.

Disabling this validation step might be useful when importing a large number of schemas that are known to be correct.

Note: The default setting will change in a later release.

Limiting Concurrent Input Columns with Merge

The merge process can now limit the number of input columns processed concurrently for a maximum throughput merge configuration. Instructions are described in the documentation for Merging Data.