Detailed Version Log Deephaven 1.20221001

Detailed Version Log: Deephaven v1.20221001

Detailed Release Candidate Version Log: Deephaven v1.20210805beta

Python 3.8 is the oldest supported Python version

Even though Python 3.8 has already reached EOL, on some versions of Deephaven, this is the newest built + tested version of Python.

As of Bard version 1.20211129.426, Python 3.8 is the only Python version built, and iris-defaults.prop changes the default from Python 3.6 to 3.8.

If you still have virtual environments setup with Python 3.6 or 3.7, you should replace them with Python 3.8 venvs. To use newer versions of Python, upgrade to a newer version of Deephaven.

For legacy systems, you can change the default back to Python 3.6 by updating your iris-environment.prop to set the various jpy.* props to the values found in iris-defaults.prop, inside the jpy.env=python36 stanza:

# Legacy python3.6 locations:
jpy.programName=/db/VEnvs/python36/bin/python3.6
jpy.pythonLib=/usr/lib64/libpython3.6m.so.1.0
jpy.jpyLib=/db/VEnvs/python36/lib/python3.6/site-packages/jpy.cpython-36m-x86_64-linux-gnu.so
jpy.jdlLib=/db/VEnvs/python36/lib/python3.6/site-packages/jdl.cpython-36m-x86_64-linux-gnu.so

The new iris-defaults.prop python props are now:

# New iris-defaults.prop python3.8 locations:
jpy.programName=/db/VEnvs/python38/bin/python3.8
jpy.pythonLib=/usr/lib/libpython3.8.so
jpy.jpyLib=/db/VEnvs/python38/lib/python3.8/site-packages/jpy.cpython-38-x86_64-linux-gnu.so
jpy.jdlLib=/db/VEnvs/python38/lib/python3.8/site-packages/jdl.cpython-38-x86_64-linux-gnu.so

Option close Tailer-DIS connections early, while continuing to monitor files

A new property is available to customize the behavior of the Tailer.

log.tailer.defaultIdlePauseTime

This property is similar to log.tailer.defaultIdleTime, but it allows the Tailer to close connections early while continuing to monitor files. When the idle time specified by log.tailer.defaultIdleTime has passed without any changes to a monitored file, the Tailer will close the corresponding connection to the DIS, and will not process any further changes to the file. The default idle time must therefore be as long as the default file rollover interval plus some buffer.

The new property enables a new feature. When the time specified by log.tailer.defaultIdlePauseTime has passed without any changes to a monitored file, the Tailer will close the corresponding connection to the DIS, but will continue to monitor the file for changes. If a change is detected, the Tailer will reopen the connection and process the changes. This will reduce or more quickly reclaim resources consumed for certain usage patterns.

Optional lenient IOJobImpl to avoid write queue overflow

New behavior is available to avoid write queue overflow errors in the TDCP process. When a write queue overflow condition is detected, the process can be configured to delay briefly - giving the queue a chance to drain.

The following properties govern the feature:

IOJobImpl.lenientWriteQueue
IOJobImpl.lenientWriteQueue.retryDelay
IOJobImpl.lenientWriteQueue.maxDelay

Set IOJobImpl.lenientWriteQueue=true to enable this behavior. By default, the writer will wait up to IOJobImpl.lenientWriteQueue.maxDelay=60_000 ms in increments of IOJobImpl.lenientWriteQueue.retryDelay=100 ms.

This should address the following fatal error in the TDCP process:

ERROR - job:1424444133/RemoteTableDataService/10.128.1.75:37440->10.128.1.75:22015 write queue overflow: r=true, w=true, p=false, s=false, u=false, h=0, rcap=69632, rbyt=0, rmax=4259840, wbyt=315407, wspc=1048832, wbuf=4097, wmax=1048576, fc=0, allowFlush=true

Option to default all user tables to Parquet

Set the configuration property db.LegacyDirectUserTableStorageFormat=Parquet to default all direct user table operations, such as db.addTable, to the Parquet storage format. The default if the property is not set is DeephavenV1.

Deephaven processes log their heap usage

The db_dis, web_api_service, log_aggregator_service, iris_controller, db_tdcp, and configuration_server processes now periodically log their heap usage.

PersistentQueryController.log.current:[2024-05-10T15:00:32.365219-0400] - INFO - Jvm Heap: 3,972,537,856 Free / 4,291,624,960 Total (4,291,624,960 Max)
PersistentQueryController.log.current:[2024-05-10T15:01:32.365404-0400] - INFO - Jvm Heap: 3,972,310,192 Free / 4,291,624,960 Total (4,291,624,960 Max)

The logging interval can be configured using the property RuntimeMemory.logIntervalMillis. The default is one minute.

Automatically Provisioned Python venv Will Only Use Binary Dependencies

All pip installs performed as part of the automatic upgrade of Python virtual environments will now pass the --only-binary=:all: flag, which will prevent pip from ever attempting to build dependencies on a customer machine.

As part of this change, we automatically upgrade pip and setuptools in all virtual environments, and have upgraded a number of dependencies which for pip refused to use prebuilt binary dependencies:

For all virtual environments:
dill==0.3.1.1 is now dill==0.3.3
wrapt==1.11.2 is now wrapt==1.13.2

For jupyter virtual environments:
backcall==0.1.0 is now backcall==0.2.0
tornado==6.0.3 is now tornado==6.1

Tailer configuration changes to isolate user actions

The tailer allocates resources for each connection to a Data Import Server for each destination (namespace, table name, internal partition, and column partition). System table characteristics are predictable and fairly consistent, and can be used to configure the tailer with appropriate memory.

User tables are controlled by system users, so their characteristics are subject to unpredictable variations. It is possible for a user to cause the tailer to consume large amounts of tailer resources, which can impact System data processing or crash the process.

This change adds more properties for configuration, and adds constraints on User table processing separate from System tables.

User table isolation

Resources for User table locations are taken from a new resource pool. The buffers are smaller by default, and the pool has a constrained size. This puts an upper limit on memory consumption when users flood the system with changed locations, which can happen with closeAndDeleteCentral or when back filling data. The resources for this pool are pre-allocated at startup. The pool size should be large enough to handle expected concurrent user table writes.

Tailer/DIS configuration options

The following properties configure the memory consumption of the Tailer and Data Import Server processes.

Revert to previous behavior

To disable the new behavior in the tailer, set the following property:

DataContent.disableUserPool = true

Logger overhead

The default Logger creates a fixed pool of buffers. Certain processes are fine with a smaller size.

The following properties can be used to override the default configuration of the standard process Logger. Every log message uses an entry from the entry pool, and at least one buffer from the buffer pool. Additional buffers are taken from the buffer pool as needed. Both pools will expand as needed, so the values below dictate the minimum memory that will be consumed.

The default value for IrisLogCreator.entryPoolCapacity has been reduced to 16384 for Tailer processes.

generate-iris-keys and generate-iris-rsa no longer overwrite output

The generate-iris-keys and generate-iris-rsa scripts use OpenSSL to generate public and private keys. If you have an existing key file, the scripts now exit with a failure and you must remove the existing file before regenerating the key.

Added Envoy properties to allow proper operation in IPv6 or very dynamic routing environments

The new properties envoy.DnsType and envoy.DnsFamily allow configuration of Envoy DNS behaviors for xds routes added by the Configuration server.

• envoy.DnsType configures the value to be set in dynamically added xds routes for type. The default if this property is not set is LOGICAL_DNS. If there is a scenario where DNS should be checked on each connection to an endpoint, this can be changed to STRICT_DNS. Refer to Envoy documentation for more details about possible settings.

• envoy.DnsFamily configures the value to be set in dynamically added xds routes for dns_lookup_family. The default if this property is not set is AUTO. In environments where IPv6 is enabled the AUTO setting may cause Envoy to resolve IPv6 addresses for Deephaven service endpoints; since these service endpoints only listen on IPv4 stacks, Envoy will return a 404 or 111 when getting "Connection refused" from the IPv6 stack. Refer to Envoy documentation for more details about possible settings.

Since Deephaven endpoint services listen only on IPv4 addresses, and Envoy, by default, prefers IPv6 addresses, it may be necessary to modify the configuration in environments where IPv6 is enabled. To do this:

1. add an entry to the iris-environment.prop properties file of envoy.DnsFamily=V4_ONLY

2. edit the envoy3.yaml (or whichever configuration file Envoy is using) and add dns_lookup_family=V4_ONLY to the xds_service section:

   static_resources:
     clusters:
       - name: xds_service
         connect_timeout: 0.25s
         type: STRICT_DNS
         dns_lookup_family: V4_ONLY
   

3. import the new configuration and restart the configuration server and the Envoy process for the changes to take effect.

Modified Bessel correction formula for weighted variance

The weighted variance computation formula has been changed to match that used in the Deephaven Community engine. We now use the standard formula for "reliability weights" instead of the previous "frequency weights" interpretation. This will affect statistics based on variance such as standard deviation.

Allow removal of "Help / Contact Support ..." via property

A new property, IrisConsole.contactSupportEnabled has been added, which may be used to remove the "Help / Contact Support ..." button from the swing front-end.

By default, this property is set to true in order to preserve current behavior. Setting this to false in properties will remove the menu-option.

Reduce default max table display size

The maximum number of rows that may be displayed in the swing front-end before the red "warning bar" is displayed is now configurable. A new default maximum has been defined as 67,108,864 (64 x 1024 x 1024). Technical limitations cause rows beyond this limit to not properly update. When necessary, the Web UI is capable of displaying much larger tables than Swing.

The previous default max may be configured with the following property:

DBTableModel.defaultMaxRows=100000000

Note that the property-defined maximum may be programmatically reduced based on technical limits.

Python Integral Widening

In the 1.20211129 release, the jpy module that Deephaven's Python integration depends on converting all Python integral results into a Java integer. This resulted in truncated results when values exceed Integer.MAX_VALUE. In 1.20221001, Deephaven is using an updated jpy Integration that returns values in the narrowest possible type; so results that previously were an integer could be returned as a byte or a short. Moreover, a formula may have different types for each row. This prevented casting the result into a primitive type, as boxed objects may not be casted to another primitive.

In 1.20221001.196, Python calls in a formula now widen Byte and Short results to an Integer. If the value returned exceeds, Integer.MAX_VALUE, then the result is a Long. Existing formulas that would not have been truncated by conversion to an int in 1.20211129, behave as they would have in that release.

As casting from an arbitrary integral type to a primitive may be required, we have introduced a utility class com.illumon.iris.db.util.NumericCast that provides objectToByte, objectToShort, objectToInt, and objectToLong methods that will convert any Byte, Short, Integer, Long, or BigInteger into the specified type. If an overflow would occur, an exception is thrown.

Numba formulas (those that are surrounded in the nb function); have the narrowing behavior as in prior versions of 1.20221001.

Changed to use DHC Fast CSV parser for readCsv

TableTools.readCsv calls now use the new DHC High-Performance CSV Parser that uses a column oriented approach to parse CSV files.

The change to DHC parser includes the following visible enhancements

1. Any column that is only populated with integer surrounded by white space will be identified as an integer column. The previous parser would identify the column as a double.

2. Only 7-bit ASCII is supported as valid delimiters. This means characters such as € (euro symbol) are not valid. In these cases the following error will be thrown, delimiter is set to '€' but is required to be 7-bit ASCII.

3. Columns populated wholly with only single characters will be identified as Character columns instead of String columns.

4. Additional date time formats are automatically converted to DBDateTime columns. Previously, these formats were imported as String columns. All other date time behavior remains unchanged.

   ---

   | Format | Displayed Value in 1.20211129 | Data Type In 1.20211129 | Displayed Value in 1.20221001 | Data Type in 1.20221001 |

   | DateTimeISO_UTC_1 | 2017-08-30 11:59:59.000Z | java.lang.String | 2017-08-30T07:59:59.000000000 NY | com.illumon.iris.db.tables.utils.DBDateTime | | DateTimeISO_UTC_2 | 2017-08-30T11:59:59.000Z | java.lang.String | 2017-08-30T07:59:59.000000000 NY | com.illumon.iris.db.tables.utils.DBDateTime | | DateTimeISO_MillisOffset_2 | 2017-08-30T11:59:59.000-04 | java.lang.String | 2017-08-30T11:59:59.000000000 NY | com.illumon.iris.db.tables.utils.DBDateTime | | DateTimeISO_MicrosOffset_2 | 2017-08-30T11:59:59.000000-04 | java.lang.String | 2017-08-30T11:59:59.000000000 NY | com.illumon.iris.db.tables.utils.DBDateTime |

To use the legacy CSV parser, set the configuration property com.illumon.iris.db.tables.utils.CsvHelpers.useLegacyCsv to true.

Remote R Groovy Sessions

The idb.init method now has an optional remote parameter. When set to TRUE, Groovy script code is not executed locally but rather on a remote Groovy session as is done in the Swing console or Web Code Studio. This eliminates a class of serialization problems that could otherwise occur with a local Groovy session serializing classes to the remote server. To enable this, you must pass the remote parameter as follows:

idb.init(devroot=devroot, workspace, propfile, keyfile=keyfile, jvmArgs=jvmLocalArgs, remote=TRUE)

Additionally, you may now call idb.close() to terminate the remote worker and release the associated server resources.

Added new DbInternal.ProcessTelemetry system table and TelemetryHelperQuery

The new ProcessTelemetry table enables monitoring UI performance within the Swing front-end. For each user-initiated action, the console logs the duration between the start of the operation and when the table is ready for use. By aggregating this telemetry, an overall picture of the system's health - as perceived by users - is available. Detailed information can then be used to investigate potential performance problems.

To write these events, the Swing console internally buffers the data and then sends it to the new TelemetryHelperQuery query. If the TelemetryHelperQuery is not running, data is buffered up to a configurable limit, at which point the oldest telemetry data is discarded.

Several new startup parameters define the Telemetry behavior for the console:

[service.name=iris_console] {
    # Identifies if telemetry metrics should be logged in the DbInternal.ProcessTelemetry table. To enable logging to
    # the remote table, set the followig property to `true`. Note that telemetry will still be logged to the local
    # client-logfile unless disabled with the `enableFor` options described below
    Telemetry.remoteTelemetry=false
    
    
    # Defines the frequency for messages to be sent to the server. Events will be batched and sent periodically. The
    # default frequence is 15s
    Telemetry.sendFrequencyMs=15_000
    
    # Defines the initial size of the buffer which stores messages to be sent in a batch, defaulting to 1,024
    Telemetry.initialSendBuffer=1_024
    
    # Defines the maximum number of messages to store and send in a single batch. New messages appended to the buffer
    # after it is full will cause "older" events to be removed. The default maximum value is 10,000
    Telemetry.maxSendBuffer=10_000
    
    
    # A number of classes will attempt to log telemetry locally and to the remote table. Individual classes may be
    # prevented from being logged by setting the following to `false`
    Telemetry.enableFor.DBTableModel=true
    Telemetry.enableFor.IrisTreeTableModel=true
    Telemetry.enableFor.DbOneClickPanel=true
}

A new method, telemetrySummary(), which accepts an optional "Date" parameter, has been added to the default Groovy session. The method will provide summary tables derived from the raw Telemetry data.

For new installations, an ACL is automatically applied to the DbInternal.ProcessTelemetry table. For an upgrade, an ACL editor must create a new "Table ACL" for the raw DbInternal.ProcessTelemetry data to be seen by unprivileged users. The ACL should be similar to the "allusers/ProcessEventLog" ACL, but for the ProcessTelemetry table:

allusers | DbInternal | ProcessTelemetry | new UsernameFilterGenerator("EffectiveUser")

Parameterized Query Lock changes

Parameterized Queries now use the shared LTM lock instead of the exclusive LTM lock. Query writers may also now instruct the query not to use any of the LTM locks with the requireComputeLock on the ParameterizedQueryBuilder.

When using no locks, the query writer must ensure that the Parameterized Query Action does not use any methods that require the LTM lock. If this parameter is set to false incorrectly, then results are undefined.

Add support for worker-scope plugin classpaths

Server processes now search in /etc/sysconfig/illumon.d/plugins/*/worker for server-only plugin jars and classpath entries in addition to searching for path items from /etc/sysconfig/illumon.d/plugins/*/global.

While global dependencies are included on both server and client classpaths, worker dependencies are only be added to server processes (any process using monit or the iris launch script, as well as any jvm started from a server python session). In particular, the client update service does not make JARs in the worker directory available to the Swing console.

Column Rename Support from UI

The Schema Editor now supports ability to rename a column between the application log file and the table schema. The UI changes include how data type is handled for new columns in the Logger/Listener Column Details section. The default data type for new columns is not set and instead inherits the Intraday Type.

Below are example schemas for LoggerListener and a Listener only schema for a table with three columns.

The table below has three columns (Date, Destination and SameName) while The logger has two columns (SameName and Source).

• Date column is not present in the log file, but rather determined by the logging process.
• SameName column is in both the log file and table schema, and does not need to be transformed.
• Source column in the logger is renamed as Destination in the table.

To rename Source column as Destination, the Listener class should include both Source and Destination columns and their attributes should be:

• Source: A value of none for dbSetter attribute. This indicates that the column is not present in the table. Additionally, the attribute intradayType should be set to the appropriate dataType.
• Destination: A value of Source for dbSetter to identify its input source. A value none for intradayType means it is not present in the log file, and cannot be used as part of a dbSetter.

Schema with only Listener class

If the table has an externally generated log file (e.g., with a C++ logger), then you only need to define a Listener block to interpret the log file.

<Table namespace="ExampleNamespace" name="RenameColumn" storageType="NestedPartitionedOnDisk" >
  <Partitions keyFormula="${autobalance_single}"/>
  <Column name="Date" dataType="String" columnType="Partitioning" />
  <Column name="Destination" dataType="int" columnType="Normal" />
  <Column name="SameName" dataType="int" columnType="Normal" />

  <Listener logFormat="1" listenerPackage="com.illumon.iris.test.gen">
    <Column name="Destination" intradayType="none" dbSetter="Source" />
    <Column name="SameName" dataType="int" />
    <Column name="Source" intradayType="int" dbSetter="none" />
  </Listener>
</Table>

Schema with a LoggerListener

If you are generating a Java logger, then you should include a LoggerListener block in your schema.

<Table name="RenameColumn1001" namespace="ExampleNamespace" defaultMergeFormat="DeephavenV1" storageType="NestedPartitionedOnDisk">
  <Partitions keyFormula="${autobalance_single}" />
  
  <Column name="Date" dataType="String" columnType="Partitioning" />
  <Column name="Destination" dataType="Integer" />
  <Column name="SameName" dataType="Integer" />
  
  <LoggerListener logFormat="1" loggerClass="RenameColumn1001Logger" loggerPackage="com.illumon.iris.test.gen" 
                  rethrowLoggerExceptionsAsIOExceptions="false" tableLogger="false" generateLogCalls="true" 
                  verifyChecksum="true" listenerClass="RenameColumn1001Listener" listenerPackage="com.illumon.iris.test.gen">
    
    <SystemInput name="Source" type="int" />
    <SystemInput name="SameName" type="int" />
    
    <Column name="Destination" intradayType="none" dataType="int" dbSetter="Source" />
    <Column name="SameName" dataType="int" />
    <Column name="Source" dataType="int" dbSetter="none" />
  </LoggerListener>
</Table>

Renaming column examples for Blob and String data types.

The above pattern can be followed for all data types except for Blob and String. The differences are detailed below.

Renaming column for a Blob data type.

To rename column of a Blob data type, users need to provide the actual data type of the data stored in Blob. The Edit Logger/Listener Column UI now provides the ability to edit the data type field when the Intraday Type field is Blob. Note the data type field can only be changed if the current displayed value is none. In addition to setting the data type for the application log file column, the Listener column's dbSetter attribute must explicitly invoke cast as shown in the example schema below.

The example below shows Destination, SameName and Source columns of data type java.util.List.

<Table name="RenameColumn101" namespace="ExampleNamespace" defaultMergeFormat="DeephavenV1" storageType="NestedPartitionedOnDisk">
  <Partitions keyFormula="${autobalance_single}" />
  
  <Column name="Date" dataType="String" columnType="Partitioning" />
  <Column name="Destination" dataType="java.util.List" />
  <Column name="SameName" dataType="java.util.List" />
  
  <LoggerListener logFormat="1" loggerClass="RenameColumn101Logger" loggerPackage="com.illumon.iris.test.gen" 
                  rethrowLoggerExceptionsAsIOExceptions="false" tableLogger="false" generateLogCalls="true" verifyChecksum="true" 
                  listenerClass="RenameColumn101Listener" listenerPackage="com.illumon.iris.test.gen">
    
    <SystemInput name="Source" type="java.util.List" />
    <SystemInput name="SameName" type="java.util.List" />
    
    <Column name="Destination" intradayType="none" dataType="java.util.List" dbSetter="(java.util.List)blobToObject(Source)" />
    <Column name="SameName" intradayType="Blob" dataType="java.util.List" autoBlobInitSize="32000" />
    <Column name="Source" intradayType="Blob" dataType="java.util.List" dbSetter="none" autoBlobInitSize="256" autoBlobMaxSize="32000" />
  </LoggerListener>
</Table>

Renaming column for a String data type.

The basic steps are similar to all previous example except the Intraday Type for a String column is EnhancedString. This is reflected in the options available to select for Intraday Type. The list of valid Intraday Type options no longer includes String.

The dbSetter value for Destination column should include a toString() on the setter value as shown below in the example.

<Table name="RenameColumn02" namespace="ExampleNamespace" defaultMergeFormat="DeephavenV1" storageType="NestedPartitionedOnDisk">
    <Partitions keyFormula="${autobalance_single}"/>

    <Column name="Date" dataType="String" columnType="Partitioning"/>
    <Column name="Destination" dataType="String"/>
    <Column name="SameName" dataType="String"/>

    <LoggerListener logFormat="1" loggerClass="RenameColumn102Logger" loggerPackage="com.illumon.iris.test.gen"
                    rethrowLoggerExceptionsAsIOExceptions="false" tableLogger="false" generateLogCalls="true"
                    verifyChecksum="true" listenerClass="RenameColumn102Listener"
                    listenerPackage="com.illumon.iris.test.gen">

        <SystemInput name="Source" type="java.lang.String"/>
        <SystemInput name="SameName" type="java.lang.String"/>

        <Column name="Destination" intradayType="none" dataType="java.lang.String" dbSetter="Source.toString()"/>
        <Column name="SameName" dataType="java.lang.String"/>
        <Column name="Source" intradayType="EnhancedString" dataType="java.lang.String" dbSetter="none"/>
    </LoggerListener>
</Table>

Fix concurrency error in Table deserialization

When multiple Tables containing the same type of SparseArrayColumnSource were concurrently deserialized, the reader (e.g., a query worker) had a static object that was incorrectly used across threads. This change corrects that error, which could result in corrupted tables or exceptions during the deserialization process.

Kafka resumeFrom should fail when previous checkpoint does not exist

When calling io.deephaven.kafka.ingest.ResumeImportFrom.resumeFrom(...), a ResumeImportFrom.NoPriorPartitionsFoundException will be thrown if there is no prior partition found. This prevents us from inadvertently setting the broker's offset to 0.

SchemaDiscovery builders requires partitioning-column be defined

Since the schema-discovery tools are intended for use with a Kafka ingester, these generated schemas should be defined as System tables. System tables cannot be written which are storageType="SplayedOnDisk", and therefore MUST have a partitioning-column defined. If a user attempts to generate a schema without first defining a .columnPartition(...), then an exception will be thrown stating that the field is required.

Added StaticGroupFilterGenerator class

A new StaticGroupFilterGenerator() class is available for managing data access in environments where multiple users access the same table. This class creates a match filter for groups to which this user belongs. By default, this filters the Group column, but an optional column name can be passed in to the generator.

• Unlike the GroupFilterGenerator, the user’s group list is determined at table access time and is not updated if the user's group membership changes. The user’s group that matches their username is excluded from the filter, which allows the filter to be memoized for two users who belong to the same groups. The GroupFilterGenerator, by contrast, is responsive to on-the-fly changes and allows a username to be the group being filtered on, but cannot memoize across users (as each user’s groups may change independently).

Apache Arrow support

Deephaven now supports reading Apache Arrow files using the com.illumon.iris.db.arrow.ArrowTools class. The new ArrowTools.readArrow method accepts a path to .arrow file and returns a Table. Many types are supported including int, long (and all others which resemble java primitives), LocalDateTime, LocalDate, Timestamp, BigDecimal, and more.

Queries should route data through the TDCP running on the same machine

The Table Data Cache Proxy is designed to cache data for local use. There is an instance on each query node, and the design intention is that queries route requests through the instance running on the same machine. An error in the Deephaven installer might have configured the system data routing so that all requests are routed to the infra node instead.

To check your data routing file and correct it if necessary, follow the instructions here.

The db_tdcp entry under tableDataServices: should have a host entry like the following, not the infra node address.

# Configuration for the TableDataCacheProxy named db_tdcp, and define
# the data published by that service.
# There is typically one such service on each worker machine.
db_tdcp:
host: *localhost

Parquet Files Properly Use Codecs

A bug was fixed where ParquetTools.writeTable() would not respect the codec settings set by TableDefinitions.
Data written before this fix would have been written with the ExternalizableCodec if the object was Externalizable, or SerializableCodec if the object was Serializable. For this reason the data will still be readable.
Data written after this fix will use the properly assigned codec.

Update Apache commons-text-1.9 to 1.10.0

The Apache commons-text package had a critical CVE reported by the owasp dependency scan, which is corrected in 1.10.0. Deephaven now packages 1.10.0 instead of 1.9, user scripts that import commons.text will use the new version. For more information on changes see the Apache Commons Text Changes.

iris_keygen.sh updates

/usr/illumon/latest/install/iris_keygen.sh has been modified:

• If the executing user is irisadmin or dbmerge, it will attempt to execute commands that require irisadmin or dbmerge rights without sudo. If the executing user is neither of these, then, as in previous versions, the tool will check that the user has rights to sudo -u as both of these users, and will then use sudo -u irisadmin or dbmerge, as appropriate, to execute other commands.

• Logging has been changed from /usr/illumon/latest/install/command to /var/log/deephaven/misc.

• A new option, --alt-cert-dir, has been added, allowing the user to specify an alternate location for tls.crt and tls.key files; by default these are expected to be in /etc/deephaven/cus-tls.

Fixed Errors in TableMap Liveness management

When TableMaps were exported, they were always managed (via liveness) regardless of if they are refreshing or not.

When a derived object is created from a LivenessObjects (e.g., a TableMap created from a Table), clients must invoke LivenessManager#manage(LivenessReferent) on the parent object if and only if the parent object is itself, live. This can be checked via DynamicNode#isDynamicAndIsRefreshing(Object).

Persistent Query Export Storage Format

Persistent Queries are exported as an XML version 1.0 file, which does not allow for certain characters that may be found in script code. These include the control characters with a numeric value below 0x20 except 0x09 (Horizontal Tab), 0x0A (Line Feed) 0x0D (Carriage Return).

With this change, when a persistent query containing any invalid characters is exported, the script code will be base64 encoded in the resulting XML file to prevent the controller from crashing by attempting to create an invalid XML document.

Persistent Query Export Storage Format

Persistent Queries are exported as an XML version 1.0 file, which does not allow for certain characters that may be found in script code. These include the control characters with a numeric value below 0x20 except 0x09 (Horizontal Tab), 0x0A (Line Feed) 0x0D (Carriage Return).

With this change, when a persistent query containing any invalid characters is exported, the script code will be base64 encoded in the resulting XML file to prevent the controller from crashing by attempting to create an invalid XML document.

withColumnDescription modified Parent Table

The withColumnDescription method would modify the parent table if the parent table already had column descriptions. If the parent table had no column descriptions it was unchanged. This change will cause the method to always return a new table when the description is change. If your query depended on modifying the parent table when setting column descriptions, those will no longer be present in the UI, and you must update your query to set the description on the proper table.

Installer improvements and fixes

There no impact from this change. It improves the robustness of some aspects of the install process.

Etcd install enhancements

If the installer process requires an etcd download and no JFrog artifactory API key is provided, the installer will download it from a public Deephaven repository hosted on Google Cloud Platform. If the download fails, the installer will stop at that point in the process making it easier to determine the cause of the failure. If an etcd rpm is present in the installer directory then that will be distributed to the cluster's target nodes in lieu of a download.

Memoize Query ACL Application

When users fetch a table from a persistent query that defines ACLs, a series of complex query operations may be applied. These operations include filtering to valid rows, applying wouldMatch and updateView to provide per-cell access control, and performing a treeTable or rollup operation when a tree or rollup is filtered.

These operations can be quite expensive and many workspaces include the same table with a different combination of filters, sorts, and other customizations; therefore performance may be improved by applying the ACL operations only once per user per exported table. These operations are now memoized, which prevents re-application of the same ACL multiple times.

Schema import now compiles listeners

There are possible schema errors that only become apparent when the listeners are compiled. This compilation can now be performed when importing a schema, as part of the validation. dhconfig schemas import now includes options to control whether to compile listeners when validating a schema.

    --compile-listeners      compile listeners when importing
    --no-compile-listeners   do not compile listeners when importing (default)

--lenient-validation changes listener compilation errors into warnings.

Disabling this validation step might be useful when importing a large number of schemas that are known to be correct.

Note: The default setting will change in a later release.

RemoteQueryDispatcher Jetty SSL Configuration

The RemoteQueryDispatcher always starts the embedded web server to serve the registration endpoint for workers (RemoteQueryProcessors). The JettyServerHelper.Parameters class reads the parameters for the Jetty server from configuration, but if the configuration did not specify enabled; the SSL and authentication properties were ignored.

As of the Jackson release the RemoteQueryDispatcher.webserver.enabled property only adds URLs for worker queue and usage information rather than enabling or disabling Jetty overall. In the default configuration, these URLs were disabled, but the RemoteQueryDispatcher.webserver.sslRequired and RemoteQueryDispatcher.webserver.sslRequired properties were set to true, but ignored by the system.

Additionally, the RemoteQueryDispatcher.tls.keystore property was not specified. This property now takes on a default value of /etc/sysconfig/illumon.d/auth-user/webServices-keystore.p12.

ACL Database SQL Initialization Script Path

Files in /etc/sysconfig/deephaven/illumon.d.${VARIANT_VERSION}/sql are no longer copied/maintained during the update process.

The files are included in the installation, and may be found in /usr/illumon/illumon-db-${VARIANT_VERSION}/etc/sql.

Limiting Concurrent Input Columns with Merge

The merge process can now limit the number of input columns processed concurrently for a maximum throughput merge configuration. Instructions are described in the documentation for Merging Data.