Detailed Version Log Deephaven 1.20231218

Certified versions

Detailed Version Log: Deephaven v1.20231218

Detailed Release Candidate Version Log: Deephaven v1.20230512beta

New API to format the log-format suffix of internal partitions

A new builder method IntradayLoggerBuilder#setSuffixInternalPartitionWithLogFormat(String) has been added that lets caller provide a single-argument String.format pattern. The formatted log-format value is appended to the internal-partition name. This overloads the existing IntradayLoggerBuilder#setSuffixInternalPartitionWithLogFormat(boolean) which, when true, appends the suffix using the default %d pattern.

Example:

For internal partition "ABC" and log-format version 4:

• For setSuffixInternalPartitionWithLogFormat(true), the actual partition used would be ABC-4
• For setSuffixInternalPartitionWithLogFormat("%02d"), the actual partition used would be ABC-04

Removed Jupyter Notebook integration

Server side Jupyter Notebook integration has been removed from Deephaven. The Legacy worker Jupyter Notebook is no longer supported and will not be updated. Use the Deephaven Core+ Python client from Jupyter notebooks beginning in Deephaven 1.20231218 and later.

Optional limit on appendCentral table size (client side)

Database.appendCentral(...) sends the given table to the Log Aggregator Service as an atomic update. A large enough table can cause the LAS to run out of memory.

You can now set a maximum table size (number of rows) that will be accepted by the appendCentral call, by setting the optional property LogAggregatorService.transactionLimit.rows. This check only looks at the number of rows and does not take the number of columns into account. Zero or unset means no limit is enforced.

To make updates larger than the configured limit, either break the table into smaller pieces, or use the RemoteTableAppender directly to make a non-atomic update:

rta = new com.illumon.iris.db.util.logging.RemoteTableAppender(log, table.getDefinition().getWritable(), namespace, tableName, columnPartitionValue)
rta.append(table)
rta.flush()
rta.close()

See also Optional server-side limit on appendCentral table size for related server-side changes.

Optional limit on appendCentral table size (server side)

Database.appendCentral(...) and RemoteTableAppender.appendAtomic(...) calls send the given table to the Log Aggregator Service as an atomic update. A large enough table can cause the LAS to run out of memory.

You can now set a maximum table size (number of rows or number of bytes) that will be accepted by the Log Aggregator, by setting the optional properties LogAggregatorService.transactionLimit.rows or LogAggregatorService.transactionLimit.bytes. Zero or unset means no limit is enforced. When the Log Aggregator accumulates more rows or more bytes in a transaction than the configured limit, it will abort the transaction and release the accumulated memory. The client will get an error.

To make updates larger than the configured limit, either break the table into smaller pieces, or use the RemoteTableAppender directly to make a non-atomic update:

rta = new com.illumon.iris.db.util.logging.RemoteTableAppender(log, table.getDefinition().getWritable(), namespace, tableName, columnPartitionValue)
rta.append(table)
rta.flush()
rta.close()

See also Optional client-side limit on appendCentral table size for related client-side changes.

Python 3.8 is the oldest supported Python version

Even though Python 3.8 has already reached EOL, on some versions of Deephaven, this is the newest built + tested version of Python.

As of Bard version 1.20211129.426, Python 3.8 is the only Python version built, and iris-defaults.prop changes the default from Python 3.6 to 3.8.

If you still have virtual environments setup with Python 3.6 or 3.7, you should replace them with Python 3.8 venvs. To use newer versions of Python, upgrade to a newer version of Deephaven.

For legacy systems, you can change the default back to Python 3.6 by updating your iris-environment.prop to set the various jpy.* props to the values found in iris-defaults.prop, inside the jpy.env=python36 stanza:

# Legacy python3.6 locations:
jpy.programName=/db/VEnvs/python36/bin/python3.6
jpy.pythonLib=/usr/lib64/libpython3.6m.so.1.0
jpy.jpyLib=/db/VEnvs/python36/lib/python3.6/site-packages/jpy.cpython-36m-x86_64-linux-gnu.so
jpy.jdlLib=/db/VEnvs/python36/lib/python3.6/site-packages/jdl.cpython-36m-x86_64-linux-gnu.so

The new iris-defaults.prop python props are now:

# New iris-defaults.prop python3.8 locations:
jpy.programName=/db/VEnvs/python38/bin/python3.8
jpy.pythonLib=/usr/lib/libpython3.8.so
jpy.jpyLib=/db/VEnvs/python38/lib/python3.8/site-packages/jpy.cpython-38-x86_64-linux-gnu.so
jpy.jdlLib=/db/VEnvs/python38/lib/python3.8/site-packages/jdl.cpython-38-x86_64-linux-gnu.so

Changes to Barrage subscriptions in Core+ Python workers

The methods subscribe and snapshotTable inside deephaven_enterprise.remote_table have been changed to return a Python deephaven.table.Table object instead of a Java io.deephaven.engine.table.Table object. This allows users to use the Python methods update_view, rename_columns, etc. as expected without wrapping the returned table.

Existing Python code that manually wrapped the table or directly called the wrapped Java methods must be updated.

Example of previous behavior:

from deephaven_enterprise import remote_table as rt
table = rt.in_local_cluster(query_name="SubscribePQ", table_name="my_table").snapshot()
table = table.updateView("NewCol = random()")

Example of new behavior:

from deephaven_enterprise import remote_table as rt
table = rt.in_local_cluster(query_name="SubscribePQ", table_name="my_table").snapshot()
table = table.update_view("NewCol = random()")

Vermilion+ Core+ updated to 0.35.2

Vermilion+ 1.20231218.440 includes version 0.35.2 of the Deephaven Core engine. This is the same version that ships with Grizzly in 1.20240517.189, enabling customers to have one Core engine version of overlap between major Deephaven Enterprise releases. Although the Core engine functionality is the same in 0.35.2, the Grizzly Core+ worker has several enhancements that are not available in the Vermilion+ Core+ worker. This change also updates grpc to 1.61.0

For details on the Core changes, see the following release notes:

• Release notes for Deephaven Core version 0.35
• Release notes for Deephaven Core version 0.34

Changes to vector support for Core+ user tables

Both the Legacy and Core engines have special database types to represent arrays of values. The Legacy engine uses the DbArray class, while the Core system uses the Vector class. While these implementations represent identical data, they pose challenges for interoperability between workers running different engines.

When a user table is written, the schema is inferred from the source table. Previously, Vectors would be recorded verbatim in the schema. This change explicitly encodes Vector types as their base java array types as follows.

This makes it possible for the Legacy engine to read User tables written by the Core engine. Note that no conversion is made when the Legacy engine writes DbArray types because the Core+ engine already supports those types.

If you want your User table array columns to be Vector types, use an .update() or .updateView() clause to wrap the native arrays.

staticUserTable = db.historicalTable("MyNamespace", "MyTable")
                    .update("Longs = (io.deephaven.vector.LongVector)io.deephaven.vector.VectorFactory.Long.vectorWrap(Longs)")

Option close Tailer-DIS connections early, while continuing to monitor files

A new property is available to customize the behavior of the Tailer.

log.tailer.defaultIdlePauseTime

This property is similar to log.tailer.defaultIdleTime, but it allows the Tailer to close connections early while continuing to monitor files. When the idle time specified by log.tailer.defaultIdleTime has passed without any changes to a monitored file, the Tailer will close the corresponding connection to the DIS, and will not process any further changes to the file. The default idle time must therefore be as long as the default file rollover interval plus some buffer.

The new property enables a new feature. When the time specified by log.tailer.defaultIdlePauseTime has passed without any changes to a monitored file, the Tailer will close the corresponding connection to the DIS, but will continue to monitor the file for changes. If a change is detected, the Tailer will reopen the connection and process the changes. This will reduce or more quickly reclaim resources consumed for certain usage patterns.

Helm Chart Tolerations, Node Selectors and Affinity

You can now add tolerations, node selection, and affinity attributes to pods created by the Deephaven Helm chart. By default, no tolerations, selectors or affinity are added. To add tolerations to all created deployments, modify your values.yaml file to include a tolerations block, which is then copied into each pod. For example:

tolerations:
- key: "foo"
  operator: "Exists"
  effect: "NoSchedule"
- key: "bar"
  value: "baz"
  operator: "Equal"
  effect: "NoSchedule"

Adds the following tolerations to each pod (in addition to the default tolerations provided by the Kubernetes system):

Tolerations:                 bar=baz:NoSchedule
                             foo:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Similarly, you can add a nodeSelector or affinity block:

nodeSelector:
- key: "foo"
  operator: "Exists"
  effect: "NoSchedule"
- key: "bar"
  value: "baz"
  operator: "Equal"
  effect: "NoSchedule"

affinity:
  nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: label
            operator: In
            values:
            - value1

Which result in pods containing node selectors like:

Node-Selectors:              key1=value1
                             key2=value2

And affinity as follows:

  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: label
            operator: In
            values:
            - value1
        weight: 1

Ability to disable password authentication in front-end

A new property, authentication.client.disablePasswordAuth=true, may be used to remove the username/password authentication option from the Swing front-end. The property has no effect if there are no other login-options available.

This property does not disable username/password authentication at the server level (see Disabling password authentication).

Allow config to override ServiceRegistry hostname

The hostname which Data Import Server (DIS) registers with the service registry may now be defined in the host tag within the DIS' routing endpoint of the routing configuration; or using the new ServiceRegistry.overrideHostname system property. The precedence for the service registry host is from:

• The routing endpoint configuration. Prior to this change, the host value within the endpoint configuration was ignored.
• ServiceRegistry.overrideHostname property.
• On Kubernetes, the worker's service's hostname.
• On bare metal, it is the result of the Java InetAddress.getLocalHost().getHostName() function.

Optional lenient IOJobImpl to avoid write queue overflow

New behavior is available to avoid write queue overflow errors in the TDCP process. When a write queue overflow condition is detected, the process can be configured to delay briefly - giving the queue a chance to drain.

The following properties govern the feature:

IOJobImpl.lenientWriteQueue
IOJobImpl.lenientWriteQueue.retryDelay
IOJobImpl.lenientWriteQueue.maxDelay

Set IOJobImpl.lenientWriteQueue=true to enable this behavior. By default, the writer will wait up to IOJobImpl.lenientWriteQueue.maxDelay=60_000 ms in increments of IOJobImpl.lenientWriteQueue.retryDelay=100 ms.

This should address the following fatal error in the TDCP process:

ERROR - job:1424444133/RemoteTableDataService/10.128.1.75:37440->10.128.1.75:22015 write queue overflow: r=true, w=true, p=false, s=false, u=false, h=0, rcap=69632, rbyt=0, rmax=4259840, wbyt=315407, wspc=1048832, wbuf=4097, wmax=1048576, fc=0, allowFlush=true

Option to default all user tables to Parquet

Set the configuration property db.LegacyDirectUserTableStorageFormat=Parquet to default all direct user table operations, such as db.addTable, to the Parquet storage format. The default if the property is not set is DeephavenV1.

Deephaven processes log their heap usage

The db_dis, web_api_service, log_aggregator_service, iris_controller, db_tdcp, and configuration_server processes now periodically log their heap usage.

PersistentQueryController.log.current:[2024-05-10T15:00:32.365219-0400] - INFO - Jvm Heap: 3,972,537,856 Free / 4,291,624,960 Total (4,291,624,960 Max)
PersistentQueryController.log.current:[2024-05-10T15:01:32.365404-0400] - INFO - Jvm Heap: 3,972,310,192 Free / 4,291,624,960 Total (4,291,624,960 Max)

The logging interval can be configured using the property RuntimeMemory.logIntervalMillis. The default is one minute.

Disabling Password Authentication

To disable password authentication within the authentication server, set the configuration property authentication.passwordsEnabled to false. When the property is set to false, the authentication server rejects all password logins and you must use SAML or private key authentication to access Deephaven.

Note that even if the UI presents a password prompt, the authentication backend rejects all passwords.

Kubernetes Heap Overhead Parameters

When running Deephaven installations in Kubernetes, the originally-implemented JVM overhead properties don't prevent some workers being killed with out-of-memory exceptions.

• Adding the BinaryStoreWriterV2.allocateDirect=false JVM parameter reduces direct memory usage, which is not counted towards dispatcher heap usage and can result in Kubernetes out-of-memory failures.
• Adding the -Xms JVM parameter allocates all requested heap at worker creation time, reducing the likelihood of after-startup worker out-of-memory failures from later memory requests.
• Adding the -XX:+AlwaysPreTouch JVM parameter to workers ensures that all worker heap is touched during startup, avoiding later page-faulting.

The following properties are being added to iris-environment.prop for new installations. Deephaven strongly suggests adding them manually to existing installations.

RemoteProcessingRequestProfile.Xms.G1 GC=$RequestedHeap
RemoteQueryDispatcher.JVMParameters=-XX:+AlwaysPreTouch
BinaryStoreWriterV2.allocateDirect=false

In addition, the property RemoteQueryDispatcher.memoryOverheadMB=500 is being updated in iris-defaults.prop, and this will automatically be picked up when the Kubernetes installation is upgraded.

Dispatcher Memory Reservation

The Remote Query Dispatcher (either db_query_server or db_merge_server) has a configurable amount of heap that can be dispatched to workers, which is controlled by setting the RemoteQueryDispatcher.maxTotalQueryProcessorHeapMB property. Setting this property requires accounting the other processes that may be running on the machine. If set too high, then workers may fail to allocate memory after being dispatched after dispatch or the kernel OOM killer may terminate processes. If set too low, then the machine may be underutilized.

As an additional safety check, the Remote Query Dispatcher can query the /proc/meminfo file for available heap. If a user requests more heap than the MemAvailable field indicates can be allocated to a new process, then the remote query dispatcher can reject scheduling the worker. By default, this new functionality is disabled.

There are two new properties that control this behavior:

• RemoteQueryDispatcher.adminReservedAvailableMemoryMB; for users that are members of RemoteQueryDispatcher.adminGroups
• RemoteQueryDispatcher.reservedAvailableMemoryMB: for all other users

When set to -1, the default, the additional check is disabled. When set to a non-negative value the dispatcher subtracts the property's value from the available memory, and verifies that the worker heap is less than this value before creating the worker.

You can examine the current status of properties, using the /config endpoint if RemoteQueryDispatcher.webserver.enabled is set to true. For example, navigate to `https://query-host.example.com:8084/config'. The available memory along with property values are displayed as an HTML table.

This property does not guarantee that workers or other processes are not terminated by the OOM killer. Running workers and processes may not have allocated their maximum heap size, and therefore can use system memory beyond what is available at dispatch time.

ILLUMON_JAVA is deprecated. Use DH_JAVA instead.

In the past, specifying which version of java to use with Deephaven was done with the ILLUMON_JAVA and it was applied inconsistently.

In this release, you can set DH_JAVA=/path/to/java_to_use/bin/java in your cluster.cnf to tell all Deephaven processes where to find the correct java executable regardless of your PATH.

DH_JAVA works correctly whether you point to a java executable or a java installation directory (like "JAVA_HOME") Both DH_JAVA=/path/to/java_to_use and DH_JAVA=/path/to/java_to_use/bin/java operate identically.

If different machines in your cluster have java executables located in different locations, it is your responsibility to set DH_JAVA correctly in /etc/sysconfig/deephaven/cluster.cnf on each machine, or (preferably) to use a symlink so you have a consistent DH_JAVA location on all machines.

Core+ Controller Python Imports

From Core+ Python workers, you may now import Python modules from repositories stored in the controller. To evaluate a single Python file:

import deephaven_enterprise.controller_import

deephaven_enterprise.controller_import.exec_script("script_to_execute.py")

To import a script as a module, you must establish a meta-import with a module prefix for the controller. The following example uses the default value of "controller" to load a module of the form "package1/package2.py" or "package1/package2/__init__.py":

import deephaven_enterprise.controller_import

deephaven_enterprise.controller_import.meta_import()

import controller.package1.package2

Refreshing Local Script Repositories

The Persistent Query Controller defines a set of script repositories that can be used from Persistent Queries or Code Studios. The repositories may be configured to use a remote Git repository or just a path on the local file system. The controller scans the repository on startup for the list of scripts that are available. Previously, only Git repositories could have updates enabled (once per minute); and local repositories would never be rescanned.

You can now set the property PersistentQueryController.scriptUpdateEnabled to true to enable script updates. If this property is not set, then the old PersistentQueryController.useLocalGit property is used (the old property has an inverse sense, meaning PersistentQueryController.useLocalGit=true stops updates and PersistentQueryController.useLocalGit=false permits updates) .

To mark a repository as local, the "uri" parameter must be set to empty. For example, if the repository was reffered to as "irisrepo" in the iris.scripts.repos property, then to mark the repository as local you would include a property like in your iris-environment.prop file:

iris.scripts.repo.irisrepo.uri=

Fixing etcd ACLs that broke after upgrading to URL encodings

Note that the following is only applicable to etcd ACLs.

In 1.20231218.116 and 1.20231218.132, Deephaven began URL encoding ACL keys to prevent special characters like '/' in keys from corrupting the ACL database. Although not all special characters corrupted the database, all of them are encoded, causing the unencoded database to be incompatible with the new version. A common occurrence of this pattern is the "@" character in usernames.

These ACL entries can be fixed using the EtcdAclEncodingTool.

First, back up your etcd database by reading our backup and restore instructions.

To rewrite these ACLs with proper encodings, run the following command as irisadmin:

sudo -u irisadmin /usr/illumon/latest/bin/iris_exec com.illumon.iris.db.v2.permissions.EtcdAclEncodingTool

To see what changes would occur without actually modifying the ACLs, run:

sudo -u irisadmin /usr/illumon/latest/bin/iris_exec com.illumon.iris.db.v2.permissions.EtcdAclEncodingTool -a --dry-run

Setting JVM JIT Compiler Options for Workers

The ability to set the maximum number of allowed JVM JIT compiler threads through the -XX:CICompilerCount JVM option has been added to JVM profiles using properties of the form RemoteProcessingRequestProfile.JitCompilerCount. See the remote processing profiles documentation for further information.

Upgrade etcd to 3.5.12

In past releases, we recommended upgrading etcd to 3.5.5.

It was later discovered that 3.5.5 has a known bug which can break your etcd cluster if you perform an etcdctl password reset.

As such, when upgrading etcd, you should prefer the Deephaven-tested 3.5.12 point release, which is the new default as of version 1.20231218.190.

All newly created systems will have 3.5.12 installed, but for existing systems, you must unpack new etcd binaries yourself.

You can find manual etcd installation instructions in the Reducing Root to Zero guide.

Configurable gRPC Retries

The configuration service now supports using a gRPC service configuration file to configure retries, and one is provided by default for the system.

{
  "methodConfig": [
    {
      "name": [
          {
              "service": "io.deephaven.proto.config.grpc.ConfigApi"
          },
          {
              "service": "io.deephaven.proto.registry.grpc.RegistryApi"
          },
          {
              "service": "io.deephaven.proto.routing.grpc.RoutingApi"
          },
          {
              "service": "io.deephaven.proto.schema.grpc.SchemaApi"
          },
          {
              "service": "io.deephaven.proto.processregistry.grpc.ProcessRegistryApi"
          },
          {
              "service": "io.deephaven.proto.unified.grpc.UnifiedApi"
          }
      ],

      "retryPolicy": {
        "maxAttempts": 60,
        "initialBackoff": "0.5s",
        "maxBackoff": "2s",
        "backoffMultiplier": 2,
        "retryableStatusCodes": [
          "UNAVAILABLE"
        ]
      },

      "waitForReady": true,
      "timeout": "120s"
    }
  ]
}

methodConfig has one or more entries. Each entry has a name section with one or more service/method sections that filter whether the retryPolicy section applies.

If the method is empty or not present, then it applies to all methods of the service. If service is empty, then method must be empty, and this is the default policy.

The retryPolicy section defines how a failing gRPC call will be retried. In this example, grpc will retry for just over 1 minute while the status code is UNAVAILABLE (e.g. the service is down). Note this applies only if the server is up but the individual RPCs are being failed as UNAVAILABLE by the server itself. It the server is down, the status returned is UNAVAILABLE but the retryPolicy defined here for the method does not apply; gRPC manages reconnection retries for a channel separately/independently as described here: https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md

There is no way to configure the parameters for reconnection; see https://github.com/grpc/grpc-java/issues/9353

If the service config file specifies waitForReady, then an RPC executed when the channel is not ready (server is down) will not fail right away but will wait for the channel to be connected. This, combined with a timeout definition will make the RPC call hold on for as long as the timeout giving the reconnection policy a chance to get the channel to ready.

For Deephaven processes, customization of service config can be done by (a) copying configuration_service_config.json to /etc/sysconfig/illumon.d/resources and modifying it there, or (b) renaming it and setting property configuration.server.service.config.json.

Note that the property needs to be set as a launching JVM argument because this is used in the gRPC connection to get the initial properties.

Note: The relevant service names are:

io.deephaven.proto.routing.grpc.RoutingApi
io.deephaven.proto.config.grpc.ConfigApi
io.deephaven.proto.registry.grpc.RegistryApi
io.deephaven.proto.schema.grpc.SchemaApi
io.deephaven.proto.unified.grpc.UnifiedApi

Add Core+ Calendar support and allow Java ZoneId strings in Legacy Calendars

Core+ workers can use the Calendars.resourcePath property to load customer provided business calendars from disk. To use calendars in Core+ workers, any custom calendars on your resource path must be updated to use a standard TimeZone value.

Legacy workers also support using ZoneId strings instead of DBTimeZone values.

Dynamic management of Data Import Server configurations

Creating a new Data Import Server configuration and integrating it into the Deephaven system requires several steps, including required adjustments to the data routing configuration. This final step can now be performed with a few simple commands, and no longer requires editing the data routing configuration file.

dhconfig dis

The dhconfig command has a new action: dis, which supports import, add, export, list, delete, validate actions. The commands themselves provide help, and more information can be found in the dhconfig documentation.

dhconfig dis import

Import one or more configurations from one or more files. For example:

/usr/illumon/latest/bin/dhconfig dis import /path/to/kafka.yml

kafka.yml

kafka:
  name: kafka
  endpoint:
    serviceRegistry: registry
    tailerPortDisabled: 'false'
    tableDataPortDisabled: 'false'
  claims:
  - {namespace: Kafka}
  storage: private

dhconfig dis add

Define and import a single configuration on the command line. For example (equivalent to the import example above):

/usr/illumon/latest/bin/dhconfig dis add --name kafka --claim Kafka

dhconfig dis export

Export one or more configurations to one or more files. These files are suitable for the import command. For example, to export all configured Data Import Servers:

/usr/illumon/latest/bin/dhconfig dis export --file /tmp/import_servers.yml

dhconfig dis list

List all configured Data Import Servers. For example:

/usr/illumon/latest/bin/dhconfig dis list

Data import server configurations:
    kafka
    kafka3

dhconfig dis delete

Delete one or more configurations. For example:

/usr/illumon/latest/bin/dhconfig dis delete kafka --force

dhconfig dis validate

Validate one or more configurations. This can validate proposed changes before committing them with the import command. This process verifies that the configuration as a whole will be valid after applying the new changes.

Caveats

"Data routing configuration" comprises the "main" configuration file (managed with dhconfig routing) and additional DIS configurations. The main routing configuration may contain DIS configurations in the dataImportServers section. These two sources of DIS configurations are managed separately and are not permitted to contain duplicates. If you want to manage an existing DIS configuration with the new commands, you must remove it from the main routing configuration.

This functionality will only be useful for querying data if the routing configuration includes "all data import servers" using the dataImportServers keyword. This is usually a source under the db_tdcp table data service:

    db_tdcp:
      host: *localhost
      port: *default-tableDataCacheProxyPort
      sources:
        - name: dataImportServers

A DIS configuration requires storage. The special value private indicates that the server will supply its own storage location. Any other value must be present in the storage section of the routing configuration.

Update jgit SshSessionFactory to a more modern/supported version

For our git integration, we have been using the org.eclipse.jgit package. Github discontinued support for SHA-1 RSA ssh keys, but jgit's ssh implementation (com.jcraft:jsch) does not support rsa-sha2 signatures and will not be updated. To enable stronger SSH keys and provide GitHub compatibility, we have configured jgit to use an external SSH executable by setting the GIT_SSH environment variable. The /usr/bin/ssh executable must be present for Git updates.

Restartable Controller

If the iris_controller process restarts quickly enough, Core+ workers that were already initialized and running normally by the time the controller restarted continue running without interruption. Legacy workers still terminate on controller restart.

• The duration that workers can survive without the controller is defined by the property PersistentQueryController.etcdPresenceLeaseTtlSeconds, which defaults to 60 (seconds).
• Only workers that have completed initialization and are in the Running state before the crashed controller died and which should still be running by that time, according to their query configuration at the time of controller restart.

If the iris_controller is stopped normally (e.g., via monit stop or a regular UNIX TERM signal), the value of the property PersistentQueryController.stopWorkersOnShutdown determines the desired behavior for workers.

• When set to true, all controller-managed workers are stopped alongside the controller. This is consistent with the traditional behavior.
• When set to false (the new default), workers do not stop alongside the controller, and have the time defined in the property PersistentQueryController.etcdPresenceLeaseTtlSeconds (defaults to 60 seconds) as a grace period where they wait for the controller to come back.

If the controller crashes (i.e., the iris_controller process stopped unexpectedly by an exception that crashes the process, a machine reboot, or a UNIX KILL signal), then workers are not proactively stopped even if the value of PersistentQueryController.stopWorkersOnShutdown is true. In this case, the dispatcher terminates those workers after the PersistentQueryController.etcdPresenceLeaseTtlSeconds timeout.

Note: irrespective of the value of the PersistentQueryController.stopWorkersOnShutdown property, if the dispatcher associated to a worker is shutdown, the worker stops.

Renamed Swing Launcher Archives

The downloadable swing launcher has been renamed as follows:
DeephavenLauncherSetup_123.exe is now deephaven-launcher-123.exe
DeephavenLauncher_123.tar is now deephaven-launcher-123.tgz

Reliable Barrage table connections

We have added a new library to provide reliable Barrage subscriptions within a Deephaven Core+ cluster. The new tables monitor the state of the source query and gracefully handle disconnection and reconnections without user intervention. This can be used to create reliable meshes of Core+ workers that are fault tolerant to the loss of other queries.

When using ResolveTools, PQ URLs (pq://MyQuery/scope/MyTable?columns=MyFirstColumn,SomeOtherColumn) use these new reliable tables.

To use this library see the following examples

import io.deephaven.enterprise.remote.RemoteTableBuilder
import io.deephaven.enterprise.remote.SubscriptionOptions

// Subscribe to the columns `MyFirstColumn` and `SomeOtherColumn` of the table `MyTable` from the query `MyQuery` 
table = RemoteTableBuilder.forLocalCluster()
    .queryName("MyQuery")
    .tableName("MyTable")
    .subscribe(SubscriptionOptions.builder()
        .addIncludedColumns("MyFirstColumn", "SomeOtherColumn").build())

from deephaven_enterprise import remote_table as rt

# Subscribe to the columns `MyFirstColumn` and `SomeOtherColumn` of the table `MyTable` from the query `MyQuery`
table = rt.in_local_cluster(query_name="MyQuery", table_name="MyTable") \
        .subscribe(included_columns=["MyFirstColumn", "SomeOtherColumn"])

Connecting to remote clusters

It is also possible to connect to queries on a different Deephaven cluster.

import io.deephaven.enterprise.remote.RemoteTableBuilder

table = RemoteTableBuilder.forRemoteCluster("https://other-server.mycompany.com:8000/iris/connection.json")
        .password("user", "password")
        .queryName("MyQuery")
        .tableName("MyTable")
        .subscribe(SubscriptionOptions.builder()
                .addIncludedColumns("MyFirstColumn", "SomeOtherColumn").build())

from deephaven_enterprise import remote_table as rt

# Subscribe to the columns `MyFirstColumn` and `SomeOtherColumn` of the table `MyTable` from the query `MyQuery`
table = rt.for_remote_cluster("https://other-server.mycompany.com:8000/iris/connection.json")
    .password("username", "password") \
    .query_name("MyQuery") \
    .table_name("MyTable") \
    .subscribe(included_columns=["MyFirstColumn", "SomeOtherColumn"])

ACLs for Update Core+ Performance Tables

Preexisting installs must manually add new ACLs for the new DbInternal tables.

First, create a text file (e.g. /tmp/new-acls.txt) with the following contents:

-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))' -group allusers -namespace DbInternal -table ProcessMetricsLogCoreV2 -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))' -group allusers -namespace DbInternal -table ServerStateLogCoreV2 -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table UpdatePerformanceLogCoreV2 -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table QueryOperationPerformanceLogCoreV2 -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table QueryPerformanceLogCoreV2 -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table UpdatePerformanceLogCoreV2Index -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table QueryOperationPerformanceLogCoreV2Index -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table QueryPerformanceLogCoreV2Index -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))' -group allusers -namespace DbInternal -table ServerStateLogCoreV2Index -overwrite_existing
exit

Then, run the following to add the new ACLs into the system:

sudo -u irisadmin /usr/illumon/latest/bin/iris iris_db_user_mod --file /tmp/new-acls.txt

Alternatively, the ACLs can be added manually one by one in the ACL Editor:

allusers | DbInternal | ServerStateLogCoreV2 | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))
allusers | DbInternal | ProcessMetricsLogCoreV2 | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))
allusers | DbInternal | UpdatePerformanceLogCoreV2 | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))
allusers | DbInternal | QueryOperationPerformanceLogCoreV2 | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))
allusers | DbInternal | QueryPerformanceLogCoreV2 | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))
allusers | DbInternal | ServerStateLogCoreV2Index | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))
allusers | DbInternal | UpdatePerformanceLogCoreV2Index | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))
allusers | DbInternal | QueryOperationPerformanceLogCoreV2Index | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))
allusers | DbInternal | QueryPerformanceLogCoreV2Index | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))

Worker name format change

Worker names are no longer assigned in an ascending manner beginning from "worker_1". Instead worker names begin with "worker_" followed by a prefix of the process info ID. Note that the worker name is not guaranteed to be unique, using process info id is the only way to reliably find a specific worker within logs.

The "request ID" field has been removed from the RemoteProcessingRequest. The client now assigns the process info id, therefore you can use that to search in logs both on the client and server.

Custom Setter Support

For CSV imports using the new Deephaven Community CSV parser CustomSetters are now supported.

The changes are backward compatible so existing CustomSetter implementations continue to work as is. However, it is recommended to use the new custom setter interface for new imports and consider transitioning existing imports to the new interface.

The new interface provides two key benefits

• Avoids creating of CSVRecord objects
• Column Data types are retained (extracted column values from CSVRecord would always be string)
• Additionally, passed in constant values may directly be accessed as getConstantColumnValue()

CustomSetter example (Legacy)

Below is a simple example Custom Setter implementation using the legacy approach. The next section details how to convert this to the new interface.

The below example is of a Full Name column that is compiled using First Name, Last Name columns and optionally may include Name Prefix if the Name Prefix constant column is included

Legacy Schema for Full Name Column

<Table name="ConstCustomSetter" namespace="Test" storageType="NestedPartitionedOnDisk">
  <Partitions keyFormula="__PARTITION_AUTOBALANCE_SINGLE__" />

  <Column name="Partition" dataType="String" columnType="Partitioning" />
  <Column name="FullName" dataType="String" />
  <Column name="FirstName" dataType="String" />
  <Column name="LastName" dataType="String" />

  <ImportSource name="IrisCSV" type="CSV" arrayDelimiter="," >
    <ImportColumn name="NamePrefix" sourceType="CONSTANT" />
    <ImportColumn name="FullName" sourceName="FirstName" class="com.illumon.iris.importers.CsvConstColumnSetterExample" />
    <ImportColumn name="FirstName" />
    <ImportColumn name="LastName" />
  </ImportSource>
</Table>

Legacy Implementation CsvConstColumnSetterExample

package com.illumon.iris.importers;

import com.fishlib.io.logger.Logger;
import com.illumon.iris.binarystore.RowSetter;
import org.apache.commons.csv.CSVRecord;

import java.io.IOException;

/**
 * Example Custom Setter implementation for Full Name Column using Legacy approach
 */
public class CsvConstColumnSetterExample extends CsvFieldWriter{

    private final RowSetter setter;
    private final ImporterColumnDefinition column;

    /**
     * Constructor using the format that is required for custom CsvFieldWriters
     * 
     * @param log The passed in logger
     * @param strict The strict parameter as chosen for the Import
     * @param column The import column definition for the CustomSetter column
     * @param setter The RowSetter to be used to populate the Column value for the Row
     * @param delimiter The array delimiter used in the import
     */
    public CsvConstColumnSetterExample(final Logger log, final boolean strict, final ImporterColumnDefinition column, final RowSetter setter,
                                 final String delimiter) {
        super(log, column.getName(), delimiter);
        this.setter = setter;
        this.column = column;
    }

    @SuppressWarnings("unchecked")
    @Override
    public void processField(final CSVRecord record) throws IOException {
        setter.set(getConstantColumnValue("NamePrefix") + " " + record.get("FirstName") + " " + record.get("LastName"));
    }

}

New Interface implementation for Full Name

Below are the schema and implementation class of the CustomSetter for the same Full Name column using the new interface.

Schema

<Table name="NewFormatConstCustomSetter" namespace="Test" storageType="NestedPartitionedOnDisk">
  <Partitions keyFormula="__PARTITION_AUTOBALANCE_SINGLE__" />

  <Column name="Partition" dataType="String" columnType="Partitioning" />
  <Column name="FullName" dataType="String" />
  <Column name="FirstName" dataType="String" />
  <Column name="LastName" dataType="String" />

  <ImportSource name="IrisCSV" type="CSV" arrayDelimiter="," >
    <ImportColumn name="FullName" class="com.illumon.iris.importers.CsvDhcConstColumnSetterExample" />
  </ImportSource>
</Table>

Implementation Class

As shown below the key differences are

1. The Base class is BaseCsvFieldWriter
2. The method to implement is void processRow(@NotNull final Map<String, CustomSetterValue<?>> columnNameToValueMap)
   1. The columnNameToValueMap as name suggests the key in the map is the ColumnName and points to CustomSetterValue object, which holds the appropriate column value by type
   2. The CustomSetterValue is an implements RowSetter<?> interface but also supports getter allowing ability to save and retrieve values by their type
3. In addition, the passed in constant can be retrieved using getConstantColumnValue(), though the legacy way of using getConstantColumnValue("NamePrefix") where NamePrefix is the ImportColumn name.
   1. XmlImports has support to pass in importProperties map which allows for multiple Constant columns in that case it would be preferred to use getConstantColumnValue(ColumnName)

package com.illumon.iris.importers;

import com.fishlib.io.logger.Logger;
import com.illumon.iris.binarystore.RowSetter;
import org.jetbrains.annotations.NotNull;

import java.util.Map;

/**
 * Example Custom Setter implementation for Full Name Column using New Format
 */
public class CsvDhcConstColumnSetterExample extends BaseCsvFieldWriter {

    private final RowSetter<String> setter;

   /**
    * Constructor required for custom BaseFieldWriter
    *
    * @param log       The passed in log
    * @param strict    The value of strict flag chosen for import
    * @param column    The import column definition for the CustomSetter column
    * @param setter    The RowSetter that will be used to set the property
    * @param delimiter The array delimiter used
    */
    public CsvDhcConstColumnSetterExample(final Logger log, 
                                          final boolean strict, 
                                          final ImporterColumnDefinition column,
                                          final RowSetter<?> setter, 
                                          final String delimiter) {
        super(log, column.getName(), delimiter);
        //noinspection unchecked
        this.setter = (RowSetter<String>) setter;
    }

    @Override
    public void processRow(@NotNull final Map<String, Object> columnNameToValueMap) {
        final String firstName = (String) columnNameToValueMap.get("FirstName");
        final String lastName = (String) columnNameToValueMap.get("LastName");
        final String fullName = getConstantColumnValue() + " " + firstName + " " + lastName;
        setter.set(fullName);
    }

}

IrisLogCreator constructor changes

The constructors in the IrisLogCreator class have been changed. Any uses of these constructors should add a new boolean parameter to the call, which is used to determine whether or not to create an audit event logger. The old constructors have been deprecated but are still available and do not create audit event loggers.

Automatically Provisioned Python venv Will Only Use Binary Dependencies

All pip installs performed as part of the automatic upgrade of Python virtual environments will now pass the --only-binary=:all: flag, which will prevent pip from ever attempting to build dependencies on a customer machine.

As part of this change, we automatically upgrade pip and setuptools in all virtual environments, and have upgraded a number of dependencies which for pip refused to use prebuilt binary dependencies:

For all virtual environments:
dill==0.3.1.1 is now dill==0.3.3
wrapt==1.11.2 is now wrapt==1.13.2

For jupyter virtual environments:
backcall==0.1.0 is now backcall==0.2.0
tornado==6.0.3 is now tornado==6.1

Product Installation File Rename

The Deephaven tar / RPM installation files have been renamed to include the Java version they are built for, and to better replace legacy names with modern product names.

The Enterprise installer tar now has a -jdkN classifier. For example, illumon-db-1.20231212.123.tar.gz is now deephaven-enterprise-1.20231212.123-jdk17.tar.gz.

The Enterprise rpm now has the jdk major version as deephaven-enterprise minor version. For example, illumon-db-1.20231212.123-1-1.rpm is now deephaven-enterprise-1.20231212.123-17-1.rpm.

The Core+ tar file has been gzipped and renamed with a jdkN classifier and a .tgz file extension.
For example, io.deephaven.enterprise.dnd-0.32.0-1.20231212.123.tar is nowdeephaven-coreplus-0.32.0-1.20231212.123-jdk17.tgz.

Note that ONLY the filenames and RPM package name have changed. All paths on the filesystem still reflect legacy locations, except for a single renamed file:

/usr/illumon/dnd/latest/bin/io.deephaven.enterprise.dnd has been renamed to /usr/illumon/dnd/latest/bin/deephaven-coreplus.

MergeDataBuilder refactored

The hierarchy of Java classes that manage merge operations had become unwieldy. In this release, we refactored the internals to consistently use the same MergeDataBuilder interface that has been the preferred mechanism for user scripts.

We expanded that Builder class to include settings needed to support operations previously accessed via special object classes and overloaded methods. One noteworthy example is the deleted MergeFromTable class, the functionality of which is now accessed directly using the sourceTable method of the builder.

The merge methods of the Data Merging Classes taking large numbers of parameters are gone. Scripts using them can be converted to use the builder pattern straightforwardly.

The details of the builder API have changed in several ways. All scripts and programs that directly initiate a merge operation will likely require attention. Merge Persistent Queries and the tools for interacting with them are not affected, but any Persistent Query that initiates a merge using script syntax will require attention.

Please refer to the Merge API Reference section of the Deephaven Enterprise user guide for full details and examples that illustrate the necessary changes.

Below are the "before" and "after" versions of the changed portion of the example script.

Before

new MergeFromTable().merge(
	log, // automatically available in a console worker
	com.fishlib.util.process.ProcessEnvironment.getGlobalFatalErrorReporter(),
	namespace,
	tableName,
	date,
	threadPoolSize,
  maxConcurrentColumns,
	lowHeapUsage,
	force,
	allowEmptyInput,
	sortColumnFormula,
	db, // automatically available in a console worker
	progress,
	null, // storageFormat not needed
	null, // parquetCodecName not needed
	null, // syncMode not needed
	lateCleanup,
	sourceTable)

After

MergeParameters params = MergeParameters.builder(db, namespace, tableName)
        .partitionColumnValue(date)
        .threadPoolSize(threadPoolSize)
        .maxConcurrentColumns(maxConcurrentColumns)
        .lowHeapUsage(lowHeapUsage)
        .force(force)
        .allowEmptyInput(allowEmptyInput)
        .sortColumnFormula(sortColumnFormula)
        .lateCleanup(lateCleanup)
        .sSourceTable(sourceTable)
        .build()
MergeData.of(params).run(com.fishlib.util.process.ProcessEnvironment.getGlobalFatalErrorReporter(), progress)

Switch to NFS v4 for Kubernetes RWX Persistent Volumes

NFS v3 Persistent Volume connections do not support locking. This manifests most obviously when attempting to work with user tables in Deephaven on Kubernetes. By default, user table activities will wait indefinitely to obtain a lock to read or write data. This can be bypassed by setting -DOnDiskDatabase.useTableLockFile=false; this work-around was provided by DH-15640.

This change (DH-15830) switches Deephaven Kubernetes RWX Persistent Volume definitions to use NFS v4 instead, which includes lock management as part of the NFS protocol itself. In order for this change to be made, the NFS server must be reconfigured to export the RWX paths relative to a shared root path (fsid=0), but the existing PVs must use the same path to connect, since PV paths are immutable.

There are two options to reconfigure the NFS server:

1. The Deephaven Kubernetes install wrapper script (dh_helm) can be used for the upgrade; it automatically checks for an NFS Pod that was deployed as part of Deephaven Kubernetes setup, and runs an upgrade script to reconfigure it if it is not already exporting an NFS v4 path.

2. In cases where the NFS server is not a Deephaven deployed Pod, or where you want to make other changes to the NFS configuration, you can manually run the upgrade-nfs-minimal.sh script against the NFS server. It is important to set the environment variable SETUP_NFS_EXPORTS to y before running the script.

   • To manually run the script against an NFS Pod:

     • Run kubectl get pods to get the name of your NFS server Pod and confirm that it is running.

     • Copy the setup script to the NFS pod by running this command, using your specific NFS pod name:

       # Run 'kubectl get pods' to find your specific nfs-server pod name and use that as the copy target host in this command.
       kubectl cp setupTools/upgrade-nfs-minimal.sh <nfs-server-name>:/upgrade-nfs-minimal.sh
       

     • Run this command to execute that script, once again substituting the name of your NFS Pod:

       kubectl exec <nfs-server-name> -- bash -c "export SETUP_NFS_EXPORTS=y && chmod 755 /upgrade-nfs-minimal.sh && /upgrade-nfs-minimal.sh"
       

The upgrade script:

• replaces /etc/exports, and backs up the original file to /etc/exports_<epoch_timestamp>. The new file will have only one entry, which exports the /exports directory with fsid=0.
• adds an exports sub-directory under /exports, and moves the dhsystem directory there. This is so clients will still find their NFS paths under /exports/dhsystem when connecting to the fsid=0 "root".

The existing PVs spec sections are updated with:

mountOptions:
    - hard
    - nfsvers=4.1

After upgrading to a version of Deephaven that includes this change (DH-15830), you should remove the -DOnDiskDatabase.useTableLockFile=false work-around, so normal file locking behavior can be used when working with user tables.

Requiring ACLs on all exported objects

When exporting objects from a Persistent Query, there are now two modes of operation controlled by the property PersistentQuery.openSharingDefault.

In either mode, when an ACL is applied to any object (e.g, tables or plots) within the query, then objects without an ACL are only visible to the query owner and admins (owners and admins never have ACLs applied).

When a viewer connects:

• If PersistentQuery.openSharingDefault is set to true, persistent queries that are shared without specifying table ACLs allow all objects to be exported to viewers of the query without any additional filters supplied. This is the existing Deephaven behavior that makes it simple to share PQ work product with others.
• If PersistentQuery.openSharingDefault is set to false, persistent queries that are shared without specifying table ACLs do not permit objects without an ACL applied to be exported to viewers. The owner of the persistent query must supply ACLs for each object that is to be exported.

Setting this property to false makes it less convenient to share queries, but reduces the risk of accidentally sharing data that the query writer did not intend. To enable this new behavior, you should update your iris-environment.prop property file.

Tailer configuration changes to isolate user actions

The tailer allocates resources for each connection to a Data Import Server for each destination (namespace, table name, internal partition, and column partition). System table characteristics are predictable and fairly consistent, and can be used to configure the tailer with appropriate memory.

User tables are controlled by system users, so their characteristics are subject to unpredictable variations. It is possible for a user to cause the tailer to consume large amounts of tailer resources, which can impact System data processing or crash the process.

This change adds more properties for configuration, and adds constraints on User table processing separate from System tables.

User table isolation

Resources for User table locations are taken from a new resource pool. The buffers are smaller by default, and the pool has a constrained size. This puts an upper limit on memory consumption when users flood the system with changed locations, which can happen with closeAndDeleteCentral or when back filling data. The resources for this pool are pre-allocated at startup. The pool size should be large enough to handle expected concurrent user table writes.

Tailer/DIS configuration options

The following properties configure the memory consumption of the Tailer and Data Import Server processes.

Revert to previous behavior

To disable the new behavior in the tailer, set the following property:

DataContent.disableUserPool = true

Added block flag to more dh_monit actions

This flag blocks scripting for the start, stop, and restart actions until the actions are completed. If any actions other than start, stop, restart, up or down are passed with the blocking flag, an error is generated. No other behaviors of the script have been changed.

These following options have been added:

/usr/illumon/latest/bin/dh_monit [ start | stop | restart ] [ process name | all ] [ -b | --block ]

These work as before:

/usr/illumon/latest/bin/dh_monit [ up | down ] [ -b | --block ]

Core Worker Notebook and Controller Groovy Script Imports

Users can now import Groovy scripts from their notebooks and the controller git integration from Community Core workers.

To qualify for such importing, Groovy scripts must:

1. Belong to a package.
2. Match their package name to their file location. For example, scripts belonging to package name com.example.compute must be found in com/example/compute.

If a script exists with the same name as a notebook and in the controller Git integration, the notebook is prioritized as it is easier for users to modify if needed.

Importing Notebook Groovy Scripts

Below is a Groovy script notebook at test/notebook/NotebookImport.groovy:

package test.notebook

return "Notebook"

String notebookMethod() {
  return "Notebook method"
}

static String notebookStaticMethod() {
    return "Notebook static method"
}

class NotebookClass {
    final String value = "Notebook class method"
    String getValue() {
        return value
    }
}

static String notebookStaticMethodUsingClass() {
  new NotebookClass().getValue()
}

Below is an example of importing and using the groovy script from a user's notebooks. Note that per standard Groovy rules, you can run the script's top-level statements via main() or run() or use its defined methods like a typical Java class:

import test.notebook.NotebookImport

NotebookImport.main()
println new NotebookImport().run()
println new NotebookImport().notebookMethod()
println NotebookImport.notebookStaticMethod()
println NotebookImport.notebookStaticMethodUsingClass()

You can also use these classes and methods within Deephaven formulas:

import test.notebook.NotebookImport

import io.deephaven.engine.context.ExecutionContext
import io.deephaven.engine.util.TableTools

ExecutionContext.getContext().getQueryLibrary().importClass(NotebookImport.class)
testTable = TableTools.emptyTable(1).updateView(
        "Test1 = new NotebookImport().run()",
        "Test2 = new NotebookImport().notebookMethod()",
        "Test3 = NotebookImport.notebookStaticMethod()",
        "Test4 = NotebookImport.notebookStaticMethodUsingClass()"
)

Importing Controller Git Integration Groovy Scripts

Importing scripts from the controller git integration works the same way, except that script package names don't necessarily need to match every directory. For example, if the following property is set:

iris.scripts.repo.<repo>.paths=module/groovy

Then the package name for the groovy script at module/groovy/com/example/compute must be com.example.compute, not module.groovy.com.example.compute.

Logging System Tables from Core+

Core+ workers can now log Table objects to a System table.

Many options are available using the Builder class returned by:

import io.deephaven.enterprise.database.SystemTableLogger
opts = SystemTableLogger.newOptionsBuilder().currentDateColumnPartition(true).build()

The only required option is what column partition to write to. You may specify a fixed column partition or use the current date (at the time the row was written, data is not introspected for a Timestamp). The default behavior is to write via the Log Aggregator Service, but you can also write via binary logs. No code generation or listener versioning is performed, you must write columns in the format that the listener expects. Complete Options are available in the Javadoc.

After creating an Options structure, you can then log the current table:

SystemTableLogger.logTable(db, "Namespace", "Tablename", tableToLog, opts)

When logging incrementally, a Closeable is returned. You must retain this object to ensure liveness. Call close() to stop logging and release resources.

lh=SystemTableLogger.logTableIncremental(db, "Namespace", "Tablename", tableToLog, opts)

The Python version does not use any options, but rather named arguments. If you specify None for the column partition, then the current date is used.

system_table_logger.log_table("Namespace", "Tablename", table_to_log, columnPartition=None)

Similarly, if you call log_table_incremental from Python; then you must close the returned object (or use it as context manager in a with statement)

Row by row logging is not yet supported in Core+ workers. Existing binary loggers cannot be executed in the context of a Core+ worker; because they reference classes that are shadowed (renamed). If row-level logging is required, then you must use io.deephaven.shadow.enterprise.com.illumon.iris.binarystore.BinaryStoreWriterV2 directly.

Only primitive types, Strings and Instants are supported. Complex data types cannot yet be logged.

Restrict available WorkerKinds with ACL groups

Use the new configuration parameter WorkerKind.<worker kind>.allowedGroups to set the ACLs for individual WorkerKinds. Groups are separated by commas. For example,

WorkerKind.DeephavenEnterprise.allowedGroups=iris-superusers,a_group

restricts the users that can create Enterprise (Legacy) workers to members of iris-superusers and a_group.

If not configured, the default is allusers.

Core+ support for multiple partitioning columns

Deephaven Core+ workers now support reading tables stored in the Apache Hive layout. Hive is a multi-level partitioned format where each directory is a Key=Value pair.

For example:

| Market                          -- A Directory for the Namespace
| -- EquityTrade                  -- A directory for the Table
|  | -- Region=US                 -- A Partition directory for the Region `US`
|  |  | -- Class=Equities         -- A Partition directory for the Class `Equities`
|  |  |  | -- Symbol=UVXY         -- A Partition directory for the Symbol `UVXY`
|  |  |  |  | -- table.parquet    -- A Parquet file containing data,
|  |  |  | -- Symbol=VXX          -- A Partition directory for the Symbol `VXX`
|  |  |  |  | -- table.size       -- A set of files for a Deephaven format table
|  |  |  |  | -- TradeSize.dat
|  |  |  |  | -- ...
|  | -- Region=Asia
|  |  | -- Class=Special
|  |  |  | -- Symbol=ABCD
|  |  |  |  | -- table.parquet
|  |  |  | -- Symbol=EFGH
|  |  |  |  | -- table.parquet

See the extended layouts documentation for more details on how to use this feature.

Core+ support for writing tables in Deephaven format

Deephaven Core+ workers now support writing tables in Deephaven format using the io.deephaven.enterprise.table.EnterpriseTableTools class in Groovy workers and the deephaven_enterprise.table_tools python module.

For example, to read a table from disk:

import io.deephaven.enterprise.table.EnterpriseTableTools
t = EnterpriseTableTools.readTable("/path/to/the/table")

from deephaven_enterprise import table_tools
t = table_tools.read_table("/path/to/the/table")

And to write a table:

import io.deephaven.enterprise.table.EnterpriseTableTools
EnterpriseTableTools.writeTable(qq, new File("/path/to/the/table"))

from deephaven_enterprise import table_tools
table_tools.write_table(table=myTable, path="/path/to/the/table")

See the Core+ documentation for more details on how to use this feature.

Core+ C++ client and derived clients support additional CURL options

When configuring a Session Manager with a URL for downloading a connection.json file, the C++ client and derived clients (like Python ticking or R) use libcurl to download the file from the supplied URL. SSL connections in this context can fail for multiple options and it is customary to support options to adjust SSL behavior and/or enable verbose output for supporting debugging. We now support the following environment variables from the clients:

• CURL_CA_BUNDLE: like the variable of the same name for the curl(1) command line utility. Points to a file containing a CA certificate chain to use instead of the system default.
• CURL_INSECURE: if set to any non-empty value, disable validation of server certificate.
• CURL_VERBOSE: if set to any non-empty value, enable debug output.

New Worker Labels

The Deephaven Enterprise system supports two kinds of workers.

The first uses the legacy Enterprise engine that predates the release of Deephaven Community Core. These workers are now labeled "Legacy" in the Code Studio and Persistent Query "Engine" field. Previously, these workers were labeled "Enterprise".

The second kind uses the Deephaven Community Core engine with Enterprise extensions. These workers are now labeled "Core+" in the Code Studio and Persistent Query "Engine" field. Previously, these workers were labeled "Community".

Although these changes may create short-term confusion for current users, Deephaven believes they better represent the function of these workers and will easily become familiar. Both Legacy and Core+ workers exist within the Deephaven Enterprise system. The Core+ workers additionally include significant Enterprise functionality that is not found within the Deepahven Community Core product.

To avoid breaking user code, we have not yet changed any package or class names that include either "Community" or "DnD" (an older abbreviation which stood for "Deephaven Community in Deephaven Enterprise").

Logger overhead

The default Logger creates a fixed pool of buffers. Certain processes are fine with a smaller size.

The following properties can be used to override the default configuration of the standard process Logger. Every log message uses an entry from the entry pool, and at least one buffer from the buffer pool. Additional buffers are taken from the buffer pool as needed. Both pools will expand as needed, so the values below dictate the minimum memory that will be consumed.

The default value for IrisLogCreator.entryPoolCapacity has been reduced to 16384 for Tailer processes.

generate-iris-keys and generate-iris-rsa no longer overwrite output

The generate-iris-keys and generate-iris-rsa scripts use OpenSSL to generate public and private keys. If you have an existing key file, the scripts now exit with a failure and you must remove the existing file before regenerating the key.

Additional Kubernetes Worker Creation Parameters

The Query Dispatcher now supports changing more Kubernetes parameters when creating a worker, which include:

• Persistent Volume Claim will mount an existing claim in your worker pod if it exists and is not already mounted elsewhere. If no claim exists, a new PersistentVolumeClaim will be created. If using a storage class that allows for dynamic volume creation, then the PersistentVolume will also be created. Note that creating a new claim is subject to a validation check that requires a configured validator which will allow it. See the link below for more.
• Storage Class is the storage class to be used for a new persistent volume claim. Acceptable Values will vary depending on your Kubernetes provider, the default is inserted into iris-endpoints.prop by the Helm chart's global.storageClass value. If using an existing claim this has no effect.
• Storage Size is the size of the volume to be requested if creating a new PersistentVolumeClaim, in bytes as documented here. If using an existing claim this has no effect.
• Mount Path denotes where in the pod the PersistentVolumeClaim will be mounted.

These are in addition to the existing parameters described in a previous release note introducing Kubernetes worker creation parameters and validators.

Kubernetes Helm Chart Changes

Some settings have changed or have been explicitly provided in place of whatever the default value was for your Kubernetes platform provider. For example terminationGracePeriodSeconds is set to a default of 10 in the management-shell. To avoid possible errors, delete the management-shell pod prior to doing the helm upgrade if you have an older version already running. The pod can be deleted with this command: kubectl -n <your-namespace> delete pod management-shell --grace-period 1.

Note that any files you may have copied or created locally on that pod will be removed. However, in the course of normal operations such files would not be present.

Controller client for Community Workers

This change reorganizes dependencies so that Community workers do not require the shadowed Controller and Console modules. It also provides a io.deephaven.enterprise.dnd.controller.PersistentQueryControllerClient interface and implementation for DnD workers to use. Both Enterprise and DnD implementations now use the same shared underlying gRPC implementation.

Relocated Classes

The com.illumon.iris.controller.HeaderPopupProvider class was moved to the Gui module as com.illumon.iris.gui.table.HeaderPopupProvider

Dependency updates

Deephaven has updated several dependencies to more recent versions. If you are using these dependencies in your scripts or other code running in the worker, then your code may need updates.

Shadowed dependencies, which generally should not be used directly have also been updated. Of note is that Jackson which must sometimes be referenced as io.deephaven.shadow.jackson.com.fasterxml to interface with Deephaven ingestion classes has been updated from 2.14.2 to 2.15.2.

Of particular note is that Groovy 3.0.19 includes at least one bug fix that changes the behavior of scripts. The Java Language Specification does not permit inheriting static members from parent classes, but Groovy versions prior to 3.0.18 did. GROOVY-8164 makes the Groovy language consistent with the JLS, but existing scripts that depend on inheriting static members fail at runtime.

Status Dashboard

A status dashboard process has been added to the Deephaven installation, providing data in a format that can be read by Prometheus. Full documentation is available in the System Administration section of the Deephaven documentation.

Updated Python Version

The default Python version is now 3.10, which is updated from 3.8. Python 3.8 and 3.9 are still supported, but Python 3.7 support has been dropped.

Python 3.10 drops support for numpy.object, numpy.bool and numpy.int. If you use these in your scripts, then you must use the corresponding built-in Python object, bool, and int types.

Of particular note is that Python 3.10 does not support OpenSSL 1.0, therefore you must install OpenSSL 1.1 to build Python 3.10. SSL support is required for the wheel package. CentOS 7 has OpenSSL 1.1 packages, which may be installed; but the default directory layout is not suitable for Python 3.10. If the installer root prepare script must build Python 3.10 on CentOS 7, then /usr/illumon/openssl11 is created with symlinks to the openssl11-devel yum package.

Kafka Offset Column Name

The default Community name for storing offsets is KafkaOffset. The Core+ Kafka ingester assumed this name, rather than using the name from the deephaven.offset.column.name consumer property.

If the default columns names of KafkaOffset, KafkaPartition, and KafkaTimestamp are not in your Enterprise schema, then the ingester ignores those columns. If you change column names for timestamp, offset, or partition; then you must also ensure that your schema contains a column of the correct type for that column.

Bypassing user table lock files

When a worker tries to write or read a User table, it will first try to lock a file in /db/Users/Metadata to avoid potential concurrency issues. If filesystem permissions are set up incorrectly, or if the underlying filesystem does not support file locking, this can cause issues.

The following property can be set to disable the use of these lock files:

OnDiskDatabase.useTableLockFile=false

Worker-to-worker table resolution configuration

Worker-to-worker table resolution now uses the Deephaven cluster's trust store by default. In some environments, there may be a SSL-related exception when when trying to resolve a table defined in one persistent query from another (see sharing tables for more). The property uri.resolver.trustall may be set to true globally in a Deephaven configuration file, or as a property in a Code Studio session as a JVM argument (e.g. -Duri.resolver.trustall=true). This will let the query worker sourcing the table trust a certificate that would otherwise be untrusted.

Added Envoy properties to allow proper operation in IPv6 or very dynamic routing environments

The new properties envoy.DnsType and envoy.DnsFamily allow configuration of Envoy DNS behaviors for xds routes added by the Configuration server.

• envoy.DnsType configures the value to be set in dynamically added xds routes for type. The default if this property is not set is LOGICAL_DNS. If there is a scenario where DNS should be checked on each connection to an endpoint, this can be changed to STRICT_DNS. Refer to Envoy documentation for more details about possible settings.

• envoy.DnsFamily configures the value to be set in dynamically added xds routes for dns_lookup_family. The default if this property is not set is AUTO. In environments where IPv6 is enabled the AUTO setting may cause Envoy to resolve IPv6 addresses for Deephaven service endpoints; since these service endpoints only listen on IPv4 stacks, Envoy will return a 404 or 111 when getting "Connection refused" from the IPv6 stack. Refer to Envoy documentation for more details about possible settings.

Since Deephaven endpoint services listen only on IPv4 addresses, and Envoy, by default, prefers IPv6 addresses, it may be necessary to modify the configuration in environments where IPv6 is enabled. To do this:

1. add an entry to the iris-environment.prop properties file of envoy.DnsFamily=V4_ONLY

2. edit the envoy3.yaml (or whichever configuration file Envoy is using) and add dns_lookup_family=V4_ONLY to the xds_service section:

   static_resources:
     clusters:
       - name: xds_service
         connect_timeout: 0.25s
         type: STRICT_DNS
         dns_lookup_family: V4_ONLY
   

3. import the new configuration and restart the configuration server and the Envoy process for the changes to take effect.

Modified Bessel correction formula for weighted variance

The weighted variance computation formula has been changed to match that used in the Deephaven Community engine. We now use the standard formula for "reliability weights" instead of the previous "frequency weights" interpretation. This will affect statistics based on variance such as standard deviation.

Managing Community Worker Python Packages

When starting a Deephaven Python worker, it executes in the context of a Python virtual environment (venv). This environment determines what packages are available to Python scripts. Packages that are important systemically or for multiple users should be added to the permanent virtual environment. With Community workers, the administrator may configure multiple worker kinds each with distinct virtual environments to enable more than one environment with a simple drop-down menu. For legacy Enterprise workers, users must manually set properties to select different virtual environments.

For experimentation, it can be convenient to install a Python package only in the context of the current worker. Community Python workers now have a deephaven_enterprise.venv module, which can be used to query the current path to the virtual environment and to install packages into the virtual environment with via pip with the install method. On Kubernetes, the container images now permit dbquery and dbmerge to write to the default virtual environment of /usr/illumon/dnd/venv/latest; which has no persistent effects on the system.

On a bare-Linux installation, the /usr/illumon/dnd/venv/latest must not be writable by users to ensure isolation between query workers. To allow users to install packages into the virtual environment, the administrator may configure a worker kind to create ephemeral environments on worker startup by setting the property WorkerKind.<name>.ephemeralVenv=true. This process increases worker startup time as it requires executing pip freeze and then pip install to create a clone of the original virtual environment. With an ephemeral virtual environment, the user can use deephaven_enterprise.venv.install to add additional packages to their worker. There is currently no interface to choose ephemeral environments at runtime.

Kubernetes Image Customization

When building container images for Kubernetes, Deephaven uses a default set of requirements that provide a working environment. However, many installations require additional packages. To facilitate adding new packages to the default virtual environment, a customer_requirements.txt file can be added to the deephaven_python and db_query_worker_dnd subdirectories of the docker build. After installing the default packages into the worker's virtual environment, pip is called to install the packages listed in customer_requirements.txt. If these files do not exist, the Deephaven build script creates empty placeholder customer_requirements.txt files.

With JDK Launcher Discontinued

The Windows launcher package that included a JDK has been discontinued. You must install a JDK on the client machine before running the Swing launcher. Please note that a JRE is not sufficient to run the Swing console, you must use a JDK.

The launcher is now compiled with JDK8, so that even if you download it from a Deephaven instance that is running a later Java version you may use it with Deephaven instances running older versions of Java. The JDK of the Swing client still must match that of the Deephaven server.

Make /db/Users mount writeable in Kubernetes

This changes both the yaml for worker templates and the permissions on the underlying volume that is mounted as /db/Users in pods. If you are installing a new cluster, there is no action necessary. However, if you have an existing cluster installed then run this command to change the permissions: kubectl exec management-shell -- /usr/bin/chmod -vR 775 /db/Users

Helm improvements

A number of items have been added to the Deephaven helm chart, which allow for the following features:

• Configuration options to use an existing persistent volume claim in Deephaven, to allow for use of historical data stored elsewhere.
• Configuration options to mount existing secrets into worker pods.
• Configurable storageClass options to allow for easier deployment in various Kubernetes providers.

Required action when upgrading from an earlier release

1. Define global.storageClass: If you have installed an earlier version of Deephaven on Kubernetes then your my-values.yaml file used for the upgrade (not the Deephaven chart's values.yaml) should be updated to include a global.storageClass value, e.g.:

   global:
      storageClass: "standard-rwo"    # Use a value suitable for your Kubernetes provider
   

   The value should be one that is suitable for your Kubernetes provider; standard-rwo is a GKE-specific storage class used as an example. To see storageClass values suitable for your cluster, consult your provider's documentation. You can view your cluster's configured storageClass by running kubectl get storageClasses

2. Delete management-shell pod prior to running helm upgrade: Run kubectl delete pod management-shell to delete the pod. Note that if you happen to have any information stored on that pod it would be removed, though in the normal course of operations that would not be the case. This pod mounts the shared volumes used elsewhere in the cluster, and so changes to the storageClass values might result in an error similar to the following if it is not deleted when the upgrade is performed:

   $ helm upgrade my-deephaven-release-name ./deephaven/ -f ./my-values.yaml --set image.tag=1.20230511.248 --debug
   
   Error: UPGRADE FAILED: cannot patch “aclwriter-binlogs” with kind PersistentVolumeClaim: PersistentVolumeClaim “aclwriter-binlogs”
   is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
     core.PersistentVolumeClaimSpec{
       ... // 2 identical fields
       Resources:        {Requests: {s”storage”: {i: {...}, s: “2Gi”, Format: “BinarySI”}}},
       VolumeName:       “pvc-80a518f6-1a24-4c27-93b5-c7e9bd25d824”,
   -   StorageClassName: &“standard-rwo”,
   +   StorageClassName: &“default”,
       VolumeMode:       &“Filesystem”,
       DataSource:       nil,
      DataSourceRef:    nil,
   }
   

Ingesting Kafka Data from DnD

The Deephaven Community Kafka ingestion framework provides several advantages over the existing Enterprise framework. Notably:

• The Community Kafka ingester can read Kafka streams into memory and store them to disk.
• Key and Value specifications are disjoint, which is an improvement over the io.deephaven.kafka.ingest.ConsumerRecordToTableWriterAdapter pattern found in Enterprise.
• The Community KafkaIngester uses chunks for improved efficiency compared to row-oriented Enterprise adapters.

You can now use the Community Kafka ingester together with an in-worker ingestion server in a DnD worker. As with the existing Enterprise Kafka ingestion, you must create a schema and create a data import server within your data routing configuration. After creating the schema and DIS configuration, create an ingestion script using a Community worker.

You must create a KafkaConsumer Properties object. Persistent ingestion requires that auto commit is disabled in order to ensure exactly once delivery. The next step is creating an Options builder object for the ingestion and passing it to the KafkaTableWriter.consumeToDis function. You can retrieve the table in the same query, or from any other query according to your data routing configuration.

import io.deephaven.kafka.KafkaTools
import io.deephaven.enterprise.kafkawriter.KafkaTableWriter

final Properties props = new Properties()
props.put('bootstrap.servers', 'http://kafka-broker:9092')
props.put('schema.registry.url', 'http://kafka-broker:8081')
props.put("fetch.min.bytes", "65000")
props.put("fetch.max.wait.ms", "200")
props.put("deephaven.key.column.name", "Key")
props.put("deephaven.key.column.type", "long")
props.put("enable.auto.commit", "false")
props.put("group.id", "dis1")

final KafkaTableWriter.Options opts = new io.deephaven.enterprise.kafkawriter.KafkaTableWriter.Options()
opts.disName("KafkaCommunity")
opts.tableName("Table").namespace("Namespace").partitionValue(today())
opts.topic("demo-topic")
opts.kafkaProperties(props)
opts.keySpec(io.deephaven.kafka.KafkaTools.FROM_PROPERTIES)
opts.valueSpec(io.deephaven.kafka.KafkaTools.Consume.avroSpec("demo-value"))

KafkaTableWriter.consumeToDis(opts)

ingestedTable=db.liveTable("Namespace", "Table").where("Date=today()")