Detailed Version Log Deephaven 1.20240517

Certified versions

Detailed Version Log: Deephaven v1.20240517

Detailed Release Candidate Version Log: Deephaven v1.20231219beta

Automatic Server Selection

Automated server-selection is now turned on by default on new non-Kubernetes installations. To turn it off or change the defaults, edit your iris-environment.prop file and remove or edit the properties from the iris_controller stanza, as described in automated server selection.

Changes to tailer process memory requirements

Tailer now uses constrained and pre-allocated buffer pools for both User and System tables

With default configuration, the tailer now uses constrained and pre-allocated buffer pools for sending data to Data Import Servers. This makes the tailer's memory consumption more predictable, and avoids potential out-of-memory conditions. There are separate pools for User and System tables, governed by the following properties:

The tailer allocates two pools of buffers, one for user tables and one for system tables. Each item in that pool requires two buffers for concurrency, so the memory required will be double the buffer size times the pool capacity.

Total direct memory required for the tailer is approximately 2 * (DataContent.producerBufferSize * DataContent.systemPoolCapacity + DataContent.producerBufferSize.user * DataContent.userPoolCapacity).

Tailer now adjusts to DIS buffer size

When the tailer establishes a connection to a DIS, the processes now exchange configuration information. If the tailer is configured with a larger maximum message size than the DIS can accept, it will reduce it's maximum message size to match the DIS.

Change configuration to handle older tailers sending data to newer Data Import Servers

This section applies only to environments using default configuration values, and where tailers are outside the Deephaven system, either tailing from external systems or from different Deephaven installations. In this scenario, the tailer may send messages that are too large for the DIS to accept.

Older tailers do not accommodate to DIS settings, and may have default configuration settings that allow them to send larger messages than a DIS with default configuration can accept. This typically only happens when the tailer gets behind, or otherwise has file to process with more than 327,710 bytes.

When this happens, the DIS will reject the tailer with a message like:

WARN - DataImportServer-db_dis-TailerConnection-/127.0.0.1:55638-FullTableLocationKey{TableLookupKey.Immutable[Namespace/table/U],TableLocationLookupKey.Immutable[internal_partition/date]}:Rejecting and closing channel: Protocol error while processing stream: Message size 4400021 exceeds the maximum 327710. The client is configured with a larger maximum message size than this server. Check parameters DataContent.producerBufferSize and DataContent.consumerBufferSize. This most likely applies to the DIS and tailer processes.

A similar message might appear in the tailer log.

When an old tailer sends data to a new DIS, the DIS will log a warning like this:

WARN - Received TableIdentifierDataItem with old version: 4. The tailer will not protect against causing a read buffer overflow.

If your environment includes older tailers that send data to current DISes, you should make one of the following configuration changes:

1. update the tailers to the current Deephaven version
2. update the tailer configurations to set the max message size, by setting the following properties in the tailer configuration or properties file:

DataContent.producerBufferSize = 262144 
DataContent.producerBufferSize.user = 262144 

3. update the DIS max message size to match the tailer max message size by setting the following properties in the DIS configuration or properties file:

DataContent.producerBufferSize = 2097160 

Database.inputTableUpdater improvements

The Database.inputTableUpdater method has been updated to return a new, more general interface io.deephaven.enterprise.database.inputtables.InputTableUpdater, which allows callers to have more control over table lineage with input tables. The InputTableUpdater can also be explicitly managed with a liveness scope, and is also now more efficient in caching intermediate operations.

When sourcing both an InputTableUpdater and an input Table view from db, the resulting objects may not share a common lineage:

def myTableUpdater = db.inputTableUpdater("MyNamespace", "MyTable")
def myTable = db.inputTable("MyNamespace", "MyTable")

myTableUpdater.add(new_data)
// myTable is not guaranteed to have new_data

With the new interface, callers can explicitly derive the corresponding input Table view from the InputTableUpdater:

def myTableUpdater = db.inputTableUpdater("MyNamespace", "MyTable")
def myTable = myTableUpdater.table()

myTableUpdater.add(new_data)
// myTable is guaranteed to have new_data

Note, the python layer does not currently expose a stand-alone equivalent of InputTableUpdater; an input Table view must be created, and in that context, table lineage is already guaranteed.

my_table = db.input_table("MyNamespace", "MyTable")
my_table.add(new_data)
# my_table is guaranteed to have new_data

Breaking API change for Core+ C++ Client

We have changed the map type used by deephaven_enterprise::controller::Subscription. This map type is defined in the typedef deephaven_enterprise::controller::Subscription::map_type and it is used in an out parameter of deephaven_enterprise::controller::Subscription::Current().

The important user-visible change is that the map's find() operation now returns a const iterator that interoperates with its begin() and end() operations. This makes it behave more like C++'s familiar std::map container. Previously, our find() operation returned a pointer to the mapped type rather than an iterator.

The following example code shows the new usage of find.

Assuming this code:

    std::int64_t version;
    Subscription::map_type map;
    if (!sub.Current(&version, &map)) {
      std::cout << "Subscription closed\n";
      return;
    }

Old API:

    const auto *pq_info_ptr = map.find(pq_serial);
    if (pq_info_ptr == nullptr) {
      std::cout << "pq_serial not found\n";
    }

    auto pq_info = *pq_info_ptr;

New API: find() now returns an iterator

    auto it = map.find(pq_serial);
    if (it == map.end()) {
      std::cout << "pq_serial not found\n";
    }
    
    auto pq_info = it->second;

Add dhconfig properties delete command

The dhconfig properties now supports a delete command, which can be used to delete a properties file from etcd.

usage: dhconfig properties delete [--configfile <arg>] [--diskprops] [--etcd] [-f <arg>] [-h] [-k <arg> | -user <arg>]
       [-pf <arg>]  [-v]
Delete properties files from the system.

Options are as follows:
    --configfile <arg>   use the named properties file instead of the default
    --diskprops          read properties from the classpath, instead of etcd or the configuration service
    --etcd               use etcd directly, instead of configuration service (when combined with --diskprops, property
                         files are read from disk but written to etcd)
 -f,--file <arg>         specify the properties files to delete
 -h,--help               print help for a properties command
 -k,--key <arg>          specify a private key file to use for authentication
 -pf,--pwfile <arg>      specify a file containing the base64 encoded password for the user that is set with --user
 -user,--user <arg>      specify a user for authentication
 -v,--verbose            print additional logging, progress messages, and full exception text

Examples:
Delete my-prop-file.prop:
    dhconfig properties delete --file my-prop-file.prop

Enable Systemic object tracking for Core+ workers

Systemic Object tracking has been enabled by default for Core+ workers. It increases the robustness of Persistent Queries by separating systematically important objects from user created ones, reducing the chance that misbehaving user requests cause outages. This default aligns Core+ worker behavior with Legacy worker behavior

Core+ updated to 0.37.0

The Core+ integration has been updated to Core version 0.37.0. As part of this update the Core+ Java Client libraries no longer support Java 8. Supported versions are now 11 and 17.

Python 3.8 is the oldest supported Python version

Even though Python 3.8 has already reached EOL, on some versions of Deephaven, this is the newest built + tested version of Python.

As of Bard version 1.20211129.426, Python 3.8 is the only Python version built, and iris-defaults.prop changes the default from Python 3.6 to 3.8.

If you still have virtual environments setup with Python 3.6 or 3.7, you should replace them with Python 3.8 venvs. To use newer versions of Python, upgrade to a newer version of Deephaven.

For legacy systems, you can change the default back to Python 3.6 by updating your iris-environment.prop to set the various jpy.* props to the values found in iris-defaults.prop, inside the jpy.env=python36 stanza:

# Legacy python3.6 locations:
jpy.programName=/db/VEnvs/python36/bin/python3.6
jpy.pythonLib=/usr/lib64/libpython3.6m.so.1.0
jpy.jpyLib=/db/VEnvs/python36/lib/python3.6/site-packages/jpy.cpython-36m-x86_64-linux-gnu.so
jpy.jdlLib=/db/VEnvs/python36/lib/python3.6/site-packages/jdl.cpython-36m-x86_64-linux-gnu.so

The new iris-defaults.prop python props are now:

# New iris-defaults.prop python3.8 locations:
jpy.programName=/db/VEnvs/python38/bin/python3.8
jpy.pythonLib=/usr/lib/libpython3.8.so
jpy.jpyLib=/db/VEnvs/python38/lib/python3.8/site-packages/jpy.cpython-38-x86_64-linux-gnu.so
jpy.jdlLib=/db/VEnvs/python38/lib/python3.8/site-packages/jdl.cpython-38-x86_64-linux-gnu.so

Changes to Barrage subscriptions in Core+ Python workers

The methods subscribe and snapshotTable inside deephaven_enterprise.remote_table have been changed to return a Python deephaven.table.Table object instead of a Java io.deephaven.engine.table.Table object. This allows users to use the Python methods update_view, rename_columns, etc. as expected without wrapping the returned table.

Existing Python code that manually wrapped the table or directly called the wrapped Java methods must be updated.

Example of previous behavior:

from deephaven_enterprise import remote_table as rt

table = rt.in_local_cluster(query_name="SubscribePQ", table_name="my_table").snapshot()
table = table.updateView("NewCol = random()")

Example of new behavior:

from deephaven_enterprise import remote_table as rt

table = rt.in_local_cluster(query_name="SubscribePQ", table_name="my_table").snapshot()
table = table.update_view("NewCol = random()")

Upgrading from releases before Grizzly 1.20240517.273

If this is your first upgrade from a release older than Grizzly 1.20240517.273, you must make some manual changes to your iris-environment.prop file before running helm upgrade or the dh_helm script. These changes are not automated because this file includes changes made by customers that should not be overwritten.

Create a bash session on the management shell. For example:

kubectl exec -it deploy/management-shell -- bash

Export and edit iris-environment.prop:

/usr/illumon/latest/bin/dhconfig properties export -f iris-environment.prop -d /tmp/
vi /tmp/iris-environment.prop

Remove these entries entirely. Note that these entries are not grouped together in the file, and some of them may not be present depending on what version you are upgrading from. If they are present, the first three are likely to be within the first 50 lines of your file, the two final properties are likely to be found within a scoping block about halfway down, and the remaining entries would be found toward the end of the file:

Kubernetes.chart.name={{ .Chart.Name }}
Kubernetes.release.name={{ .Release.Name }}
Kubernetes.release.namespace={{ .Release.Namespace }}

final configuration.reload.userGroups=superuser
final PersistentQueryController.keyPairFile=/etc/sysconfig/deephaven/syskey/priv-controllerConsole.base64.txt

RemoteQueryDispatcher.workerStartupTimeoutMS=240000
RemoteQueryDispatcher.workerControlType=Kubernetes
EmbeddedDbConsole.remoteDatabaseRequestTimeoutMS=60000

Kubernetes.deployment=true
Kubernetes.start-worker-timeout-seconds=240
Kubernetes.query-worker-k8s-template=/configs/k8s-query-worker-template.yaml
Kubernetes.start-worker-dnd-timeout-seconds=240
Kubernetes.query-worker-dnd-k8s-template=/configs/k8s-query-worker-coreplus-template.yaml

Webapi.server.cus.enabled=true
Webapi.server.cus.home=/etc/sysconfig/deephaven/cus
[service.name=web_api_service] {
    BusinessCalendar.storeRawData=true
}

RemoteProcessingRequestProfile.Xms.G1 GC=$RequestedHeap
RemoteQueryDispatcher.JVMParameters=-XX:+AlwaysPreTouch
BinaryStoreWriterV2.allocateDirect=false

authentication.server.localsuperusers.file=/etc/sysconfig/deephaven/superusers.txt

Import the modified file:

/usr/illumon/latest/bin/dhconfig properties import -f /tmp/iris-environment.prop

You are now ready to run the helm upgrade command or the dh_helm script to perform the upgrade.

Managed User Authentication flag is required for local password authentication

When the property iris.enableManagedUserAuthentication is set to true, user passwords may be stored in the ACL store (etcd or MySQL). In previous versions of Deephaven, if this was set to false existing passwords could still be used for authentication. When the property is set to false, the authentication server now rejects the password authentication. If you have changed this property to false from the default value of true, users must have alternative means of authenticating to the system (e.g., SAML or Active Directory) to use Deephaven with a password.

Added support for Iceberg tables in Core+

Core+ workers now support reading Iceberg tables as historical tables. To configure Iceberg, you must configure an Iceberg Endpoint, then discover and deploy a schema.

Configuring an IcebergEndpoint

The first step to link Iceberg tables into Deephaven is configuring an IcebergEndpoint, which we refer to as simply an endpoint. The endpoint contains the parameters required to locate and connect to the Iceberg catalog, the data warehouse, and the storage-specific parameters required to read the data. An endpoint is configured using the example below.

import io.deephaven.enterprise.iceberg.IcebergTools
import io.deephaven.extensions.s3.S3Instructions

// Create a new endpoint
endpoint = IcebergTools.newEndpoint()
        .catalogType("rest")                                 // The catalog is a REST catalog
        .catalogUri("http://mydata.com:8181")                // located at this URI. 
        .warehouseUri("s3://warehouse/")                     // The data warehouse is an S3 warehouse at this URI
        .putProperties("s3.access-key-id", "my_access_key",  // These are the properties required by the Iceberg API.
                       "client.region"   , "us-east-1")      // See https://iceberg.apache.org/docs/nightly/configuration/#configuration
        .putSecrets("s3.secret-access-key", "s3.key")        // Include any named secrets
        .dataInstructions(S3Instructions.builder()           // Configure the S3 data parameters
                .regionName("us-east-1")
                .build())
        .build("my_company_iceberg");                        // Explicitly name the endpoint.

from deephaven_enterprise import iceberg
from deephaven.experimental import s3

# Create the data instructions for reading data.
s3i = s3.S3Instructions(region_name="us-east-1")

# Create a new endpoint
endpoint = iceberg.make_endpoint("rest", \          # The catalog is a REST catalog
              "http://mydata.com:8181", \            # Located at this URI
              "s3://warehouse/", \                   # The data warehouse is an S3 warehouse at this URI
              s3i, \                                 # Attach the data instructions
              endpoint_name="my_company_iceberg", \  # Explicitly name this endpoint
              properties={"s3.access-key-id" : "my_access_key", "client.region" : "us-east-1"}, # Set Iceberg configuration properties. See https://iceberg.apache.org/docs/nightly/configuration/#configuration  
              secrets={"s3.secret-access-key" : "s3.key"})  # Include any named secrets

Properties

The properties component of the endpoint is a key-value map of Iceberg configuration parameters to their values. Valid property keys can be found in the Iceberg documentation at https://iceberg.apache.org/docs/nightly/configuration/#configuration.

Secrets

The secrets component is also a key, value map where the keys are Iceberg configuration properties, and the values are named references to secrets stored within Deephaven so you do not need to include secrets in script text. When needed, the secrets are retrieved from Deephaven and merged into the properties before being used to access Iceberg. Secrets may be stored either in the Deephaven configuration file as a property or as a JSON map in a protected file on disk. More sophisticated secret stores are possible. Contact support for more information. Secrets providers are visited in ascending priority order until one supplies a value or none can be found.

From Properties

Secrets may be stored in Deephaven configuration files as a simple property; for example, s3.access_key=1234secret4321. The default priority of the Properties secrets provider is 100 and can be adjusted using the property PropertiesSecretsProvider.priority.

From Files

Secrets may be stored in files on disk containing a simple JSON map. This format is more secure and better supports more complex secret values. You may configure multiple secrets files and their priorities using these properties:

You may provide as many of these as you need, ensuring that each name is unique.

An example file:

{
  "s3.key" : "some_secret_key",
  "secret_url" : "https://verysecret.com:9001",
  "complicated" : "<Secret type=\"important\"><Internal>Secrecy</Internal></Secret>"
}

Deployment

The endpoint can be deployed to Deephaven configuration as long as a name has been provided. Once deployed, you may reference the endpoint by name in the schema for Iceberg tables to avoid duplication.

// Deploy the endpoint to Deephaven configuration, failing if it already exists.
endpoint.deploy(false)

# Deploy the endpoint to Deephaven configuration, Overwriting if it already exists.
# The overwrite_existing parameter defaults to False. The deployment will fail if the endpoint already exists.
endpoint.deploy(overwrite_existing=True)

Discovering an Iceberg table

Once an endpoint has been configured, you can discover an Iceberg endpoint to create and deploy a Deephaven schema. If you have previously configured and deployed an endpoint, you can retrieve it by name as well.

import io.deephaven.enterprise.iceberg.IcebergTools

// Load an endpoint that was already configured
endpoint = IcebergTools.getEndpointByName("my_company_iceberg")

// Discover a table derive the schema and deploy it, deriving the namespace and table name
// from the table identifier, referencing the endpoint by name.
discovery = IcebergTools.discover(DiscoveryConfig.builder()
                .tableIdentifier("market.trades")
                .endpoint(endpoint)
                .build())

discovery.deployWithEndpointReference()

from deephaven_enterprise import iceberg

# Load an endpoint that was already configured
endpoint = iceberg.get_named_endpoint("my_company_iceberg")

# Discover a table derive the schema and deploy it, deriving the namespace and table name
# from the table identifier, referencing the endpoint by name.
result = iceberg.discover("market.trades", endpoint)
result.deploy_named()

In the examples above, the Deephaven namespace and table name are derived directly from the Iceberg table identifier. You may specify your own by setting the namespace and tableName properties during discovery.

discovery = IcebergTools.discover(DiscoveryConfig.builder()
                .tableIdentifier("market.trades")
                .namespace("MarketUS")
                .tableName("EqTrades")
                .endpoint(endpoint)
                .build())

result = iceberg.discover(
    "market.trades", namespace="MarketUS", table_name="EqTrades", endpoint
)

Complete Examples

Below are complete examples that create an endpoint, discover a table, deploy a schema, and then fetch the table.

import io.deephaven.enterprise.iceberg.IcebergTools
import io.deephaven.enterprise.iceberg.discovery.DiscoveryConfig
import io.deephaven.extensions.s3.S3Instructions

// Create a new endpoint
endpoint = IcebergTools.newEndpoint()
        .catalogType("rest")                                 
        .catalogUri("http://mydata.com:8181")                 
        .warehouseUri("s3://warehouse/")                     
        .putProperties("s3.access-key-id", "access_key",
                       "client.region" , "us-east-1")
        .putSecrets("s3.secret-access-key", "s3.key")
        .dataInstructions(S3Instructions.builder()
                .regionName("us-east-1")
                .build())
        .build("my_company_iceberg");

endpoint.deploy(true)

discovery = IcebergTools.discover(DiscoveryConfig.builder()
                .tableIdentifier("market.trades")
                .endpoint(endpoint)
                .build())

data = db.historicalTable("market", "trades")

from deephaven_enterprise import iceberg
from deephaven.experimental import s3

s3i = s3.S3Instructions(region_name="us-east-1")

endpoint = iceberg.make_endpoint(
    "rest",
    "http://mydata.com:8181",
    "s3://warehouse/",
    s3i,
    endpoint_name="my_company_iceberg",
    properties={"s3.access-key-id": "access_key", "client.region": "us-east-1"},
    secrets={"s3.secret-access-key": "s3.key"},
)
endpoint.deploy(True)

result = iceberg.discover("market.trades", endpoint)
result.deploy_named()

data = db.historical_table("market", "trades")

Tailer Pool for System Tables

The tailer now uses data content pools for system tables, making the system table behavior controlled in a similar fashion to user table memory parameters. Before this change, a tailer servicing many system table partitions, could consume unbounded memory. The configuration properties that control the pool size are:

The data buffers are stored in either direct memory or heap memory, depending on the value of the DataContent.producersUseDirectBuffers property. Direct memory is recommended, because when passed an on-heap buffer the JVM allocates a new direct buffer while reading data from the file. If you adjust the size of the pool, then you should also adjust the tailer's maximum heap size or maximum memory size. The size of each buffer defaults to 256KB, but can be controlled by the DataContent.producerBufferSize property.

Vermilion+ Core+ updated to 0.35.2

Vermilion+ 1.20231218.440 includes version 0.35.2 of the Deephaven Core engine. This is the same version that ships with Grizzly in 1.20240517.189, enabling customers to have one Core engine version of overlap between major Deephaven Enterprise releases. Although the Core engine functionality is the same in 0.35.2, the Grizzly Core+ worker has several enhancements that are not available in the Vermilion+ Core+ worker. This change also updates grpc to 1.61.0

For details on the Core changes, see the following release notes:

• Release notes for Deephaven Core version 0.35
• Release notes for Deephaven Core version 0.34

Changes to vector support for Core+ user tables

Both the Legacy and Core engines have special database types to represent arrays of values. The Legacy engine uses the DbArray class, while the Core system uses the Vector class. While these implementations represent identical data, they pose challenges for interoperability between workers running different engines.

When a user table is written, the schema is inferred from the source table. Previously, Vectors would be recorded verbatim in the schema. This change explicitly encodes Vector types as their base java array types as follows.

This makes it possible for the Legacy engine to read User tables written by the Core engine. Note that no conversion is made when the Legacy engine writes DbArray types because the Core+ engine already supports those types.

If you want your User table array columns to be Vector types, use an .update() or .updateView() clause to wrap the native arrays.

staticUserTable = db.historicalTable("MyNamespace", "MyTable")
                    .update("Longs = (io.deephaven.vector.LongVector)io.deephaven.vector.VectorFactory.Long.vectorWrap(Longs)")

Option close Tailer-DIS connections early, while continuing to monitor files

A new property is available to customize the behavior of the Tailer.

log.tailer.defaultIdlePauseTime

This property is similar to log.tailer.defaultIdleTime, but it allows the Tailer to close connections early while continuing to monitor files. When the idle time specified by log.tailer.defaultIdleTime has passed without any changes to a monitored file, the Tailer will close the corresponding connection to the DIS, and will not process any further changes to the file. The default idle time must therefore be as long as the default file rollover interval plus some buffer.

The new property enables a new feature. When the time specified by log.tailer.defaultIdlePauseTime has passed without any changes to a monitored file, the Tailer will close the corresponding connection to the DIS, but will continue to monitor the file for changes. If a change is detected, the Tailer will reopen the connection and process the changes. This will reduce or more quickly reclaim resources consumed for certain usage patterns.

Helm Chart Tolerations, Node Selectors and Affinity

You can now add tolerations, node selection, and affinity attributes to pods created by the Deephaven Helm chart. By default, no tolerations, selectors or affinity are added. To add tolerations to all created deployments, modify your values.yaml file to include a tolerations block, which is then copied into each pod. For example:

tolerations:
- key: "foo"
  operator: "Exists"
  effect: "NoSchedule"
- key: "bar"
  value: "baz"
  operator: "Equal"
  effect: "NoSchedule"

Adds the following tolerations to each pod (in addition to the default tolerations provided by the Kubernetes system):

Tolerations:                 bar=baz:NoSchedule
                             foo:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Similarly, you can add a nodeSelector or affinity block:

nodeSelector:
- key: "foo"
  operator: "Exists"
  effect: "NoSchedule"
- key: "bar"
  value: "baz"
  operator: "Equal"
  effect: "NoSchedule"

affinity:
  nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: label
            operator: In
            values:
            - value1

Which result in pods containing node selectors like:

Node-Selectors:              key1=value1
                             key2=value2

And affinity as follows:

  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: label
            operator: In
            values:
            - value1
        weight: 1

Ability to disable password authentication in front-end

A new property, authentication.client.disablePasswordAuth=true, may be used to remove the username/password authentication option from the Swing front-end. The property has no effect if there are no other login-options available.

This property does not disable username/password authentication at the server level (see Disabling password authentication).

Allow config to override ServiceRegistry hostname

The hostname which Data Import Server (DIS) registers with the service registry may now be defined in the host tag within the DIS' routing endpoint of the routing configuration; or using the new ServiceRegistry.overrideHostname system property. The precedence for the service registry host is from:

• The routing endpoint configuration. Prior to this change, the host value within the endpoint configuration was ignored.
• ServiceRegistry.overrideHostname property.
• On Kubernetes, the worker's service's hostname.
• On bare metal, it is the result of the Java InetAddress.getLocalHost().getHostName() function.

Selection provider marking a query server as down

You can temporarily remove a query server from the list of servers that the controller uses for the Auto_Query and Auto_Merge groups. This allows the administrator to prevent new queries or consoles from being automatically assigned to that query server. The query server can still be selected manually and existing running queries are not evicted. This can be useful, for example, if a query server is malfunctioning, and you would like to remove it from the rotation, but still have access to it for debugging. To mark Query_1 as administratively down, use dhconfig pq selection-provider:

dhconfig pq selection-provider --command down --node Query_1

To mark the server as up:

dhconfig pq selection-provider --command up --node Query_1

The down state of a server is not persisted; on controller restart, all servers are marked as up. To permanently remove a server, you should re-run the Deephaven installer or edit the iris-endpoints.prop file and reload the controller.

Note: these commands are only relevant if you have the default SimpleServerSelectionProvider enabled.

Server Selection Provider Backoff

If an individual server fails to acquire workers, it often becomes the least-loaded server, resulting in the SimpleServerSelectionProvider assigning all new workers to that server. A new backoff policy has been introduced to prevent immediately assigning a worker to a server with an acquisition failure. Before assigning another worker to that server, the selection provider ensures that each of the other servers has had a worker assigned to them. Once a worker is successfully assigned to the query server, the state is cleared.

This new behavior prevents the algorithm from getting "stuck" assigning workers to the failed server but does periodically attempt to assign workers to handle transient failures or misconfigured queries.

To enable the new backoff behavior, set the configuration property:

SimpleServerSelectionProvider.FailureBackoffPolicy=ALL_OTHERS_AFTER_ACQUISITION_FAILURE

The default behavior remains the same as previous versions. To explicitly set this behavior, set the property to NONE:

SimpleServerSelectionProvider.FailureBackoffPolicy=NONE

Active Directory Group Synchronization

As with SAML, you can now synchronize groups from an Active Directory server to your Deephaven installation.

After a user authenticates with the Active Directory server, if group synchronization is enabled, the Authentication Server examines the memberOf attribute of the user's record. If the groups from Active Directory do not match the user's Deephaven groups, the relevant group names are added to or removed from the ACL store using the ACL write server.

The following properties control the behavior of group synchronization:

Column ACL Normalization

In previous versions, column ACLs were not normalized, meaning a system could contain column ACL keys that only differed by column order or duplicates, e.g.:

Group| Namespace|      TableName|   Columns|                                          Filter
----------+----------+---------------+----------+------------------------------------------------
test      |DbInternal|ProcessEventLog|b1        |new UsernameFilterGenerator("EffectiveUser")
test      |DbInternal|ProcessEventLog|b1,b1,b1  |new UsernameFilterGenerator("AuthenticatedUser")
test      |DbInternal|ProcessEventLog|b1,b2,b3  |new UsernameFilterGenerator("EffectiveUser")
test      |DbInternal|ProcessEventLog|b3,b2,b1  |new UsernameFilterGenerator("AuthenticatedUser")

Column specifications "b1" and "b1,b1,b1" refer to the same columns, as do "b1,b2,b3" and "b3,b2,b1". This is problematic, particularly when effectively equivalent column ACL keys have different filter values, as shown above. In this case, behavior is undefined.

The ACL write server attempts to normalize column ACLs on startup automatically. However, if there are effectively equivalent column ACLs with different filter values, automatic normalization fails with a report of the problematic column ACLs in the ACL write server logs. At this point, the Deephaven administrator must:

1. Set property DbAclWriteServer.startup.normalizeColumnAcls=false.
2. Restart the ACL write server.
3. Remove problematic column ACLs.
4. Set property DbAclWriteServer.startup.normalizeColumnAcls=true.
5. Restart the ACL write server.

Continuing to run with de-normalized column ACLs is not recommended.

As always, before a software upgrade, you should backup your ACL database.

New Persistent Query range scheduling

A new Range scheduling option allows date/time ranges. A single start date/time and a single stop date/time can be specified.

• If the start date/time is not specified, the Persistent Query is started immediately.
• If the stop date/time is not specified, the Persistent Query is not stopped once it is started.

Removal of dhconfig base64 password options

The dhconfig tool no longer accepts the --password option for plaintext passwords to help reduce the chance of leaking secrets into shell history or log files. Interactive password authentication is still supported. Private key authentication should be used for non-interactive operations.

Managing users, groups, and ACLs with dhconfig

The dhconfig tool can now manage users, groups and ACLS using the subcommand acls. For example, you can add users to groups with:

dhconfig acl groups add-member --name user1 user2 --group groupA groupB

Complete help is available by running dhconfig acls --help. This example demonstrates one of the improvements enabled by using dhconfig; the old iris_db_user_mod tool cannot add multiple users to multiple groups in a single command.

All existing operations that were performed by /usr/illumon/latest/bin/iris iris_db_user_mod have been integrated into dhconfig, and the old iris_db_user_mod tool will be removed in a future release.

Run dhconfig acls --help for more detailed help. Each operation provides additional help information for its specific arguments; for example: dhconfig acls user --help provides help for user management commands.

DataImportServers respond to data routing changes

DataImportServers now respond to data routing changes. If the data routing configuration is changed (via the dhconfig routing or dhconfig dis commands), the DIS receives a notification of the change. The only change it can respond to is a change in the filters, either explicit changes to the filter or changes implied by claims made by this or other DISes. If any other configuration change is detected, the DIS prints error messages indicating that the configuration is out of date until it is restarted.

Disable changes (other than logging) with property:

DataImportServer.ignoreRoutingConfigChanges=true

DIS interface changes may require script updates

The DataImportServer interface changes may require script changes. Please see the sections below.

Deprecated methods

The following deprecated methods in com.illumon.iris.db.tables.dataimport.logtailer.DataImportServer have been removed:

DataImportServer(
    com.fishlib.io.logger.Logger,
    com.illumon.iris.db.v2.routing.DataImportServiceConfig,
    com.fishlib.configuration.Configuration)

DataImportServer.getDataImportServer(
    com.fishlib.io.logger.Logger,
    com.illumon.iris.db.v2.routing.DataImportServiceConfig,
    com.fishlib.configuration.Configuration,
    com.illumon.iris.db.schema.SchemaService)

DataImportServer.start()

Instead, use DataImportServer.startInWorker() or DataImportServer.start(StatsIntradayLogger.NULL)

The following methods are newly deprecated to support this functionality change and will be removed in the next release:

DataImportServer(
    com.illumon.iris.db.util.logging.DataImportServerLogFactory,
    com.illumon.iris.db.v2.routing.DataImportServiceConfig,
    com.fishlib.configuration.Configuration,
    com.illumon.iris.db.schema.SchemaService)

DataImportServer.getDataImportServer(
    com.illumon.iris.db.util.logging.DataImportServerLogFactory,
    com.illumon.iris.db.v2.routing.DataImportServiceConfig,
    com.fishlib.configuration.Configuration,
    com.illumon.iris.db.schema.SchemaService)

The replacement methods for constructing a DataImportServer are:

    /**
     * Construct a {@link DataImportServer}.
     *
     * @param logFactory     optional IrisLogCreator for access to loggers
     * @param disName        the name of the DIS configuration in the data routing service
     * @param configuration  the configuration to use when fetching settings that aren't included in disConfig
     * @param schemaService  the SchemaService to use
     * @param routingService the DataRoutingService to use for configuration
     * @param storageRoot    optional storage root to use if the DIS configuration specifies "private"
     */
    public DataImportServer(@Nullable DataImportServerLogFactory logFactory,
                            @NotNull final String disName,
                            @NotNull final Configuration configuration,
                            @NotNull final SchemaService schemaService,
                            @NotNull final DataRoutingService routingService,
                            @Nullable final String storageRoot);

    /**
     * Create a new DataImportServer instance, according to the configuration passed in.
     *
     * @param logCreatorParam optional IrisLogCreator, will be used to create logger. If null, a global instance will be used, if available.
     * @param disName         the name of the DIS configuration to use
     * @param configuration   the configuration to use when fetching settings that aren't included in disConfig
     * @param schemaService   the SchemaService to use
     * @param routingService  the DataRoutingService to use for configuration
     * @param storageRoot     optional storage root to use when configured storage is "private"
     * @return a new DataImportServer instance
     */
    public static DataImportServer getDataImportServer(@Nullable final DataImportServerLogFactory logCreatorParam,
                                                       @NotNull final String disName,
                                                       @NotNull final Configuration configuration,
                                                       @NotNull final SchemaService schemaService,
                                                       @NotNull final DataRoutingService routingService,
                                                       @Nullable final String storageRoot);

Script changes

Make the following changes to your scripts:

The new methods are more explicit about the data storage location and need a DataRoutingService to respond dynamically to data routing configuration changes.

• Instead of getting a DataImportServiceConfig from the DataRoutingService, pass the DataRoutingService and configuration name as parameters.
• Specify a storage location or leave it as null if the storage is specified in the DIS configuration.

This example shows the required changes from a previously documented example.

Before:

routingService = DataRoutingServiceFactory.getDefault()

// if the dis configuration specifies "private" storage, then you must use getDataImportServiceConfigWithStorage
disConfig = routingService.getDataImportServiceConfigWithStorage("Ingester1", "/db/dataImportServers/Ingester1")
// this call assumes named storage configured in the routing file
//disConfig = routingService.getDataImportServiceConfig("Ingester1")
dis = DataImportServer.getDataImportServer(ProcessEnvironment.getDefaultLog(), disConfig, Configuration.getInstance(), SchemaServiceFactory.getDefault())
dis.startInWorker()

After:

routingService = DataRoutingServiceFactory.getDefault()

// if the dis configuration specifies "private" storage, then you must provide a storage root, otherwise set this to null
storage = "/db/dataImportServers/Ingester1"
disName = "Ingester1"

dis = DataImportServer.getDataImportServer(
        null, 
        disName, 
        Configuration.getInstance(),
        SchemaServiceFactory.getDefault(),
        routingService,
        storage)
dis.startInWorker()

Alternate launch option for Deephaven Core+ local workers

Deephaven launches local worker processes by creating a command using various system configurations and user-specified inputs, such as additional classpath or memory settings, and executes that process directly. With this release, it is possible to configure Core+ workers so that the command is supplied to another process that can launch it. As an example, this would allow for configuring the system to run workers that are authenticated with a Kerberos keytab.

The default Core+ worker is launched with this command:

$ /usr/illumon/latest/bin/deephaven-coreplus ..

If your configuration provides a commandPrefix for your Core+ worker kind, then the default command will be provided to the command you specify as the prefix.

As an example, this configuration setting:

WorkerKind.DeephavenCommunity.commandPrefix=k5start -f my-krb5.keytab -U -- /bin/sh -c

Results in the workers being launched like this:

$ k5start -f my-krb5.keytab -U -- /bin/sh -c '/usr/illumon/latest/bin/deephaven-coreplus ..'

By default, the commandPrefix is tokenized by spaces. If you require a different delimiter, you can define it with this property:

# Use a comma instead of the default space
WorkerKind.DeephavenCommunity.commandPrefixDelimiter=,

Optional lenient IOJobImpl to avoid write queue overflow

New behavior is available to avoid write queue overflow errors in the TDCP process. When a write queue overflow condition is detected, the process can be configured to delay briefly - giving the queue a chance to drain.

The following properties govern the feature:

IOJobImpl.lenientWriteQueue
IOJobImpl.lenientWriteQueue.retryDelay
IOJobImpl.lenientWriteQueue.maxDelay

Set IOJobImpl.lenientWriteQueue=true to enable this behavior. By default, the writer will wait up to IOJobImpl.lenientWriteQueue.maxDelay=60_000 ms in increments of IOJobImpl.lenientWriteQueue.retryDelay=100 ms.

This should address the following fatal error in the TDCP process:

ERROR - job:1424444133/RemoteTableDataService/10.128.1.75:37440->10.128.1.75:22015 write queue overflow: r=true, w=true, p=false, s=false, u=false, h=0, rcap=69632, rbyt=0, rmax=4259840, wbyt=315407, wspc=1048832, wbuf=4097, wmax=1048576, fc=0, allowFlush=true

Persistent Query Replicas and Spares

Persistent Queries now support redundancy and automatic failover with the replicas and spares feature. Any Live Persistent Query can now specify a number of Replica and Spare workers to start. Replica workers are identical copies of the query that the Controller distributes user load across. Spares are identical copies used by the controller to replace failed replicas. When more than one replica is running, the replica workers are independent from each other. Replica queries are suitable for UI interactions, but care must be taken that the replicas do not perform actions that would be unsafe to concurrently perform without coordination.

Administrators of a query (including members of the iris-querymanagers or iris-superusers group) see the state of all replicas in the Query Monitor. A viewer only sees the replica that they are assigned to in the Query Monitor.

Load Balancing

When a query is configured with more than one replica, the controller uses a configurable policy called an assignment policy to distribute users across all available workers. Each Persistent Query Configuration has a number of "slots" equal to the number of replicas. An assignment policy, which is sticky to prevent a single user's UI-driven actions from negatively impacting more than one replica, assigns a user to a specific slot.

By default, Deephaven provides a single "Round Robin" assignment policy that distributes users evenly across the available workers. Customers may implement the io.deephaven.enterprise.controller.assignment.AssignmentPolicy interface to provide custom assignment policies.

Assignment policies are configured by adding values to your property file as follows:

PersistentQueryController.AssignmentPolicy.PolicyName.class=com.company.MyPolicyClass
PersistentQueryController.AssignmentPolicy.PolicyName.displayName=A Display Name
PersistentQueryController.AssignmentPolicy.PolicyName.description=Describes the policy

The assignment policy receives two callbacks whenever the number of replicas change. The first callback occurs before any currently running replicas are shut down so that users can be reassigned. The second callback occurs after the changes are applied so that user load can be redistributed.

Automatic Failover

When a query is configured with spares and a replica crashes for any reason, the controller will immediately replace the crashed worker with a running spare if it is available. Each replica slot independently respects the "Error Restart Attempts" scheduling parameter of the query configuration. This can prevent a single slot from continually claiming a spare query to the exclusion of other replicas.

Additional Changes

A new DbInternal table QueryUserAssignmentLog has been added so that Assignment Policies can log how users are assigned to available replicas.

-add_acl 'new UsernameFilterGenerator("QueryOwner")' -group allusers -namespace DbInternal -table QueryUserAssignmentLog -overwrite_existing
-add_acl * -group iris-superusers -namespace DbInternal -table QueryUserAssignmentLog -overwrite_existing

Server Selection Provider Replica Distribution

The default SimpleServerSelectionProvider attempts to more evenly spread replicas and spares of a single persistent query across the cluster so that a single server failure is less likely to impact all replicas of a running query.

The IServerSelectionProvider interface now requires three additional methods to be implemented:

• addWorkerForSerial is invoked when a worker for a persistent query is assigned to a server
• removeWorkerForSerial is invoked when a worker for a persistent query is terminated
• generateStatusInformation is generates internal state suitable for display by dhconfig pq selection-provider

Option to default all user tables to Parquet

Set the configuration property db.LegacyDirectUserTableStorageFormat=Parquet to default all direct user table operations, such as db.addTable, to the Parquet storage format. The default if the property is not set is DeephavenV1.

Default Landing Page

The default landing page for a Deephaven installation now automatically redirects to the web UI. To download the Swing Launcher, go to the /launcher path (e.g., https://deephaven.example.com:8123/launcher/ or https://deephaven.example.com:8000/launcher/).

Disabling Password Authentication

To disable password authentication within the authentication server, set the configuration property authentication.passwordsEnabled to false. When the property is set to false, the authentication server rejects all password logins and you must use SAML or private key authentication to access Deephaven.

Note that even if the UI presents a password prompt, the authentication backend rejects all passwords.

Deephaven processes log their heap usage

The db_dis, web_api_service, log_aggregator_service, iris_controller, db_tdcp, and configuration_server processes now periodically log their heap usage.

PersistentQueryController.log.current:[2024-05-10T15:00:32.365219-0400] - INFO - Jvm Heap: 3,972,537,856 Free / 4,291,624,960 Total (4,291,624,960 Max)
PersistentQueryController.log.current:[2024-05-10T15:01:32.365404-0400] - INFO - Jvm Heap: 3,972,310,192 Free / 4,291,624,960 Total (4,291,624,960 Max)

The logging interval can be configured using the property RuntimeMemory.logIntervalMillis. The default is one minute.

Worker Additional Memory Request

When creating a Code Studio or Persistent Query, Deephaven allows you to configure the heap size of the Java virtual machine running your process. The RemoteQueryDispatcher uses this parameter to not only to pass appropriate arguments to the Java process, but also to account for the RAM used by workers and when running in Kubernetes to set the resource requests and limits.

A Deephaven worker can use off-heap memory in various circumstances. For example, Java libraries may use direct buffers to reduce the overhead on the garbage collector or to improve I/O performance. When executing a Python worker, Python object are not part of the Java heap, but are instead allocated and garbage collected by the Python interpreter's memory management subsystem. Similarly, any native libraries also use off-heap memory.

If your worker process uses such off-heap memory, can request that memory from the dispatcher using the "Additional Memory" field under "Advanced Settings" in the Code Studio start screen and Persistent Query Settings tab. This is especially important on Kubernetes, where if a process's memory usage exceeds the container's limit the OS kernel forcibly kills the process.

Authorized Keys and Local Users Text Files

Deephaven no longer uses the dsakeys.txt or authusers.txt files by default. The Deephaven 1.20221001 release added support for storing public keys in the ACL store; and passwords hashes can be stored in the ACL store when iris.enableManagedUserAuthentication is set to true (which has been the default in iris-environment.prop in all supported software versions). Passwords and public keys can be manipulated with dhconfig acl user set-password and dhconfig acl publickey, respectively.

On upgrade, any keys stored in dsakeys.txt are migrated to the ACL store. On installation or upgrade, system keys for the controller, merge server, and TDCP are written to the ACL store.

To re-enable text file based keys, the following properties must be set in your iris-environment.prop file:

    authentication.server.localusers.enabled=true
    authentication.server.localusers.file=<name>
    authentication.server.authorizedkeys.enabled=true
    authentication.server.authorizedkeys.file=<name>

In-Worker Service Persistent Query Type Removed

The previously deprecated In-Worker Service Persistent Query type has been removed. Any existing persistent queries that are an In-Worker Service should be converted to use an appropriate script type before upgrading Deephaven. Most existing In-Worker Service queries are likely Data Import Servers, which can be converted with a script to a type of Live Query - Merge Server.

etcd configuration parameter 'bcrypt-cost: 4' added

When a system is installed, the etcd configuration file config.yaml adds the configuration parameter bcrypt-cost: 4.

This affects creation of new etcd users used internally by the Deephaven software. No changes are needed for Deephaven usernames or passwords stored in etcd. For a newly installed system this change applies immediately.

On a system that is being updated from an existing version, the installer will not modify the etcd configuration files; this can be done manually: on each node running etcd, modify the file /etc/etcd/dh/ETCD_CLUSTER_ID/config.yaml (replace ETCD_CLUSTER_ID by your etcd cluster id, a string looking something like 'cfa6c474b'). On each config.yaml file, add a line containing

bcrypt-cost: 4

On a system where the config file has been modified to add the new bcrypt cost value, this new value will only apply for new users created after the change, or for old users if their passwords are updated after the change; for existing users that do not update their password nothing changes.

A script redo-etcd-passwords is provided under the bin directory that updates the passwords for all etcd users except root; the password is not changed from the actual password value, but updating the password to the same value has the side effect that the user will use the new configuration parameter when the password is validated. Run the script as the irisadmin user while the etcd cluster is up but the Deephaven Enterprise system is down.

Background

etcd uses go's bcrypt function to hash passwords. Passwords are stored on disk hashed, to avoid compromising them immediately if an attacker is able to observe the data on disk (this is similar to how Unix/Linux systems store passwords hashed in the /etc/passwd file). When a user tries to authenticate with etcd and provides a password, etcd hashes the password again and compares the hash with the value stored on disk. The bcrypt function is computationally intensive; this is intentional to prevent an attacker from trying to guess passwords from being able to probe passwords quickly. This, however, has an impact in a system like Deephaven Enterprise:

• Deephaven Enterprise creates workers on demand, sometimes starting them as scheduled persistent queries. When many workers are starting at the same time, an etcd server machine can become very busy validating credentials via bcrypt for many new clients.

• When an etcd server that is leader fails, and a new server takes over as leader, many clients switch to the new server, all of them requiring the server to execute the bcrypt function to validate credentials; this can load a machine at the worst possible moment from a fault tolerance perspective.

For these reasons, it is desirable to reduce the cost of bcrypt to the minimum configurable. The default value used by bcrypt's go library (and etcd) is 10. The minimum value is 4. This configuration change sets the value explicitly to 4, where we previously were not configuring it and thus using the default.

etcd stores the bcrypt value used when a password is created or modified, to know at what value of bcrypt cost to validate the hash. Therefore, when the bcrypt cost value is changed it only applies for new users or after updating an existing user's password.

/usr/illumon/dnd changed to /usr/illumon/coreplus

The installer now uses /usr/illumon/coreplus for packages instead of /usr/illumon/dnd to reflect the updated Core+ name.

Kubernetes Heap Overhead Parameters

When running Deephaven installations in Kubernetes, the originally-implemented JVM overhead properties don't prevent some workers being killed with out-of-memory exceptions.

• Adding the BinaryStoreWriterV2.allocateDirect=false JVM parameter reduces direct memory usage, which is not counted towards dispatcher heap usage and can result in Kubernetes out-of-memory failures.
• Adding the -Xms JVM parameter allocates all requested heap at worker creation time, reducing the likelihood of after-startup worker out-of-memory failures from later memory requests.
• Adding the -XX:+AlwaysPreTouch JVM parameter to workers ensures that all worker heap is touched during startup, avoiding later page-faulting.

The following properties are being added to iris-environment.prop for new installations. Deephaven strongly suggests adding them manually to existing installations.

RemoteProcessingRequestProfile.Xms.G1 GC=$RequestedHeap
RemoteQueryDispatcher.JVMParameters=-XX:+AlwaysPreTouch
BinaryStoreWriterV2.allocateDirect=false

In addition, the property RemoteQueryDispatcher.memoryOverheadMB=500 is being updated in iris-defaults.prop, and this will automatically be picked up when the Kubernetes installation is upgraded.

dhconfig replaces etcd_prop_file

The etcd_prop_file tool has been removed. You must use dhconfig properties import instead.

For recovery or bootstrapping purposes, the --diskprops option can be used in conjunction with the --etcd option to read property files from disk and import new files into etcd.

Dispatcher Memory Reservation

The Remote Query Dispatcher (either db_query_server or db_merge_server) has a configurable amount of heap that can be dispatched to workers, which is controlled by setting the RemoteQueryDispatcher.maxTotalQueryProcessorHeapMB property. Setting this property requires accounting the other processes that may be running on the machine. If set too high, then workers may fail to allocate memory after being dispatched after dispatch or the kernel OOM killer may terminate processes. If set too low, then the machine may be underutilized.

As an additional safety check, the Remote Query Dispatcher can query the /proc/meminfo file for available heap. If a user requests more heap than the MemAvailable field indicates can be allocated to a new process, then the remote query dispatcher can reject scheduling the worker.

There are two new properties that control this behavior:

• RemoteQueryDispatcher.adminReservedAvailableMemoryMB; for users that are members of RemoteQueryDispatcher.adminGroups. By default this is set to 1024MiB.
• RemoteQueryDispatcher.reservedAvailableMemoryMB: For all other users. By default this is set to 2048MiB.

When set to -1, the additional check is disabled. When set to a non-negative value the dispatcher subtracts the property's value from the available memory, and verifies that the worker heap is less than this value before creating the worker.

You can examine the current status of properties, using the /config endpoint if RemoteQueryDispatcher.webserver.enabled is set to true. For example, navigate to `https://query-host.example.com:8084/config'. The available memory along with property values are displayed as an HTML table.

This property does not guarantee that workers or other processes are not terminated by the OOM killer. Running workers and processes may not have allocated their maximum heap size, and therefore can use system memory beyond what is available at dispatch time.

These properties have no effect on Kubernetes deployments. The dispatcher does not have an complete view of the available cluster resources.

ILLUMON_JAVA is deprecated. Use DH_JAVA instead.

In the past, specifying which version of java to use with Deephaven was done with the ILLUMON_JAVA and it was applied inconsistently.

In this release, you can set DH_JAVA=/path/to/java_to_use/bin/java in your cluster.cnf to tell all Deephaven processes where to find the correct java executable regardless of your PATH.

DH_JAVA works correctly whether you point to a java executable or a java installation directory (like "JAVA_HOME") Both DH_JAVA=/path/to/java_to_use and DH_JAVA=/path/to/java_to_use/bin/java operate identically.

If different machines in your cluster have java executables located in different locations, it is your responsibility to set DH_JAVA correctly in /etc/sysconfig/deephaven/cluster.cnf on each machine, or (preferably) to use a symlink so you have a consistent DH_JAVA location on all machines.

Managing Persistent Queries with dhconfig

The dhconfig tool can now manage Persistent Queries using the subcommand pq. For example, you can restart a query with:

dhconfig pq restart --name "Persistent Query Name"

Complete help is available by running dhconfig pq --help. This example demonstrates one of the improvements enabled by using dhconfig: the old controller tool does not handle query names with spaces. All existing operations that were performed by /usr/illumon/latest/bin/iris controller_tool have been integrated into dhconfig, and the old controller tool will be removed in a future release. See the help pages for more details dhctl controller --help

Legacy C# Open API Client Removed

Deephaven no longer builds or maintains the Legacy C# Open API client. C# Binary Loggers and the C# SBE client are still supported.

The Excel plugin for Legacy workers which depended on this client is also no longer supported.

Core+ Controller Python Imports

From Core+ Python workers, you may now import Python modules from repositories stored in the controller. To evaluate a single Python file:

import deephaven_enterprise.controller_import

deephaven_enterprise.controller_import.exec_script("script_to_execute.py")

To import a script as a module, you must establish a meta-import with a module prefix for the controller. The following example uses the default value of "controller" to load a module of the form "package1/package2.py" or "package1/package2/__init__.py":

import deephaven_enterprise.controller_import

deephaven_enterprise.controller_import.meta_import()

import controller.package1.package2

Refreshing Local Script Repositories

The Persistent Query Controller defines a set of script repositories that can be used from Persistent Queries or Code Studios. The repositories may be configured to use a remote Git repository or just a path on the local file system. The controller scans the repository on startup for the list of scripts that are available. Previously, only Git repositories could have updates enabled (once per minute); and local repositories would never be rescanned.

You can now set the property PersistentQueryController.scriptUpdateEnabled to true to enable script updates. If this property is not set, then the old PersistentQueryController.useLocalGit property is used (the old property has an inverse sense, meaning PersistentQueryController.useLocalGit=true stops updates and PersistentQueryController.useLocalGit=false permits updates) .

To mark a repository as local, the "uri" parameter must be set to empty. For example, if the repository was reffered to as "irisrepo" in the iris.scripts.repos property, then to mark the repository as local you would include a property like in your iris-environment.prop file:

iris.scripts.repo.irisrepo.uri=

Fixing etcd ACLs that broke after upgrading to URL encodings

Note that the following is only applicable to etcd ACLs.

In 1.20231218.116 and 1.20231218.132, Deephaven began URL encoding ACL keys to prevent special characters like '/' in keys from corrupting the ACL database. Although not all special characters corrupted the database, all of them are encoded, causing the unencoded database to be incompatible with the new version. A common occurrence of this pattern is the "@" character in usernames.

These ACL entries can be fixed using the EtcdAclEncodingTool.

First, back up your etcd database by reading our backup and restore instructions.

To rewrite these ACLs with proper encodings, run the following command as irisadmin:

sudo -u irisadmin /usr/illumon/latest/bin/iris_exec com.illumon.iris.db.v2.permissions.EtcdAclEncodingTool

To see what changes would occur without actually modifying the ACLs, run:

sudo -u irisadmin /usr/illumon/latest/bin/iris_exec com.illumon.iris.db.v2.permissions.EtcdAclEncodingTool -a --dry-run

Setting JVM JIT Compiler Options for Workers

The ability to set the maximum number of allowed JVM JIT compiler threads through the -XX:CICompilerCount JVM option has been added to JVM profiles using properties of the form RemoteProcessingRequestProfile.JitCompilerCount. See the remote processing profiles documentation for further information.

Upgrade etcd to 3.5.12

In past releases, we recommended upgrading etcd to 3.5.5.

It was later discovered that 3.5.5 has a known bug which can break your etcd cluster if you perform an etcdctl password reset.

As such, when upgrading etcd, you should prefer the Deephaven-tested 3.5.12 point release, which is the new default as of version 1.20231218.190.

All newly created systems will have 3.5.12 installed, but for existing systems, you must unpack new etcd binaries yourself.

You can find manual etcd installation instructions in the third-party dependencies guide.

Configurable gRPC Retries

The configuration service now supports using a gRPC service configuration file to configure retries, and one is provided by default for the system.

{
  "methodConfig": [
    {
      "name": [
          {
              "service": "io.deephaven.proto.config.grpc.ConfigApi"
          },
          {
              "service": "io.deephaven.proto.registry.grpc.RegistryApi"
          },
          {
              "service": "io.deephaven.proto.routing.grpc.RoutingApi"
          },
          {
              "service": "io.deephaven.proto.schema.grpc.SchemaApi"
          },
          {
              "service": "io.deephaven.proto.processregistry.grpc.ProcessRegistryApi"
          },
          {
              "service": "io.deephaven.proto.unified.grpc.UnifiedApi"
          }
      ],

      "retryPolicy": {
        "maxAttempts": 60,
        "initialBackoff": "0.5s",
        "maxBackoff": "2s",
        "backoffMultiplier": 2,
        "retryableStatusCodes": [
          "UNAVAILABLE"
        ]
      },

      "waitForReady": true,
      "timeout": "120s"
    }
  ]
}

methodConfig has one or more entries. Each entry has a name section with one or more service/method sections that filter whether the retryPolicy section applies.

If the method is empty or not present, then it applies to all methods of the service. If service is empty, then method must be empty, and this is the default policy.

The retryPolicy section defines how a failing gRPC call will be retried. In this example, grpc will retry for just over 1 minute while the status code is UNAVAILABLE (e.g. the service is down). Note this applies only if the server is up but the individual RPCs are being failed as UNAVAILABLE by the server itself. It the server is down, the status returned is UNAVAILABLE but the retryPolicy defined here for the method does not apply; gRPC manages reconnection retries for a channel separately/independently as described here: https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md

There is no way to configure the parameters for reconnection; see https://github.com/grpc/grpc-java/issues/9353

If the service config file specifies waitForReady, then an RPC executed when the channel is not ready (server is down) will not fail right away but will wait for the channel to be connected. This, combined with a timeout definition will make the RPC call hold on for as long as the timeout giving the reconnection policy a chance to get the channel to ready.

For Deephaven processes, customization of service config can be done by (a) copying configuration_service_config.json to /etc/sysconfig/illumon.d/resources and modifying it there, or (b) renaming it and setting property configuration.server.service.config.json.

Note that the property needs to be set as a launching JVM argument because this is used in the gRPC connection to get the initial properties.

Note: The relevant service names are:

io.deephaven.proto.routing.grpc.RoutingApi
io.deephaven.proto.config.grpc.ConfigApi
io.deephaven.proto.registry.grpc.RegistryApi
io.deephaven.proto.schema.grpc.SchemaApi
io.deephaven.proto.unified.grpc.UnifiedApi

Add Core+ Calendar support and allow Java ZoneId strings in Legacy Calendars

Core+ workers can use the Calendars.resourcePath property to load customer provided business calendars from disk. To use calendars in Core+ workers, any custom calendars on your resource path must be updated to use a standard TimeZone value.

Legacy workers also support using ZoneId strings instead of DBTimeZone values.

Dynamic management of Data Import Server configurations

Creating a new Data Import Server configuration and integrating it into the Deephaven system requires several steps, including required adjustments to the data routing configuration. This final step can now be performed with a few simple commands, and no longer requires editing the data routing configuration file.

dhconfig dis

The dhconfig command has a new action: dis, which supports import, add, export, list, delete, validate actions. The commands themselves provide help, and more information can be found in the dhconfig documentation.

dhconfig dis import

Import one or more configurations from one or more files. For example:

/usr/illumon/latest/bin/dhconfig dis import /path/to/kafka.yml

kafka.yml

kafka:
  name: kafka
  endpoint:
    serviceRegistry: registry
    tailerPortDisabled: 'false'
    tableDataPortDisabled: 'false'
  claims:
  - {namespace: Kafka}
  storage: private

dhconfig dis add

Define and import a single configuration on the command line. For example (equivalent to the import example above):

/usr/illumon/latest/bin/dhconfig dis add --name kafka --claim Kafka

dhconfig dis export

Export one or more configurations to one or more files. These files are suitable for the import command. For example, to export all configured Data Import Servers:

/usr/illumon/latest/bin/dhconfig dis export --file /tmp/import_servers.yml

dhconfig dis list

List all configured Data Import Servers. For example:

/usr/illumon/latest/bin/dhconfig dis list

Data import server configurations:
    kafka
    kafka3

dhconfig dis delete

Delete one or more configurations. For example:

/usr/illumon/latest/bin/dhconfig dis delete kafka --force

dhconfig dis validate

Validate one or more configurations. This can validate proposed changes before committing them with the import command. This process verifies that the configuration as a whole will be valid after applying the new changes.

Caveats

"Data routing configuration" comprises the "main" configuration file (managed with dhconfig routing) and additional DIS configurations. The main routing configuration may contain DIS configurations in the dataImportServers section. These two sources of DIS configurations are managed separately and are not permitted to contain duplicates. If you want to manage an existing DIS configuration with the new commands, you must remove it from the main routing configuration.

This functionality will only be useful for querying data if the routing configuration includes "all data import servers" using the dataImportServers keyword. This is usually a source under the db_tdcp table data service:

    db_tdcp:
      host: *localhost
      port: *default-tableDataCacheProxyPort
      sources:
        - name: dataImportServers

A DIS configuration requires storage. The special value private indicates that the server will supply its own storage location. Any other value must be present in the storage section of the routing configuration.

Update jgit SshSessionFactory to a more modern/supported version

For our git integration, we have been using the org.eclipse.jgit package. Github discontinued support for SHA-1 RSA ssh keys, but jgit's ssh implementation (com.jcraft:jsch) does not support rsa-sha2 signatures and will not be updated. To enable stronger SSH keys and provide GitHub compatibility, we have configured jgit to use an external SSH executable by setting the GIT_SSH environment variable. The /usr/bin/ssh executable must be present for Git updates.

Restartable Controller

If the iris_controller process restarts quickly enough, Core+ workers that were already initialized and running normally by the time the controller restarted continue running without interruption. Legacy workers still terminate on controller restart.

• The duration that workers can survive without the controller is defined by the property PersistentQueryController.etcdPresenceLeaseTtlSeconds, which defaults to 60 (seconds).
• Only workers that have completed initialization and are in the Running state before the crashed controller died and which should still be running by that time, according to their query configuration at the time of controller restart.

If the iris_controller is stopped normally (e.g., via monit stop or a regular UNIX TERM signal), the value of the property PersistentQueryController.stopWorkersOnShutdown determines the desired behavior for workers.

• When set to true, all controller-managed workers are stopped alongside the controller. This is consistent with the traditional behavior.
• When set to false (the new default), workers do not stop alongside the controller, and have the time defined in the property PersistentQueryController.etcdPresenceLeaseTtlSeconds (defaults to 60 seconds) as a grace period where they wait for the controller to come back.

If the controller crashes (i.e., the iris_controller process stopped unexpectedly by an exception that crashes the process, a machine reboot, or a UNIX KILL signal), then workers are not proactively stopped even if the value of PersistentQueryController.stopWorkersOnShutdown is true. In this case, the dispatcher terminates those workers after the PersistentQueryController.etcdPresenceLeaseTtlSeconds timeout.

Note: irrespective of the value of the PersistentQueryController.stopWorkersOnShutdown property, if the dispatcher associated to a worker is shutdown, the worker stops.

Renamed Swing Launcher Archives

The downloadable swing launcher has been renamed as follows:
DeephavenLauncherSetup_123.exe is now deephaven-launcher-123.exe DeephavenLauncher_123.tar is now deephaven-launcher-123.tgz

Reliable Barrage table connections

We have added a new library to provide reliable Barrage subscriptions within a Deephaven Core+ cluster. The new tables monitor the state of the source query and gracefully handle disconnection and reconnections without user intervention. This can be used to create reliable meshes of Core+ workers that are fault tolerant to the loss of other queries.

When using ResolveTools, PQ URLs (pq://MyQuery/scope/MyTable?columns=MyFirstColumn,SomeOtherColumn) use these new reliable tables.

To use this library see the following examples

import io.deephaven.enterprise.remote.RemoteTableBuilder
import io.deephaven.enterprise.remote.SubscriptionOptions

// Subscribe to the columns `MyFirstColumn` and `SomeOtherColumn` of the table `MyTable` from the query `MyQuery` 
table = RemoteTableBuilder.forLocalCluster()
    .queryName("MyQuery")
    .tableName("MyTable")
    .subscribe(SubscriptionOptions.builder()
        .addIncludedColumns("MyFirstColumn", "SomeOtherColumn").build())

from deephaven_enterprise import remote_table as rt

# Subscribe to the columns `MyFirstColumn` and `SomeOtherColumn` of the table `MyTable` from the query `MyQuery`
table = rt.in_local_cluster(query_name="MyQuery", table_name="MyTable").subscribe(
    included_columns=["MyFirstColumn", "SomeOtherColumn"]
)

Connecting to remote clusters

It is also possible to connect to queries on a different Deephaven cluster.

import io.deephaven.enterprise.remote.RemoteTableBuilder

table = RemoteTableBuilder.forRemoteCluster("https://other-server.mycompany.com:8000/iris/connection.json")
        .password("user", "password")
        .queryName("MyQuery")
        .tableName("MyTable")
        .subscribe(SubscriptionOptions.builder()
                .addIncludedColumns("MyFirstColumn", "SomeOtherColumn").build())

from deephaven_enterprise import remote_table as rt

# Subscribe to the columns `MyFirstColumn` and `SomeOtherColumn` of the table `MyTable` from the query `MyQuery`
table = rt.for_remote_cluster("https://other-server.mycompany.com:8000/iris/connection.json")
    .password("username", "password") \
    .query_name("MyQuery") \
    .table_name("MyTable") \
    .subscribe(included_columns=["MyFirstColumn", "SomeOtherColumn"])