Detailed Version Log Deephaven 1.20230511

Certified versions

Detailed Version Log: Deephaven v1.20230511

Detailed Release Candidate Version Log: Deephaven v1.20230201beta

Option to default all user tables to Parquet

Set the configuration property db.LegacyDirectUserTableStorageFormat=Parquet to default all direct user table operations, such as db.addTable, to the Parquet storage format. The default if the property is not set is DeephavenV1.

Deephaven processes log their heap usage

The db_dis, web_api_service, log_aggregator_service, iris_controller, db_tdcp, and configuration_server processes now periodically log their heap usage.

PersistentQueryController.log.current:[2024-05-10T15:00:32.365219-0400] - INFO - Jvm Heap: 3,972,537,856 Free / 4,291,624,960 Total (4,291,624,960 Max)
PersistentQueryController.log.current:[2024-05-10T15:01:32.365404-0400] - INFO - Jvm Heap: 3,972,310,192 Free / 4,291,624,960 Total (4,291,624,960 Max)

The logging interval can be configured using the property RuntimeMemory.logIntervalMillis. The default is one minute.

Configurable gRPC Retries

The configuration service now supports using a gRPC service configuration file to configure retries, and one is provided by default for the system.

{
  "methodConfig": [
    {
      "name": [
          {
              "service": "io.deephaven.proto.config.grpc.ConfigApi"
          },
          {
              "service": "io.deephaven.proto.registry.grpc.RegistryApi"
          },
          {
              "service": "io.deephaven.proto.routing.grpc.RoutingApi"
          },
          {
              "service": "io.deephaven.proto.schema.grpc.SchemaApi"
          },
          {
              "service": "io.deephaven.proto.processregistry.grpc.ProcessRegistryApi"
          },
          {
              "service": "io.deephaven.proto.unified.grpc.UnifiedApi"
          }
      ],

      "retryPolicy": {
        "maxAttempts": 60,
        "initialBackoff": "0.5s",
        "maxBackoff": "2s",
        "backoffMultiplier": 2,
        "retryableStatusCodes": [
          "UNAVAILABLE"
        ]
      },

      "waitForReady": true,
      "timeout": "120s"
    }
  ]
}

methodConfig has one or more entries. Each entry has a name section with one or more service/method sections that filter whether the retryPolicy section applies.

If the method is empty or not present, then it applies to all methods of the service. If service is empty, then method must be empty, and this is the default policy.

The retryPolicy section defines how a failing gRPC call will be retried. In this example, grpc will retry for just over 1 minute while the status code is UNAVAILABLE (e.g. the service is down). Note this applies only if the server is up but the individual RPCs are being failed as UNAVAILABLE by the server itself. It the server is down, the status returned is UNAVAILABLE but the retryPolicy defined here for the method does not apply; gRPC manages reconnection retries for a channel separately/independently as described here: https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md

There is no way to configure the parameters for reconnection; see https://github.com/grpc/grpc-java/issues/9353

If the service config file specifies waitForReady, then an RPC executed when the channel is not ready (server is down) will not fail right away but will wait for the channel to be connected. This, combined with a timeout definition will make the RPC call hold on for as long as the timeout giving the reconnection policy a chance to get the channel to ready.

For Deephaven processes, customization of service config can be done by (a) copying configuration_service_config.json to /etc/sysconfig/illumon.d/resources and modifying it there, or (b) renaming it and setting property configuration.server.service.config.json.

Note that the property needs to be set as a launching JVM argument because this is used in the gRPC connection to get the initial properties.

Note: The relevant service names are:

io.deephaven.proto.routing.grpc.RoutingApi
io.deephaven.proto.config.grpc.ConfigApi
io.deephaven.proto.registry.grpc.RegistryApi
io.deephaven.proto.schema.grpc.SchemaApi
io.deephaven.proto.unified.grpc.UnifiedApi

Update jgit SshSessionFactory to a more modern/supported version

For our git integration, we have been using the org.eclipse.jgit package. Github discontinued support for SHA-1 RSA ssh keys, but jgit's ssh implementation (com.jcraft:jsch) does not support rsa-sha2 signatures and will not be updated. To enable stronger SSH keys and provide GitHub compatibility, we have configured jgit to use an external SSH executable by setting the GIT_SSH environment variable. The /usr/bin/ssh executable must be present for Git updates.

Automatically Provisioned Python venv Will Only Use Binary Dependencies

All pip installs performed as part of the automatic upgrade of Python virtual environments will now pass the --only-binary=:all: flag, which will prevent pip from ever attempting to build dependencies on a customer machine.

As part of this change, we automatically upgrade pip and setuptools in all virtual environments, and have upgraded a number of dependencies which for pip refused to use prebuilt binary dependencies:

For all virtual environments:
dill==0.3.1.1 is now dill==0.3.3
wrapt==1.11.2 is now wrapt==1.13.2

For jupyter virtual environments:
backcall==0.1.0 is now backcall==0.2.0
tornado==6.0.3 is now tornado==6.1

Switch to NFS v4 for Kubernetes RWX Persistent Volumes

NFS v3 Persistent Volume connections do not support locking. This manifests most obviously when attempting to work with user tables in Deephaven on Kubernetes. By default, user table activities will wait indefinitely to obtain a lock to read or write data. This can be bypassed by setting -DOnDiskDatabase.useTableLockFile=false; this work-around was provided by DH-15640.

This change (DH-15830) switches Deephaven Kubernetes RWX Persistent Volume definitions to use NFS v4 instead, which includes lock management as part of the NFS protocol itself. In order for this change to be made, the NFS server must be reconfigured to export the RWX paths relative to a shared root path (fsid=0), but the existing PVs must use the same path to connect, since PV paths are immutable.

To reconfigure the NFS server: manually run the upgrade-nfs-minimal.sh script in the NFS server's Pod. It is important to set the environment variable SETUP_NFS_EXPORTS to y before running the script.

• To manually run the script against an NFS Pod:

  • Run kubectl get pods to get the name of your NFS server Pod and confirm that it is running.

  • Copy the setup script to the NFS pod by running this command, using your specific NFS pod name:

    # Run 'kubectl get pods' to find your specific nfs-server pod name and use that as the copy target host in this command.
    kubectl cp setupTools/upgrade-nfs-minimal.sh <nfs-server-name>:/upgrade-nfs-minimal.sh
    

  • Run this command to execute that script, once again substituting the name of your NFS Pod:

    kubectl exec <nfs-server-name> -- bash -c "export SETUP_NFS_EXPORTS=y && chmod 755 /upgrade-nfs-minimal.sh && /upgrade-nfs-minimal.sh"
    

The upgrade script:

• replaces /etc/exports, and backs up the original file to /etc/exports_<epoch_timestamp>. The new file will have only one entry, which exports the /exports directory with fsid=0.
• adds an exports sub-directory under /exports, and moves the dhsystem directory there. This is so clients will still find their NFS paths under /exports/dhsystem when connecting to the fsid=0 "root".

The existing PVs spec sections are updated with:

mountOptions:
    - hard
    - nfsvers=4.1

After upgrading to a version of Deephaven that includes this change (DH-15830), you should remove the -DOnDiskDatabase.useTableLockFile=false work-around, so normal file locking behavior can be used when working with user tables.

Requiring ACLs on all exported objects

When exporting objects from a Persistent Query, there are now two modes of operation controlled by the property PersistentQuery.openSharingDefault.

In either mode, when an ACL is applied to any object (e.g, tables or plots) within the query, then objects without an ACL are only visible to the query owner and admins (owners and admins never have ACLs applied).

When a viewer connects:

• If PersistentQuery.openSharingDefault is set to true, persistent queries that are shared without specifying table ACLs allow all objects to be exported to viewers of the query without any additional filters supplied. This is the existing Deephaven behavior that makes it simple to share PQ work product with others.
• If PersistentQuery.openSharingDefault is set to false, persistent queries that are shared without specifying table ACLs do not permit objects without an ACL applied to be exported to viewers. The owner of the persistent query must supply ACLs for each object that is to be exported.

Setting this property to false makes it less convenient to share queries, but reduces the risk of accidentally sharing data that the query writer did not intend. To enable this new behavior, you should update your iris-environment.prop property file.

Tailer configuration changes to isolate user actions

The tailer allocates resources for each connection to a Data Import Server for each destination (namespace, table name, internal partition, and column partition). System table characteristics are predictable and fairly consistent, and can be used to configure the tailer with appropriate memory.

User tables are controlled by system users, so their characteristics are subject to unpredictable variations. It is possible for a user to cause the tailer to consume large amounts of tailer resources, which can impact System data processing or crash the process.

This change adds more properties for configuration, and adds constraints on User table processing separate from System tables.

User table isolation

Resources for User table locations are taken from a new resource pool. The buffers are smaller by default, and the pool has a constrained size. This puts an upper limit on memory consumption when users flood the system with changed locations, which can happen with closeAndDeleteCentral or when back filling data. The resources for this pool are pre-allocated at startup. The pool size should be large enough to handle expected concurrent user table writes.

Tailer/DIS configuration options

The following properties configure the memory consumption of the Tailer and Data Import Server processes.

Revert to previous behavior

To disable the new behavior in the tailer, set the following property:

DataContent.disableUserPool = true

Added block flag to more dh_monit actions

This flag blocks scripting for the start, stop, and restart actions until the actions are completed. If any actions other than start, stop, restart, up or down are passed with the blocking flag, an error is generated. No other behaviors of the script have been changed.

These following options have been added:

/usr/illumon/latest/bin/dh_monit [ start | stop | restart ] [ process name | all ] [ -b | --block ]

These work as before:

/usr/illumon/latest/bin/dh_monit [ up | down ] [ -b | --block ]

Logging System Tables from Core+

Core+ workers can now log Table objects to a System table.

Many options are available using the Builder class returned by:

import io.deephaven.enterprise.database.SystemTableLogger
opts = SystemTableLogger.newOptionsBuilder().currentDateColumnPartition(true).build()

The only required option is what column partition to write to. You may specify a fixed column partition or use the current date (at the time the row was written, data is not introspected for a Timestamp). The default behavior is to write via the Log Aggregator Service, but you can also write via binary logs. No code generation or listener versioning is performed, you must write columns in the format that the listener expects. Complete Options are available in the Javadoc.

After creating an Options structure, you can then log the current table:

SystemTableLogger.logTable(db, "Namespace", "Tablename", tableToLog, opts)

When logging incrementally, a Closeable is returned. You must retain this object to ensure liveness. Call close() to stop logging and release resources.

lh=SystemTableLogger.logTableIncremental(db, "Namespace", "Tablename", tableToLog, opts)

The Python version does not use any options, but rather named arguments. If you specify None for the column partition, then the current date is used.

system_table_logger.log_table("Namespace", "Tablename", table_to_log, columnPartition=None)

Similarly, if you call log_table_incremental from Python; then you must close the returned object (or use it as context manager in a with statement)

Row by row logging is not yet supported in Core+ workers. Existing binary loggers cannot be executed in the context of a Core+ worker; because they reference classes that are shadowed (renamed). If row-level logging is required, then you must use io.deephaven.shadow.enterprise.com.illumon.iris.binarystore.BinaryStoreWriterV2 directly.

Only primitive types, Strings and Instants are supported. Complex data types cannot yet be logged.

Core+ support for multiple partitioning columns

Deephaven Core+ workers now support reading tables stored in the Apache Hive layout. Hive is a multi-level partitioned format where each directory is a Key=Value pair.

For example:

| Market                          -- A Directory for the Namespace
| -- EquityTrade                  -- A directory for the Table
|  | -- Region=US                 -- A Partition directory for the Region `US`
|  |  | -- Class=Equities         -- A Partition directory for the Class `Equities`
|  |  |  | -- Symbol=UVXY         -- A Partition directory for the Symbol `UVXY`
|  |  |  |  | -- table.parquet    -- A Parquet file containing data,
|  |  |  | -- Symbol=VXX          -- A Partition directory for the Symbol `VXX`
|  |  |  |  | -- table.size       -- A set of files for a Deephaven format table
|  |  |  |  | -- TradeSize.dat
|  |  |  |  | -- ...
|  | -- Region=Asia
|  |  | -- Class=Special
|  |  |  | -- Symbol=ABCD
|  |  |  |  | -- table.parquet
|  |  |  | -- Symbol=EFGH
|  |  |  |  | -- table.parquet

Core+ support for writing tables in Deephaven format

Deephaven Core+ workers now support writing tables in Deephaven format using the io.deephaven.enterprise.table.EnterpriseTableTools class in Groovy workers and the deephaven_enterprise.table_tools python module.

For example, to read a table from disk:

import io.deephaven.enterprise.table.EnterpriseTableTools
t = EnterpriseTableTools.readTable("/path/to/the/table")

from deephaven_enterprise import table_tools
t = table_tools.read_table("/path/to/the/table")

And to write a table:

import io.deephaven.enterprise.table.EnterpriseTableTools
EnterpriseTableTools.writeTable(qq, new File("/path/to/the/table"))

from deephaven_enterprise import table_tools
table_tools.write_table(table=myTable, path="/path/to/the/table")

See the Core+ documentation for more details on how to use this feature.

Core+ C++ client and derived clients support additional CURL options

When configuring a Session Manager with a URL for downloading a connection.json file, the C++ client and derived clients (like Python ticking or R) use libcurl to download the file from the supplied URL. SSL connections in this context can fail for multiple options and it is customary to support options to adjust SSL behavior and/or enable verbose output for supporting debugging. We now support the following environment variables from the clients:

• CURL_CA_BUNDLE: like the variable of the same name for the curl(1) command line utility. Points to a file containing a CA certificate chain to use instead of the system default.
• CURL_INSECURE: if set to any non-empty value, disable validation of server certificate.
• CURL_VERBOSE: if set to any non-empty value, enable debug output.

New Worker Labels

The Deephaven Enterprise system supports two kinds of workers.

The first uses the legacy Enterprise engine that predates the release of Deephaven Community Core. These workers are now labeled "Legacy" in the Code Studio and Persistent Query "Engine" field. Previously, these workers were labeled "Enterprise".

The second kind uses the Deephaven Community Core engine with Enterprise extensions. These workers are now labeled "Core+" in the Code Studio and Persistent Query "Engine" field. Previously, these workers were labeled "Community".

Although these changes may create short-term confusion for current users, Deephaven believes they better represent the function of these workers and will easily become familiar. Both Legacy and Core+ workers exist within the Deephaven Enterprise system. The Core+ workers additionally include significant Enterprise functionality that is not found within the Deepahven Community Core product.

To avoid breaking user code, we have not yet changed any package or class names that include either "Community" or "DnD" (an older abbreviation which stood for "Deephaven Community in Deephaven Enterprise").

Logger overhead

The default Logger creates a fixed pool of buffers. Certain processes are fine with a smaller size.

The following properties can be used to override the default configuration of the standard process Logger. Every log message uses an entry from the entry pool, and at least one buffer from the buffer pool. Additional buffers are taken from the buffer pool as needed. Both pools will expand as needed, so the values below dictate the minimum memory that will be consumed.

The default value for IrisLogCreator.entryPoolCapacity has been reduced to 16384 for Tailer processes.

generate-iris-keys and generate-iris-rsa no longer overwrite output

The generate-iris-keys and generate-iris-rsa scripts use OpenSSL to generate public and private keys. If you have an existing key file, the scripts now exit with a failure and you must remove the existing file before regenerating the key.

Kubernetes Helm Chart Changes

Some settings have changed or have been explicitly provided in place of whatever the default value was for your Kubernetes platform provider. For example terminationGracePeriodSeconds is set to a default of 10 in the management-shell. To avoid possible errors, delete the management-shell pod prior to doing the helm upgrade if you have an older version already running. The pod can be deleted with this command: kubectl -n <your-namespace> delete pod management-shell --grace-period 1.

Note that any files you may have copied or created locally on that pod will be removed. However, in the course of normal operations such files would not be present.

Kafka Offset Column Name

The default Community name for storing offsets is KafkaOffset. The Core+ Kafka ingester assumed this name, rather than using the name from the deephaven.offset.column.name consumer property.

If the default columns names of KafkaOffset, KafkaPartition, and KafkaTimestamp are not in your Enterprise schema, then the ingester ignores those columns. If you change column names for timestamp, offset, or partition; then you must also ensure that your schema contains a column of the correct type for that column.

Bypassing user table lock files

When a worker tries to write or read a User table, it will first try to lock a file in /db/Users/Metadata to avoid potential concurrency issues. If filesystem permissions are set up incorrectly, or if the underlying filesystem does not support file locking, this can cause issues.

The following property can be set to disable the use of these lock files:

OnDiskDatabase.useTableLockFile=false

Worker-to-worker table resolution configuration

Worker-to-worker table resolution now uses the Deephaven cluster's trust store by default. In some environments, there may be a SSL-related exception when when trying to resolve a table defined in one persistent query from another (see sharing tables for more). The property uri.resolver.trustall may be set to true globally in a Deephaven configuration file, or as a property in a Code Studio session as a JVM argument (e.g. -Duri.resolver.trustall=true). This will let the query worker sourcing the table trust a certificate that would otherwise be untrusted.

Added Envoy properties to allow proper operation in IPv6 or very dynamic routing environments

The new properties envoy.DnsType and envoy.DnsFamily allow configuration of Envoy DNS behaviors for xds routes added by the Configuration server.

• envoy.DnsType configures the value to be set in dynamically added xds routes for type. The default if this property is not set is LOGICAL_DNS. If there is a scenario where DNS should be checked on each connection to an endpoint, this can be changed to STRICT_DNS. Refer to Envoy documentation for more details about possible settings.

• envoy.DnsFamily configures the value to be set in dynamically added xds routes for dns_lookup_family. The default if this property is not set is AUTO. In environments where IPv6 is enabled the AUTO setting may cause Envoy to resolve IPv6 addresses for Deephaven service endpoints; since these service endpoints only listen on IPv4 stacks, Envoy will return a 404 or 111 when getting "Connection refused" from the IPv6 stack. Refer to Envoy documentation for more details about possible settings.

Since Deephaven endpoint services listen only on IPv4 addresses, and Envoy, by default, prefers IPv6 addresses, it may be necessary to modify the configuration in environments where IPv6 is enabled. To do this:

1. add an entry to the iris-environment.prop properties file of envoy.DnsFamily=V4_ONLY

2. edit the envoy3.yaml (or whichever configuration file Envoy is using) and add dns_lookup_family=V4_ONLY to the xds_service section:

   static_resources:
     clusters:
       - name: xds_service
         connect_timeout: 0.25s
         type: STRICT_DNS
         dns_lookup_family: V4_ONLY
   

3. import the new configuration and restart the configuration server and the Envoy process for the changes to take effect.

Modified Bessel correction formula for weighted variance

The weighted variance computation formula has been changed to match that used in the Deephaven Community engine. We now use the standard formula for "reliability weights" instead of the previous "frequency weights" interpretation. This will affect statistics based on variance such as standard deviation.

Managing Community Worker Python Packages

When starting a Deephaven Python worker, it executes in the context of a Python virtual environment (venv). This environment determines what packages are available to Python scripts. Packages that are important systemically or for multiple users should be added to the permanent virtual environment. With Community workers, the administrator may configure multiple worker kinds each with distinct virtual environments to enable more than one environment with a simple drop-down menu. For legacy Enterprise workers, users must manually set properties to select different virtual environments.

For experimentation, it can be convenient to install a Python package only in the context of the current worker. Community Python workers now have a deephaven_enterprise.venv module, which can be used to query the current path to the virtual environment and to install packages into the virtual environment with via pip with the install method. On Kubernetes, the container images now permit dbquery and dbmerge to write to the default virtual environment of /usr/illumon/dnd/venv/latest; which has no persistent effects on the system.

On a bare-Linux installation, the /usr/illumon/dnd/venv/latest must not be writable by users to ensure isolation between query workers. To allow users to install packages into the virtual environment, the administrator may configure a worker kind to create ephemeral environments on worker startup by setting the property WorkerKind.<name>.ephemeralVenv=true. This process increases worker startup time as it requires executing pip freeze and then pip install to create a clone of the original virtual environment. With an ephemeral virtual environment, the user can use deephaven_enterprise.venv.install to add additional packages to their worker. There is currently no interface to choose ephemeral environments at runtime.

Kubernetes Image Customization

When building container images for Kubernetes, Deephaven uses a default set of requirements that provide a working environment. However, many installations require additional packages. To facilitate adding new packages to the default virtual environment, a customer_requirements.txt file can be added to the deephaven_python and db_query_worker_dnd subdirectories of the docker build. After installing the default packages into the worker's virtual environment, pip is called to install the packages listed in customer_requirements.txt. If these files do not exist, the Deephaven build script creates empty placeholder customer_requirements.txt files.

Make /db/Users mount writeable in Kubernetes

This changes both the yaml for worker templates and the permissions on the underlying volume that is mounted as /db/Users in pods. If you are installing a new cluster, there is no action necessary. However, if you have an existing cluster installed then run this command to change the permissions: kubectl exec management-shell -- /usr/bin/chmod -vR 775 /db/Users

Helm improvements

A number of items have been added to the Deephaven helm chart, which allow for the following features:

• Configuration options to use an existing persistent volume claim in Deephaven, to allow for use of historical data stored elsewhere.
• Configuration options to mount existing secrets into worker pods.
• Configurable storageClass options to allow for easier deployment in various Kubernetes providers.

Required action when upgrading from an earlier release

1. Define global.storageClass: If you have installed an earlier version of Deephaven on Kubernetes then your my-values.yaml file used for the upgrade (not the Deephaven chart's values.yaml) should be updated to include a global.storageClass value, e.g.:

   global:
      storageClass: "standard-rwo"    # Use a value suitable for your Kubernetes provider
   

   The value should be one that is suitable for your Kubernetes provider; standard-rwo is a GKE-specific storage class used as an example. To see storageClass values suitable for your cluster, consult your provider's documentation. You can view your cluster's configured storageClass by running kubectl get storageClasses

2. Delete management-shell pod prior to running helm upgrade: Run kubectl delete pod management-shell to delete the pod. Note that if you happen to have any information stored on that pod it would be removed, though in the normal course of operations that would not be the case. This pod mounts the shared volumes used elsewhere in the cluster, and so changes to the storageClass values might result in an error similar to the following if it is not deleted when the upgrade is performed:

   $ helm upgrade my-deephaven-release-name ./deephaven/ -f ./my-values.yaml --set image.tag=1.20230511.248 --debug
   
   Error: UPGRADE FAILED: cannot patch “aclwriter-binlogs” with kind PersistentVolumeClaim: PersistentVolumeClaim “aclwriter-binlogs”
   is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
     core.PersistentVolumeClaimSpec{
       ... // 2 identical fields
       Resources:        {Requests: {s”storage”: {i: {...}, s: “2Gi”, Format: “BinarySI”}}},
       VolumeName:       “pvc-80a518f6-1a24-4c27-93b5-c7e9bd25d824”,
   -   StorageClassName: &“standard-rwo”,
   +   StorageClassName: &“default”,
       VolumeMode:       &“Filesystem”,
       DataSource:       nil,
      DataSourceRef:    nil,
   }
   

Ingesting Kafka Data from DnD

The Deephaven Community Kafka ingestion framework provides several advantages over the existing Enterprise framework. Notably:

• The Community Kafka ingester can read Kafka streams into memory and store them to disk.
• Key and Value specifications are disjoint, which is an improvement over the io.deephaven.kafka.ingest.ConsumerRecordToTableWriterAdapter pattern found in Enterprise.
• The Community KafkaIngester uses chunks for improved efficiency compared to row-oriented Enterprise adapters.

You can now use the Community Kafka ingester together with an in-worker ingestion server in a DnD worker. As with the existing Enterprise Kafka ingestion, you must create a schema and create a data import server within your data routing configuration. After creating the schema and DIS configuration, create an ingestion script using a Community worker.

You must create a KafkaConsumer Properties object. Persistent ingestion requires that auto commit is disabled in order to ensure exactly once delivery. The next step is creating an Options builder object for the ingestion and passing it to the KafkaTableWriter.consumeToDis function. You can retrieve the table in the same query, or from any other query according to your data routing configuration.

import io.deephaven.kafka.KafkaTools
import io.deephaven.enterprise.kafkawriter.KafkaTableWriter

final Properties props = new Properties()
props.put('bootstrap.servers', 'http://kafka-broker:9092')
props.put('schema.registry.url', 'http://kafka-broker:8081')
props.put("fetch.min.bytes", "65000")
props.put("fetch.max.wait.ms", "200")
props.put("deephaven.key.column.name", "Key")
props.put("deephaven.key.column.type", "long")
props.put("enable.auto.commit", "false")
props.put("group.id", "dis1")

final KafkaTableWriter.Options opts = new io.deephaven.enterprise.kafkawriter.KafkaTableWriter.Options()
opts.disName("KafkaCommunity")
opts.tableName("Table").namespace("Namespace").partitionValue(today())
opts.topic("demo-topic")
opts.kafkaProperties(props)
opts.keySpec(io.deephaven.kafka.KafkaTools.FROM_PROPERTIES)
opts.valueSpec(io.deephaven.kafka.KafkaTools.Consume.avroSpec("demo-value"))

KafkaTableWriter.consumeToDis(opts)

ingestedTable=db.liveTable("Namespace", "Table").where("Date=today()")

Customers can now provide their own JARs to Community in Enterprise (i.e. DnD) workers

Customers can now provide their own JARs into three locations that DnD workers can load from:

1. Arbitrary locations specified by the "Extra Classpaths" field from e.g. a console or Persistent Query configuration
2. A user-created location specific to a DnD Worker Kind configuration, specified by the WorkerKind.<Name>.customLib property
3. A default directory found in every DnD installation, e.g. /usr/illumon/dnd/latest/custom_lib/

Data routing file checks for duplicate keys

The data routing file is a YAML file. The YAML syntax includes name:value maps, and like most maps, cannot contain duplicate keys. Data routing file validation now raises an error when duplicate map keys are detected. The prior behavior was for the duplicate keys to silently replace the value in the map.

Reading Hierarchical Parquet Data

Deephaven Community workers can now read more complex Parquet formats through the db.historical_table method (or db.historicalTable from Groovy). Three new types of Parquet layouts are supported:

1. metadata: A hierarchical structure where a root table_metadata.parquet file contains the metadata and paths for each partition of the table.
2. kv: A hierarchical directory with key=value pairs for partitioning columns.
3. flat: A directory containing one or more Parquet files that are combined into a single table.

To read a Parquet table with the historical_table, you must first create a Schema that matches the underlying Parquet data. The Table element must have storageType="Extended", and a child element for ExtendedStorage that specifies a type. The valid type values are parquet:metadata, parquet:kv, and parquet:flat, corresponding to the supported layouts.

Legacy workers cannot read advanced Parquet layouts. If you call db.t with a table that defines Extended storage, an exception is raised.

com.illumon.iris.db.exceptions.ScriptEvaluationException: Error encountered at line 1: t=db.t("NAMESPACE", "TABLENAME")
...
caused by:
java.lang.UnsupportedOperationException: Tables with storage type Extended are only supported by Community workers.

Extended storage tables may have more than one partitioning column. The data import server can only ingest tables with a single partitioning column of type String. Attempts to tail binary files for tables that don't meet these criteria will raise an exception.

java.lang.RuntimeException: Could not create table listener
...
Caused by: com.illumon.iris.db.schema.SchemaValidationException: Tailing of schemas with multiple partitioning columns is not supported.

java.lang.RuntimeException: Could not create table listener
...
Caused by: com.illumon.iris.db.schema.SchemaValidationException: Tailing of schemas with a non-String partitioning column is not supported.

Discovering a Schema from an Existing Parquet Layout

You can read the Parquet directory using the standard community readTable function and create an Enterprise schema and table definition as follows:

import static io.deephaven.parquet.table.ParquetTools.readTable
import io.deephaven.enterprise.compatibility.TableDefinitionCompatibility
import static  io.deephaven.shadow.enterprise.com.illumon.iris.db.tables.TableDefinition.STORAGETYPE_EXTENDED

result = readTable("/db/Systems/PQTest/Extended/commodities")
edef = TableDefinitionCompatibility.convertToEnterprise(result.getDefinition())
edef.setName("commodities")
edef.setNamespace("PQTest")
edef.setStorageType(STORAGETYPE_EXTENDED)
ss=io.deephaven.shadow.enterprise.com.illumon.iris.db.schema.SchemaServiceFactory.getDefault()
ss.authenticate()
schema=io.deephaven.shadow.enterprise.com.illumon.iris.db.schema.xml.SchemaXmlFactory.getXmlSchema(edef, io.deephaven.shadow.enterprise.com.illumon.iris.db.schema.NamespaceSet.SYSTEM)
// If this is a new namespace
ss.createNamespace(io.deephaven.shadow.enterprise.com.illumon.iris.db.schema.NamespaceSet.SYSTEM, "PQTest")

// insert the ExtendedStorage type
schema.setExtendedStorageType("parquet:kv")
ss.addSchema(schema)

Read the table with:

db.historicalTable("PQTest", "Commodities")

Java Exception Logging

Deephaven logs now use the Java standard format for Exception stack traces, which includes suppressed exceptions and collapses repetitive stack trace elements, among other improvements.

ACLs for DbInternal CommunityIndex tables

Preexisting installs must manually add new ACLs for the new DbInternal tables.

First, create a text file (e.g. /tmp/new-acls.txt) with the following contents:

-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))' -group allusers -namespace DbInternal -table ServerStateLogCommunityIndex -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table UpdatePerformanceLogCommunityIndex -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table QueryOperationPerformanceLogCommunityIndex -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table QueryPerformanceLogCommunityIndex -overwrite_existing
exit

Then, run the following to add the new ACLs into the system:

sudo -u irisadmin /usr/illumon/latest/bin/iris iris_db_user_mod --file /tmp/new-acls.txt

Alternatively, the ACLs can be added manually one by one in the Swing ACL Editor:

allusers | DbInternal | ServerStateLogCommunityIndex | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))
allusers | DbInternal | UpdatePerformanceLogCommunityIndex | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))
allusers | DbInternal | QueryOperationPerformanceLogCommunityIndex | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))
allusers | DbInternal | QueryPerformanceLogCommunityIndex | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))

Seamless integration of Community panels in Deephaven Enterprise

Deephaven Enterprise now supports opening plots and tables from Community queries via the Panels menu. Community panels can be linked and filtered the same way as Enterprise.

Allow removal of "Help / Contact Support ..." via property

A new property, IrisConsole.contactSupportEnabled has been added, which may be used to remove the "Help / Contact Support ..." button from the swing front-end.

By default, this property is set to true in order to preserve current behavior. Setting this to false in properties will remove the menu-option.

db available via import in Community Python workers

In Community Python workers, the Database object db can now be imported into user scripts and modules directly using import statements, for example:

from deephaven_enterprise.database import db

my_table = db.live_table(namespace="MyNamespace", table_name="MyTable").where("Date=today()")

The db object is still available as a global variable for Consoles and Persistent Query scripts.

OperationUser columns added to DnD DbInternal tables

The internal performance tables for Community workers now have columns for OperationAuthenticatedUser and OperationEffectiveUser. This updates the schema for QueryPerformanceLogCommunity, QueryOperationPerformanceLogCommunity, and UpdatePerformanceLogCommunity. The operation user reflects the user that initiated an operation over the network, which is especially important for analyzing the performance of shared persistent queries. For example, filtering, sorting, or rolling up a table can require significant server resources.

No manual changes are needed. The Deephaven installer will deploy the new DbInternal schemas and the new data is ingested into separate internal partitions.

ProcessMetrics logging is now disabled by default

ProcessMetrics logging is now disabled by default in both Enterprise (DHE) and Community in Enterprise (DnD). To enable ProcessMetrics logging, set IrisLogDefaults.writeDatabaseProcessMetrics to true. If desired, you can control DnD ProcessMetrics logging separately from DHE via statsLoggingEnabled.

Kafka Version Upgrade

We have upgraded our Kafka code from version 2.4 to version 3.4.

Confluent Breaking Changes

Confluent code must be upgraded to version 7.4 to be compatible with version 3.4. https://docs.confluent.io/platform/current/installation/versions-interoperability.html

Clients using Avro or POJO for in-worker DISes must switch to the 7.4 versions of the required jars, as specified here: https://deephaven.io/enterprise/docs/importing-data/advanced/streaming/kafka/#generic-record-adapter

The following dependencies are now included in the Deephaven installation:

jackson-core-2.10.0.jar
jackson-databind-2.10.0.jar
jackson-annotations-2.10.0.jar

Users should remove these from their classpath (probably /etc/sysconfig/illumon.d/java_lib) to avoid conflict with the included jars.

Flight can now resolve Live, Historical and Catalog tables from the database

DnD workers now support retrieving live, historical and catalog tables through Arrow Flight. DnD's Python client has been updated with DndSession.live_table(), DndSession.historical_table() and DndSession.catalog_table() to support this.

For example, to fetch the static FeedOS.EquityQuoteL1 table

from deephaven_enterprise.client.session_manager import SessionManager

connection_info = "https://my-deephaven-host.com:8000/iris/connection.json"
session_mgr: SessionManager = SessionManager(connection_info)
session_mgr.password("iris","iris")

session = session_mgr.connect_to_persistent_query("CommunityQuery")
Quotes = session.historical_table("FeedOS", "EquityQuoteL1").where("Date=`2023-06-15`")

Flight ticket structure

Database flight tickets start with a prefix d and then are followed by a path consisting of three parts. The first part selects the type, the second is the namespace, and the third is the table name. Available types are catalog for the catalog table, live for live tables and hist for historical tables.

For example d/live/Market/EquityQuote would fetch a the live Market.EquityQuote table. Note that the catalog version does not use a namespace or tablename d/catalog will fetch the catalog table.

Reduce default max table display size

The maximum number of rows that may be displayed in the swing front-end before the red "warning bar" is displayed is now configurable. A new default maximum has been defined as 67,108,864 (64 x 1024 x 1024). Technical limitations cause rows beyond this limit to not properly update. When necessary, the Web UI is capable of displaying much larger tables than Swing.

The previous default max may be configured with the following property:

DBTableModel.defaultMaxRows=100000000

Note that the property-defined maximum may be programmatically reduced based on technical limits.

Improved Metadata Indexer tool

The Metadata Indexer tool has been improved so that it can now validate and list table metadata indexes on disk.
The tool can be invoked using the dhctl script with the metadata command.

Deephaven now supports subplotting in the Web UI

Users now have the ability to view multiple charts subplotted in one figure using the Web UI. Create subplots using the newChart, colSpan, and rowSpan functions available on a Figure. Details are available in the plotting guide.

Example Groovy code of subplots

tt = timeTable("00:00:00.01").update("X=0.01*ii", "Y=ii*ii", "S=sin(X)", "C=cos(X)", "T=tan(X)").tail(1000)

// Figure with single plot
f1 = figure().plot("Y", tt, "X", "Y").show()

// Figure with two plots, one on top of the other
f2 = figure(2, 1)
    .newChart(0,0).plot("S", tt, "X", "S")
    .newChart(1,0).plot("C", tt, "X", "C")
    .show()

// Figure with 3 plots, one that takes up the full width and then two smaller ones
f3_c = figure(2, 2)
    .newChart(0,0).plot("T", tt, "X", "T").colSpan(2)
    .newChart(1,0).plot("S", tt, "X", "S")
    .newChart(1,1).plot("C", tt, "X", "C")
    .show()

// Figure with 3 plots, one that takes up the full height and then two smaller ones
f3_r = figure(2, 2)
    .newChart(0,0).plot("T", tt, "X", "T")
    .newChart(1,0).plot("S", tt, "X", "S")
    .newChart(0,1).plot("C", tt, "X", "C").rowSpan(2)
    .show()
    
// Figure with 4 plots arranged in a grid
f4 = figure(2, 2)
    .newChart(0,0).plot("Y", tt, "X", "Y")
    .newChart(1,0).plot("S", tt, "X", "S")
    .newChart(0,1).plot("C", tt, "X", "C")
    .newChart(1,1).plot("T", tt, "X", "T")
    .show()

// Re-ordered operations from f4, should appear the same though
f5 = figure(2, 2)
    .newChart(1,1).plot("T", tt, "X", "T")
    .newChart(0,1).plot("C", tt, "X", "C")
    .newChart(1,0).plot("S", tt, "X", "S")
    .newChart(0,0).plot("Y", tt, "X", "Y")
    .show()

Improved validation of data routing configuration can cause errors in existing configurations

This Deephaven release includes new data routing features, and additional validation checks to detect possible configuration errors. Because of the additional validation, it is possible that an existing data routing configuration that was previously valid is now illegal and will cause parsing errors when the configuration server reads it.

If this occurs, the data routing configuration must be corrected, with the dhconfig tool in --etcd mode to bypass the configuration server (which fails to start when the routing configuration is invalid).

Export the configuration:

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig routing export --file /tmp/routing.yml --etcd

Edit the exported file to correct errors, and import it:

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig routing import --file /tmp/routing.yml --etcd

Python Integral Widening

In the 1.20211129 release, the jpy module that Deephaven's Python integration depends on converting all Python integral results into a Java integer. This resulted in truncated results when values exceed Integer.MAX_VALUE. In 1.20221001, Deephaven is using an updated jpy Integration that returns values in the narrowest possible type; so results that previously were an integer could be returned as a byte or a short. Moreover, a formula may have different types for each row. This prevented casting the result into a primitive type, as boxed objects may not be casted to another primitive.

In 1.20221001.196, Python calls in a formula now widen Byte and Short results to an Integer. If the value returned exceeds, Integer.MAX_VALUE, then the result is a Long. Existing formulas that would not have been truncated by conversion to an int in 1.20211129, behave as they would have in that release.

As casting from an arbitrary integral type to a primitive may be required, we have introduced a utility class com.illumon.iris.db.util.NumericCast that provides objectToByte, objectToShort, objectToInt, and objectToLong methods that will convert any Byte, Short, Integer, Long, or BigInteger into the specified type. If an overflow would occur, an exception is thrown.

Numba formulas (those that are surrounded in the nb function); have the narrowing behavior as in prior versions of 1.20221001.

Changed to use DHC Fast CSV parser for readCsv

TableTools.readCsv calls now use the new DHC High-Performance CSV Parser that uses a column oriented approach to parse CSV files.

The change to DHC parser includes the following visible enhancements

1. Any column that is only populated with integer surrounded by white space will be identified as an integer column. The previous parser would identify the column as a double.

2. Only 7-bit ASCII is supported as valid delimiters. This means characters such as € (euro symbol) are not valid. In these cases the following error will be thrown, delimiter is set to '€' but is required to be 7-bit ASCII.

3. Columns populated wholly with only single characters will be identified as Character columns instead of String columns.

4. Additional date time formats are automatically converted to DBDateTime columns. Previously, these formats were imported as String columns. All other date time behavior remains unchanged.

   ---

   | Format | Displayed Value in 1.20211129 | Data Type In 1.20211129 | Displayed Value in 1.20221001 | Data Type in 1.20221001 |

   | DateTimeISO_UTC_1 | 2017-08-30 11:59:59.000Z | java.lang.String | 2017-08-30T07:59:59.000000000 NY | com.illumon.iris.db.tables.utils.DBDateTime | | DateTimeISO_UTC_2 | 2017-08-30T11:59:59.000Z | java.lang.String | 2017-08-30T07:59:59.000000000 NY | com.illumon.iris.db.tables.utils.DBDateTime | | DateTimeISO_MillisOffset_2 | 2017-08-30T11:59:59.000-04 | java.lang.String | 2017-08-30T11:59:59.000000000 NY | com.illumon.iris.db.tables.utils.DBDateTime | | DateTimeISO_MicrosOffset_2 | 2017-08-30T11:59:59.000000-04 | java.lang.String | 2017-08-30T11:59:59.000000000 NY | com.illumon.iris.db.tables.utils.DBDateTime |

To use the legacy CSV parser, set the configuration property com.illumon.iris.db.tables.utils.CsvHelpers.useLegacyCsv to true.

Support Barrage subscriptions between DnD workers

DnD workers can now subscribe to tables in other DnD workers using Barrage.

This can be done using ResolveTools and a new URI scheme pq://<Query Identifier>/scope/<Table name>[?snapshot=true] The Query Identifier can be either the query name or the query serial. The Table Name is the name of the table in the server query's scope. The optional snapshot=true parameter indicates that a snapshot should be fetched instead of a live subscription.

import io.deephaven.uri.ResolveTools
TickingTable = ResolveTools.resolve("pq://CommunityQuery/scope/TickingTable?snapshot=true") 

from deephaven_enterprise.uri import resolve
TickingTable = resolve("pq://CommunityQuery/scope/TickingTable?snapshot=true") 

Improvements to command line scripts

Deephaven provides many maintenance and utility scripts in /usr/illumon/latest/bin. This release changes many of these tools to more consistently handle configuration files, setting java path and classpath, error handling, and logging.

Classpaths now include customer plugins and custom jars. This is important for features that can include custom data types, including table definitions and schemas.

For the tools included in this update, there is now a consistent way to handle invalid configuration and other unforeseen errors.

Override the configuration (properties) file

If the default properties file is invalid for some reason, override it by setting DHCONFIG_ROOTFILE. For example:

DHCONFIG_ROOTFILE=iris-defaults.prop /usr/illumon/latest/bin/dhconfig properties list

Add custom JVM arguments

Add java arguments to be passed into the java program invoked by these scripts by setting EXTRA_JAVA_ARGS. For example:

EXTRA_JAVA_ARGS="-DConfiguration.rootFile=foo.prop" /usr/illumon/latest/bin/dhconfig properties list

Scripts included in this update

The following scripts have been updated:

• crcat
• data_routing
• defcat
• delete_schema
• dhconfig
• dhctl
• export_schema
• iriscat
• iristail
• migrate_acls
• migrate_controller_cache
• validate_routing_yml

Controller Tool "Status" Option

The new --status subcommand for the persistent query controller tool generates a report to standard output with details of selected persistent queries.

With --verbose, more details are included. If a query has a failure recorded and only one query is selected, the stack trace is printed after the regular report. Use the --serial option to directly select a specific query.

With --jsonOutput, a JSON block detailing the selected query states is emitted instead of the formatted report. Use --jsonFile to specify an output location other than standard output.

Possible breaking changes were introduced with this feature:

• Previously (before Silverheels) the flag options --continueAfterError, --includeTemporary and --includeNonDisplayable required but ignored a parameter. For example, --includeTemporary=false and --continueAfterError=never were both accepted as "true" conditions. In Silverheels, the argument is still required, but only true and 1 will be accepted as true, false and 0 will be accepted as false, and anything else will be treated as a command line error.
• Details of information log entries generated by command_tool have changed. Important functionality had previously been deferred to after the starting/finished log entries for the corresponding items had been emitted. Those actions are now bracketed by the log marker entries to better inform troubleshooting.
• A warning message is emitted to the console when no queries are processed due to selection (filtering) criteria. An informational console message summarizing the filter actions has also been added.

Code Studio Engine Display Order

When selecting the engine (Enterprise or Community) in a Code Studio, existing Deephaven installations show the Enterprise engine first for backwards compatibility. New installations show the Community engine first. This is controlled by a display order property defined for each worker kind. Lower values are displayed first by the Code Studio drop down.

By default, the Enterprise engine has a display order of 100 and Community engine has a display order of 200. For a new installation, the iris-environment.prop file sets the priority of the Community engine to 50 as follows:

WorkerKind.DeephavenCommunity.displayOrder=50

You may adjust the display order properties for community workers by changing the display order property as desired.

etcd ownership

In previous releases, if the Deephaven installer installed etcd, the etcd and etcdctl executables in /usr/bin were created with the ownership of the user who ran the installation. They should be owned by root.

ls -l /usr/bin/etcd*

If the ownership isn't root:

sudo chown root:root /usr/bin/etcd*

ACLs for DbInternal Index and Community tables

Preexisting installs must manually add new ACLs for the new DbInternal tables.

First, create a text file (e.g. /tmp/new-acls.txt) with the following contents:

-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))' -group allusers -namespace DbInternal -table ProcessEventLogIndex -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))' -group allusers -namespace DbInternal -table ProcessTelemetryIndex -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table UpdatePerformanceLogIndex -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table QueryOperationPerformanceLogIndex -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table QueryPerformanceLogIndex -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))' -group allusers -namespace DbInternal -table ProcessInfoLogCommunity -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))' -group allusers -namespace DbInternal -table ProcessMetricsLogCommunity -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))' -group allusers -namespace DbInternal -table ServerStateLogCommunity -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table UpdatePerformanceLogCommunity -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table QueryOperationPerformanceLogCommunity -overwrite_existing
-add_acl 'new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))' -group allusers -namespace DbInternal -table QueryPerformanceLogCommunity -overwrite_existing
exit

Then, run the following to add the new ACLs into the system:

sudo -u irisadmin /usr/illumon/latest/bin/iris iris_db_user_mod --file /tmp/new-acls.txt

Alternatively, the ACLs can be added manually one by one in the Swing ACL Editor:

allusers | DbInternal | ProcessEventLogIndex | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))
allusers | DbInternal | ProcessTelemetryIndex | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))
allusers | DbInternal | UpdatePerformanceLogIndex | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))
allusers | DbInternal | QueryOperationPerformanceLogIndex | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))
allusers | DbInternal | QueryPerformanceLogIndex | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))
allusers | DbInternal | ProcessInfoLogCommunity | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))
allusers | DbInternal | ProcessMetricsLogCommunity | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))
allusers | DbInternal | ServerStateLogCommunity | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("EffectiveUser"), new UsernameFilterGenerator("AuthenticatedUser"))
allusers | DbInternal | UpdatePerformanceLogCommunity | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))
allusers | DbInternal | QueryOperationPerformanceLogCommunity | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))
allusers | DbInternal | QueryPerformanceLogCommunity | new DisjunctiveFilterGenerator(new UsernameFilterGenerator("PrimaryEffectiveUser"), new UsernameFilterGenerator("PrimaryAuthenticatedUser"))

DnD Now supports Edge ACLs

Query writers can now specify ACLs on derived tables. These ACLs will be applied when tables or plots are fetched by a client based upon the client's groups.

Edge ACLs are created using the EdgeAclProvider class in the io.deephaven.enterprise.acl package. Additionally, the io.deephaven.enterprise.acl.AclFilterGenerator interface contains some helpful factory methods for commonly used ACL types.

The following example assumes that a table "TickingTable" has already been created. Edge ACLs are created using a builder that contains a few simple methods for building up ACL sets. Once build() is called you have an ACL object which can then be used to transform one or more tables using the applyTo() method. Note that you must overwrite the scope variable with the result of the application, since Table properties are immutable.

import io.deephaven.enterprise.acl.EdgeAclProvider
import io.deephaven.enterprise.acl.AclFilterGenerator

def ACL = EdgeAclProvider.builder()
        .rowAcl("NYSE", AclFilterGenerator.where("Exchange in `NYSE`"))
        .columnAcl("LimitPrice", "*", AclFilterGenerator.fullAccess())
        .columnAcl("LimitPrice", ["Price", "TradeVal"], AclFilterGenerator.group("USym"))
        .build()

TickingTable = ACL.applyTo(TickingTable)

from deephaven_enterprise.edge_acl import EdgeAclProvider
import deephaven_enterprise.acl_generator as acl_generator

ACL = EdgeAclProvider.builder() \
    .row_acl("NYSE", acl_generator.where("Exchange in `NYSE`") \
    .column_acl("LimitPrice", "*", acl_generator.full_access()) \
    .column_acl("LimitPrice", ["Price", "TradeVal"], acl_generator.group("USym")) \
    .build()
    
TickingTable = ACL.apply_to(TickingTable)

See the DnD documentation for details on the AclFilterGenerator and EdgeAclProvider interfaces.

Remote R Groovy Sessions

The idb.init method now has an optional remote parameter. When set to TRUE, Groovy script code is not executed locally but rather on a remote Groovy session as is done in the Swing console or Web Code Studio. This eliminates a class of serialization problems that could otherwise occur with a local Groovy session serializing classes to the remote server. To use the old local Groovy session, you must pass the remote parameter as follows:

idb.init(devroot=devroot, workspace, propfile, keyfile=keyfile, jvmArgs=jvmLocalArgs, remote=FALSE)

Additionally, you may now call idb.close() to terminate the remote worker and release the associated server resources.

LogEntry Interface Change

In the com.fishlib.io.log.LogEntry class, the end() and endl() methods have been changed so that instead of returning the LogEntry instance on which they are operating, they don't return anything. After these methods have been called their LogEntry instance should not be operated on, and further operations on that LogEntry can introduce run-time issues.

Because of this change, any code that uses the Deephaven logging classes will need to be recompiled. If the logging calls rely on the returned LogEntry they will need to be updated.

The Client Update Service has been removed as a stand-alone service

The Client Update Service (CUS) that serves Deephaven jars, properties, etc. to clients has been integrated into the same web server as the Web API Service. This eliminates the need to use two separate web ports on the infrastructure server and simplifies updating TLS certificates.

How to configure

The host, port, and protocol of the old CUS could be configured using the files getdown.host, getdown.port, and getdown.protocol. This is no longer supported. The new CUS uses the same protocol as the Web API Server. If Envoy configuration is not enabled, the host and port can be configured with the client_update_service.host and client_update_service.port properties.

What has changed

The old CUS was a stand-alone process with its own web server and was accessed from a separate port from the Web API Server. The old CUS served files from the /var/www/lighttpd/iris/iris/ directory, which it built on startup. This meant that it needed to be restarted to make new files or configuration available to a user using the legacy Swing Launcher.

The new CUS uses the same web server as the Web API Service and is accessed on the same host and port (by default, 8123). The new CUS will still be available on the old default port, 8443. Instead of restarting the Web API Service to make new files available to Swing clients, you can navigate to the URL https://<WEB_HOST:WEB_PORT>/reload. This will build a new directory inside a temporary location from which the CUS will serve files.

The webpage for downloading the legacy Swing launcher has moved to https://<WEB_HOST:WEB_PORT>/launcher.

If you are upgrading without using the installer

Delete the standalone client update service process configuration from monit: sudo -u irisadmin rm /etc/monit.d/deephaven/service/cus.conf

You will need to reload monit afterwards to see changes (note that this will not restart any processes): /usr/illumon/latest/bin/dh_monit reload

Inside iris-environment.prop, set the property client_update_service.host to the fully-qualified domain name of the host on which the Web API Service will run. If this is not set, the Web API Service will fail with com.illumon.iris.web.server.DHFileDigester$DigestException: Unknown host for client update service, client_update_service.host must be set

Replace PersistentQueryController communication with gRPC

The PersistentQueryController now uses gRPC as a communication transport for clients. All data that was previously serialized and sent using Java serialization is now packaged and sent as protobufs. This will enable cross-language clients to access the Deephaven backend using standard tools.

In order to facilitate this, a few polymorphic classes contained in messages exchanged by the Controller and Client have been explicitly converted to JSON.

ScriptPathLoaderState

The ScriptPathLoaderState contains information about the git repository, branch and commit to use when fetching script details. This should be transparent unless you have implemented your own ScriptPathLoader, or are directly using it in a custom PersistentQueryConfiguration type. A method has been added to ScriptPathLoaderState to encode into JSON:

/**
 * Encode this state object as a JSON string.
 * @return a JSON encoded version of this object.
 */
String encodeJSON();

A new method has been added to the ScriptPathLoader to decode a JSON object back into a ScriptPathLoaderState instance:

/**
 * Create a state object from the serialized JSON String provided by {@link ScriptPathLoaderState#encodeJSON()}
 * 
 * @param stateJSON the JSON state
 * @return the deserialized object from the JSON state
 */
void ScriptPathLoaderState makeState(@NotNull final String stateJSON)

Special Persistent Query types

If you have implemented your own PersistentQueryConfigurationTypes you will need to implement JSON encoding for each of the following classes.

TypeSpecificWireConfig

The type specific configuration is still sent to the SetupQuery that runs within the worker as an object using Java serialization. However, the UI Components for configuring the query in Swing must be updated to expect this value as a JSON string. A new method has been added to the interface for you to serialize the object to JSON.

 public interface TypeSpecificWireConfig {
    /**
     * Encode this object as JSON for serialization.
     * @return a JSON string representing this object
     */
    String encodeJSON();
}

TypeSpecificState

The TypeSpecific state for custom query types is set during the SetupQuery phase of initialization. Previously, you would pass the TypeSpecificState object directly into the PersistentQueryState object. You must now encode it into a JSON string. This is reflected in the signature change of the PersistentQuery.getSessionState() method. The last parameter is now a String instead of a TypeSpecificState.

public static PersistentQueryState getSessionState(final PersistentQueryConfiguration config,
                                                   final ScriptSession session,
                                                   @Nullable final String typeSpecificStateJson)

TypeSpecificFields

Type specific fields encode information about the query configuration. For example, Merge queries have a TableName and Namespace field. In the backend, type specific fields are stored as a Map<String, Object>. When communicating with the front end, they must now be encoded as JSON using the following methods:

PersistentQueryConfigurationMarshaller.encodeTypeSpecificFieldsJson(Map<String, Object> values)
PersistentQueryConfigurationMarshaller.decodeTypeSpecificFieldsJson(String typeSpecificFieldsJson)

The types allowed in the map are now restricted to primitive types (eg. byte, int long, etc.), String, and arrays of those types. More complex types must be pre-encoded into a string.

Envoy Configuration for ACL Write Server

The ACL write server now uses a REST API instead of a Java serialization protocol. When using Envoy in prior versions, the setting the property envoy.hasCommRoutes=false would disable the ACL write server route. To disable the ACL write server route, you must now explicitly set envoy.hasAclSvc=false.

Simplified data routing

Deephaven supports multiple data import servers (DIS), including in-worker implementations that support importing specific data sources. The existing data routing configuration file can become very complicated because the tables handled by one of these DISes must be excluded from the others.

This change creates a framework wherein specific namespaces and tables can be assigned globally to specific DIS instances.

Claiming a table

In the YML data routing configuration file, claim a table or namespace for a DIS by adding claims to the configuration section for that DIS.

  dataImportServers:
    dis_claims_kafka:
      ...
      claims:
        - {namespace: Kafka, tableName: table1}
        - {namespace: Kafka2, tableName: "*"}
        - {namespace: Kafka3}

The tableName is optional in the claim. If omitted, or if the value is *, then all tables in the namespace are claimed. If given, only that single table is claimed.

Each DIS may claim multiple tables or namespaces. Any number of DISes may claim distinct tables or namespaces. A DIS that uses filters may not make claims and vice versa.

A DIS with claims handles only data for those claims. No other DIS (except in the same failover group) handles data for those claimed tables.

Multiple claims on the same table or namespace are not allowed (but see failover groups below).

Overriding claims

A claim may be made for a single table, or for an entire namespace. A single table claim overrides a namespace claim. All claims in the file are evaluated together, so file order is not meaningful.

In this example, dis_claims_kafka_t1 will be the source for Kafka.table1, and dis_claims_kafka will be the source for all other tables in the Kafka namespace.

  dataImportServers:
    dis_claims_kafka:
      ...
      claims:
        - {namespace: Kafka}
    dis_claims_kafka_t1:
      ...
      claims:
        - {namespace: Kafka, tableName: table1}

Failover groups

A failover group is a set of DISes that handle equivalent data. All members of a failover group must have identical filters and claims. If one DIS in this list is unavailable, the other will be automatically used.

This release adds a failoverGroup keyword to make failover groups more explicit.

DISes in the same failover group are permitted to make identical claims.

Create a failover group by adding the same failoverGroup tag to all member dataImportServers sections, and then use the group name in the tableDataServices section:

  dataImportServers:
    dis_kafka1:
      ...
      failoverGroup: kafkaGroup
      claims:
        - {namespace: Kafka}
    dis_kafka2:
      ...
      failoverGroup: kafkaGroup
      claims:
        - {namespace: Kafka}

  tableDataServices:
    db_tdcp:
      sources:
        - name: kafkaGroup

Note: failover groups were supported in previous releases with this syntax in the tableDataServices section:

  tableDataServices:
    db_tdcp:
      sources:
        - name: [dis1, dis2]

Implications

The notion of tables being claimed by specific DISes is enforced at a global level. This means that the claimed tables do not need additional exclusions in the configuration file, for purposes of determining tailer targets (to which DISes should this data be sent) and table data services (TDS) (from which DIS or service should this data be requested).

Important: Data Import Servers only supply online data. Claims and related filtering only apply to online data.

This new concept has the following implications (note that a DIS is implicitly also a TDS): Data Import Servers:

• If a claim is made, the implicit DIS filter is "have I claimed this table?"
• If no claims are made, the DIS filter is "is the table unclaimed?" plus any filter specified

When composing Table Data Services (sources tag):

• A failover group ([dis1, dis2]) is filtered with "is table unclaimed" + plus any filter given. Note that this ignores filters on the DISes themselves.
• If a filter is provided, that filter will be used for the named source.
• If a filter is not provided:
  • dataImportServers causes the section to be repeated for all applicable DISes.
  • A named failover group is filtered with "is claimed by this group".
  • A named DIS is filtered with that DIS's filter, which will be an is-claimed or is-not-claimed filter.
  • Anything else is delegating to another TDS and will be filtered with "is table unclaimed".

All the Data Import Servers

This release introduces a shortcut for referring to the Data Import Servers as a group. This can be useful for complex routing and filtering. It also means it is possible now to add new DIS sections without updating the TDS section.

The syntax below will direct the db_tdcp to include all DISes, except for db_rta and db_also_special.

  tableDataServices:
    db_tdcp:
      sources:
        - name: dataImportServers
          except:
            - db_rta
            - db_also_special

Note on "legacy format"

New features sometimes require new syntax. Deephaven strives to continue supporting data routing configuration files in "legacy" format, but new features are often unavailable until updates are made. This is generally painless, because if you want to use a new feature, you must ask for it.

However, if you change a section of the YML file to use a new feature, you may have to make additional changes to bring the section into the current file format.

claims is a new feature, and it requires the current format. Specifically here, if you add claims, you will likely also need to add an endpoint section. See the "Data Import Servers" section of "Data routing service configuration via YAML" in the documentation for more details.

For example, change this:

    dis1:
      host: host-value
      tailerPort: *default-tailerPort
      tableDataPort: *default-tableDataPort

to this:

    dis1:
      endpoint:
        serviceRegistry: none
        host: host-value
        tailerPort: *default-tailerPort
        tableDataPort: *default-tableDataPort

New YML tags

This release creates the following new tags in the data routing configuration file:

LiveDBPlot Support Removed

The LiveDBPlot and DBPlot classes have been removed from Deephaven 1.20230131. Users should migrate to new plotting methods introduced in 2017 which provide improved functionality and web support.

The default Groovy session previously inherited a static import for min that returned a double. The min function is no longer referenced from StaticGroovyImports allowing other min overloads to be accepted; thus min(1, 2) now returns an int rather than a double in the default Groovy session.

Parse-once builder-options added to JSON/Kafka

There are cases where a non-JSON message is encapsulated within a JSON message. A pair of new builder options are added to the JSON/Kafka adapter to allow the user to avoid parsing the embedded message multiple times.

For example, the following example parses the embedded FIX message each time a field is read from the message:

builder
    .addColumnToValueField("SomeTag", "someTag")
    .addColumnToValueFunction("MsgType", jsonRecord -> {
        // parse the FIX message, and get MsgType<35> from the header
        return FixMessage.parse(jsonRecord.getRecord().get("fixMsg")).getHeader().getField(35);
    })
    .addColumnToValueFunction("FixVersion", jsonRecord -> {
        // parse the FIX message again, and get FixVersion<8> from the header
        return FixMessage.parse(jsonRecord.getRecord().get("fixMsg")).getHeader().getField(8);
    })

The builder now provides the ability to parse the JSON message and store an arbitrary Object, which may be accessed multiple times without parsing multiple times:

builder
    .setParseOnce(jsonRecord -> {
        // parse the FIX message once, and store the value for later access
        return FixMessage.parse(jsonRecord.getRecord().get("fixMsg"));
    })
    .addColumnToValueField("SomeTag", "someTag")
    .addColumnToValueFunction("MsgType", jsonRecord -> {
        // access the pre-parsed object, and get MsgType<35> from the header
        return ((FixMessage) jsonRecord.getParsedObject()).getHeader().getField(35);
    })
    .addColumnToValueFunction("FixVersion", jsonRecord -> {
        // access the pre-parsed object, and get FixVersion<8> from the header
        return ((FixMessage) jsonRecord.getParsedObject()).getHeader().getField(8);
    })

C++ BinaryStoreWriter Produces Corrupt Application Version Records

If the BinaryStoreHeader setApplicationVersion record method was called, then a corrupt application version would be added to the header of the resultant binary log file. This meant that a C++ logger would have been unable to generate anything but a default "0" application version log format. This fix applies to all Deephaven versions, as the C++ binary logging library is independent of the Deephaven version.

Workspace Data Key Frames

The process that serves as persistent state storage for the Deephaven web console is backed by Deephaven tables in the DbInternal namespace. These tables grow a little every time a user updates their workspace and it is re-scanned each time the WebClientData query restarts. In the aggregate this can become a significant resource drain, so we have implemented a "snapshot" facility specific to this table.

New snapshots should be recorded periodically. The tool is designed to be run daily and will only create a new snapshot if sufficient changes have been made, or sufficient time has passed since the last snapshot. The following console command will attempt to do so and should be run in a merge server session or persistent query. The updateSnapshot() function returns true if a new snapshot was made, or false if there were not enough changes to justify a new snapshot.

wws = new com.illumon.iris.utils.WriteableWorkspaceSnapshot(log,db)
wws.updateSnapshot()

The internal logic steps are logged to the ProcessEventLog with LogEntry strings beginning with loadLatestSnapshot and updateSnapshot.

The following configuration parameters control the frequency of snapshot recording. A new snapshot is recorded only if at least one of the criteria are met.

WorkspaceDataSnapshot.daysSinceLastSnapshotThreshold  # default 7
WorkspaceDataSnapshot.changesToSnapshotSizeRatioThreshold # default 0.2 (20% new)

Added new DbInternal.ProcessTelemetry system table and TelemetryHelperQuery

The new ProcessTelemetry table enables monitoring UI performance within the Swing front-end. For each user-initiated action, the console logs the duration between the start of the operation and when the table is ready for use. By aggregating this telemetry, an overall picture of the system's health - as perceived by users - is available. Detailed information can then be used to investigate potential performance problems.

To write these events, the Swing console internally buffers the data and then sends it to the new TelemetryHelperQuery query. If the TelemetryHelperQuery is not running, data is buffered up to a configurable limit, at which point the oldest telemetry data is discarded.

Several new startup parameters define the Telemetry behavior for the console:

[service.name=iris_console] {
    # Identifies if telemetry metrics should be logged in the DbInternal.ProcessTelemetry table. To enable logging to
    # the remote table, set the followig property to `true`. Note that telemetry will still be logged to the local
    # client-logfile unless disabled with the `enableFor` options described below
    Telemetry.remoteTelemetry=false
    
    
    # Defines the frequency for messages to be sent to the server. Events will be batched and sent periodically. The
    # default frequence is 15s
    Telemetry.sendFrequencyMs=15_000
    
    # Defines the initial size of the buffer which stores messages to be sent in a batch, defaulting to 1,024
    Telemetry.initialSendBuffer=1_024
    
    # Defines the maximum number of messages to store and send in a single batch. New messages appended to the buffer
    # after it is full will cause "older" events to be removed. The default maximum value is 10,000
    Telemetry.maxSendBuffer=10_000
    
    
    # A number of classes will attempt to log telemetry locally and to the remote table. Individual classes may be
    # prevented from being logged by setting the following to `false`
    Telemetry.enableFor.DBTableModel=true
    Telemetry.enableFor.IrisTreeTableModel=true
    Telemetry.enableFor.DbOneClickPanel=true
}

A new method, telemetrySummary(), which accepts an optional "Date" parameter, has been added to the default Groovy session. The method will provide summary tables derived from the raw Telemetry data.

For new installations, an ACL is automatically applied to the DbInternal.ProcessTelemetry table. For an upgrade, an ACL editor must create a new "Table ACL" for the raw DbInternal.ProcessTelemetry data to be seen by unprivileged users. The ACL should be similar to the "allusers/ProcessEventLog" ACL, but for the ProcessTelemetry table:

allusers | DbInternal | ProcessTelemetry | new UsernameFilterGenerator("EffectiveUser")

Parameterized Query Lock changes

Parameterized Queries now use the shared LTM lock instead of the exclusive LTM lock. Query writers may also now instruct the query not to use any of the LTM locks with the requireComputeLock on the ParameterizedQueryBuilder.

When using no locks, the query writer must ensure that the Parameterized Query Action does not use any methods that require the LTM lock. If this parameter is set to false incorrectly, then results are undefined.

Add support for worker-scope plugin classpaths

Server processes now search in /etc/sysconfig/illumon.d/plugins/*/worker for server-only plugin jars and classpath entries in addition to searching for path items from /etc/sysconfig/illumon.d/plugins/*/global.

While global dependencies are included on both server and client classpaths, worker dependencies are only be added to server processes (any process using monit or the iris launch script, as well as any jvm started from a server python session). In particular, the client update service does not make JARs in the worker directory available to the Swing console.

Exporting Dashboards from the Web UI

Dashboards and their associated queries can now be exported from the Web UI. This allows you to easily archive, backup, or transfer dashboards between systems.

To export dashboards:

1. Go to the New tab screen.
2. Select the dashboards you want to export.
3. Click the Export button above the dashboard list.
4. Select whether to Export related queries along with the dashboards.
5. Click Export.

The browser then downloads a zip or archive file containing the dashboards (and queries, if specified).

Importing Dashboards from the Web UI

Dashboards and their associated queries can also be imported with the Web UI.

1. Go to the New tab screen.
2. Click the Import button above the dashboard list.
3. Select the zip or archive file containing the dashboards and queries you wish to import.
4. Make any modifications to the import data, if desired.
5. Click Import.

DnD Workers as PQs

DnD workers can now be started as first class PQs in Live Query, and Run and Done modes. The Community IDE can be launched from the Query Monitor tab of the Web UI.

Changes were made to the following classes that may affect customer code

PersistentQuery

The PersistentQuery class used to be where the worker control code was located for the Controller. That code has been extracted and moved into smaller more self contained classes. The PersistentQuery class remains as a non-constructable class that contains only static methods that can be used in PQ scripts, or ContextAwareRemoteQueries.

Custom Persistent Query Types

There is a new tag CommunityInitializer that defines the class that will be invoked once a ConsoleSession to a community worker has been established. If this tag is not present on a type, the Community engine will not be selectable when configuring those PQ types.

Due to the split up of the PersistentQuery class, the expected type for the ShutdownProcedure class has changed to Consumer<PersistentQueryHandle>>. Any customers using custom PQ configurations must update their code with this new signature.

In the PersistentQueryState class, details related to worker connections such as worker name, host, ports, and processInfoID have moved into the QueryProcessorConnectionDetails class, accessible through PersistentQueryState.getConnectionDetails().

Deephaven Enterprise supports Downstream Barrage

Deephaven Enterprise now supports subscribing to Barrage tables from Deephaven Community workers, anonymously to regular stock community workers, and authenticated via three-way-handshake with an authentication server token for DnD workers.

How to use

There are two different ways to subscribe to tables. The first, and simplest, is the URI method. In order to use this method simply add the following code to your Query:

import io.deephaven.uri.UriConfig
import io.deephaven.uri.ResolveTools
UriConfig.initDefault()

This will initialize the integration and prepare the system for connecting to Deephaven Community instances. Next, simply use ResolveTools.resolve(String uri) to subscribe to a table. The supported URIs can be found in the Community Reference.

For example, to subscribe to the table 'MarketData' in the Scope of a Community worker you might use:

MarketData = ResolveTools.resolve("dh://my.server.com:8888/scope/MarketData")

The first part of the URI selects SSL(dh:) or plaintext(dh+plain:), the second is the path to the server and its port. The next part selects either the Query Scope(/scope/) or application scope(/app/my_app/). Finally, the name of the table to subscribe to is the last part of the URI.

Finer Grained Control

The next method is more complicated, but provides finer grained control over the resulting subscription. You must create the individual components of the subscription as well as the gRPC channel and session for the Barrage exchange.

When using this method, take care to re-use the BarrageSession when you are going to make further subscriptions to tables within the same workers.

dndWorkerHost = 'myserver.com'
dndWorkerPort = 24003

import io.deephaven.client.impl.BarrageSession
import io.deephaven.client.impl.ClientConfig
import io.deephaven.client.impl.ChannelHelper
import io.deephaven.client.impl.SessionImplConfig
import io.deephaven.client.impl.SessionImpl
import io.deephaven.proto.DeephavenChannelImpl
import io.deephaven.qst.table.TicketTable
import io.deephaven.uri.DeephavenTarget
import io.deephaven.shadow.client.flight.io.grpc.ManagedChannel
import io.deephaven.shadow.client.flight.org.apache.arrow.memory.BufferAllocator
import io.deephaven.shadow.client.flight.org.apache.arrow.memory.RootAllocator
import io.deephaven.barrage.BarrageSubscriptionOptions
import io.deephaven.barrage.util.SessionUtil
import io.deephaven.enterprise.auth.AuthenticationClientManager
import io.deephaven.enterprise.auth.DhService

import java.util.concurrent.Executors
import java.util.concurrent.ScheduledExecutorService

bufferAllocator = new RootAllocator()
scheduler = Executors.newScheduledThreadPool(4)

deephavenTarget = DeephavenTarget.builder()
    .host(dndWorkerHost)
    .port(dndWorkerPort)
    .isSecure(true)
    .build()

clientConfig = ClientConfig.builder()
    .target(deephavenTarget)
    .build()

managedChannel = ChannelHelper.channel(clientConfig)

authToken = AuthenticationClientManager.getDefault().createToken(DhService.QUERY_PROCESSOR.serviceName())

authStr = SessionUtil.authenticateAndObtainAuthenticationInfo(managedChannel, authToken)

sessionConfig = SessionImplConfig.builder()
    .executor(scheduler)
    .channel(new DeephavenChannelImpl(managedChannel))
    .authenticationTypeAndValue(authStr)
    .build()

sessionImpl = SessionImpl.create(sessionConfig)

session = BarrageSession.of(sessionImpl, bufferAllocator, managedChannel)

// Use a prefix of "s/" before the variable name for the table in the remote worker
MarketData = session.subscribe(TicketTable.of("s/MarketData"), BarrageSubscriptionOptions.builder().build()).entireTable()

Column Rename Support from UI

The Schema Editor now supports ability to rename a column between the application log file and the table schema. The UI changes include how data type is handled for new columns in the Logger/Listener Column Details section. The default data type for new columns is not set and instead inherits the Intraday Type.

Below are example schemas for LoggerListener and a Listener only schema for a table with three columns.

The table below has three columns (Date, Destination and SameName) while The logger has two columns (SameName and Source).

• Date column is not present in the log file, but rather determined by the logging process.
• SameName column is in both the log file and table schema, and does not need to be transformed.
• Source column in the logger is renamed as Destination in the table.

To rename Source column as Destination, the Listener class should include both Source and Destination columns and their attributes should be:

• Source: A value of none for dbSetter attribute. This indicates that the column is not present in the table. Additionally, the attribute intradayType should be set to the appropriate dataType.
• Destination: A value of Source for dbSetter to identify its input source. A value none for intradayType means it is not present in the log file, and cannot be used as part of a dbSetter.

Schema with only Listener class

If the table has an externally generated log file (e.g., with a C++ logger), then you only need to define a Listener block to interpret the log file.

<Table namespace="ExampleNamespace" name="RenameColumn" storageType="NestedPartitionedOnDisk" >
  <Partitions keyFormula="${autobalance_single}"/>
  <Column name="Date" dataType="String" columnType="Partitioning" />
  <Column name="Destination" dataType="int" columnType="Normal" />
  <Column name="SameName" dataType="int" columnType="Normal" />

  <Listener logFormat="1" listenerPackage="com.illumon.iris.test.gen">
    <Column name="Destination" intradayType="none" dbSetter="Source" />
    <Column name="SameName" dataType="int" />
    <Column name="Source" intradayType="int" dbSetter="none" />
  </Listener>
</Table>

Schema with a LoggerListener

If you are generating a Java logger, then you should include a LoggerListener block in your schema.

<Table name="RenameColumn1001" namespace="ExampleNamespace" defaultMergeFormat="DeephavenV1" storageType="NestedPartitionedOnDisk">
  <Partitions keyFormula="${autobalance_single}" />
  
  <Column name="Date" dataType="String" columnType="Partitioning" />
  <Column name="Destination" dataType="Integer" />
  <Column name="SameName" dataType="Integer" />
  
  <LoggerListener logFormat="1" loggerClass="RenameColumn1001Logger" loggerPackage="com.illumon.iris.test.gen" 
                  rethrowLoggerExceptionsAsIOExceptions="false" tableLogger="false" generateLogCalls="true" 
                  verifyChecksum="true" listenerClass="RenameColumn1001Listener" listenerPackage="com.illumon.iris.test.gen">
    
    <SystemInput name="Source" type="int" />
    <SystemInput name="SameName" type="int" />
    
    <Column name="Destination" intradayType="none" dataType="int" dbSetter="Source" />
    <Column name="SameName" dataType="int" />
    <Column name="Source" dataType="int" dbSetter="none" />
  </LoggerListener>
</Table>

Renaming column examples for Blob and String data types.

The above pattern can be followed for all data types except for Blob and String. The differences are detailed below.

Renaming column for a Blob data type.

To rename column of a Blob data type, users need to provide the actual data type of the data stored in Blob. The Edit Logger/Listener Column UI now provides the ability to edit the data type field when the Intraday Type field is Blob. Note the data type field can only be changed if the current displayed value is none. In addition to setting the data type for the application log file column, the Listener column's dbSetter attribute must explicitly invoke cast as shown in the example schema below.

The example below shows Destination, SameName and Source columns of data type java.util.List.

<Table name="RenameColumn101" namespace="ExampleNamespace" defaultMergeFormat="DeephavenV1" storageType="NestedPartitionedOnDisk">
  <Partitions keyFormula="${autobalance_single}" />
  
  <Column name="Date" dataType="String" columnType="Partitioning" />
  <Column name="Destination" dataType="java.util.List" />
  <Column name="SameName" dataType="java.util.List" />
  
  <LoggerListener logFormat="1" loggerClass="RenameColumn101Logger" loggerPackage="com.illumon.iris.test.gen" 
                  rethrowLoggerExceptionsAsIOExceptions="false" tableLogger="false" generateLogCalls="true" verifyChecksum="true" 
                  listenerClass="RenameColumn101Listener" listenerPackage="com.illumon.iris.test.gen">
    
    <SystemInput name="Source" type="java.util.List" />
    <SystemInput name="SameName" type="java.util.List" />
    
    <Column name="Destination" intradayType="none" dataType="java.util.List" dbSetter="(java.util.List)blobToObject(Source)" />
    <Column name="SameName" intradayType="Blob" dataType="java.util.List" autoBlobInitSize="32000" />
    <Column name="Source" intradayType="Blob" dataType="java.util.List" dbSetter="none" autoBlobInitSize="256" autoBlobMaxSize="32000" />
  </LoggerListener>
</Table>

Renaming column for a String data type.

The basic steps are similar to all previous example except the Intraday Type for a String column is EnhancedString. This is reflected in the options available to select for Intraday Type. The list of valid Intraday Type options no longer includes String.

The dbSetter value for Destination column should include a toString() on the setter value as shown below in the example.

<Table name="RenameColumn02" namespace="ExampleNamespace" defaultMergeFormat="DeephavenV1" storageType="NestedPartitionedOnDisk">
    <Partitions keyFormula="${autobalance_single}"/>

    <Column name="Date" dataType="String" columnType="Partitioning"/>
    <Column name="Destination" dataType="String"/>
    <Column name="SameName" dataType="String"/>

    <LoggerListener logFormat="1" loggerClass="RenameColumn102Logger" loggerPackage="com.illumon.iris.test.gen"
                    rethrowLoggerExceptionsAsIOExceptions="false" tableLogger="false" generateLogCalls="true"
                    verifyChecksum="true" listenerClass="RenameColumn102Listener"
                    listenerPackage="com.illumon.iris.test.gen">

        <SystemInput name="Source" type="java.lang.String"/>
        <SystemInput name="SameName" type="java.lang.String"/>

        <Column name="Destination" intradayType="none" dataType="java.lang.String" dbSetter="Source.toString()"/>
        <Column name="SameName" dataType="java.lang.String"/>
        <Column name="Source" intradayType="EnhancedString" dataType="java.lang.String" dbSetter="none"/>
    </LoggerListener>
</Table>

New parameter for Replay Queries

The option to replay data at variable speeds has been added to Replay Queries. This new parameter can be leveraged in stress testing to simulate greater data loads.

New builder-methods for JSON/Kafka

A number of new builder-options have been added to the JSON/Kafka builder, which are included in javadoc for JsonConsumerRecordToTableWriterAdapter#Builder. The new builder fields allow unpacking/expanding JSON arrays into a number of rows, and allow for easy builder-defined parsing of nested JSON messages.
Additionally, we are now able to parse multiple JSON messages in parallel to improve overall throughput while still guaranteeing proper row-ordering.