Customize the installation

Caution

We are in the process of updating the information contained in this document. If you need assistance with the steps/processes described within, please contact Deephaven Support.

Configuration-as-Code

Configuration-as-Code is an important prerequisite to have reproducible and auditable configurations. It also makes configurations testable and easier to reuse.

This is especially important with distributed systems and micro services. Maintaining multiple services across a network of distributed systems, and keeping all their settings and configurations in sync is very hard to achieve when you have to manually configure everything by clicking through UI configuration screens.

It is therefore recommended to add the Deephaven configuration files to a version control system (VCS) such as Git. This allows updates to configuration files to be tested in a separate environment before deploying any configuration changes to your production Deephaven hosts.

Deephaven configuration files

The Deephaven configuration files are installed on the Deephaven server in:

/etc/sysconfig/illumon.d/

The following section provides a brief summary of the configuration files.

M/Monit Configurations:

/etc/sysconfig/illumon.d/monit/*.conf

Cron Configuration:

/etc/cron.d/illumon.iris.localhost

Host Configuration:

/etc/sysconfig/illumon.confs/illumon.iris.hostconfig

Deephaven Process Configurations:

/etc/sysconfig/illumon.d/resources/iris-common.prop

Authentication Configurations:

/etc/sysconfig/illumon.d/auth/keystore.authserver.pkcs12

/etc/sysconfig/illumon.d/auth/priv-authreconnect.base64.txt

/etc/sysconfig/illumon.d/auth/priv-iris.base64.txt

/etc/sysconfig/illumon.d/auth/priv-superuser.base64.txt

Schema Configurations:

/etc/sysconfig/illumon.d/schema/**/*.schema

/etc/sysconfig/illumon.d/schema/footer.schema

/etc/sysconfig/illumon.d/schema/header.schema

Installing custom classes

There are two methods by which customer classes can be added to your system: adding custom classes to the customer directory, or using the class push list. When Deephaven launches, these classes, such as custom widgets, will be available to end users to use in the console.

Customer directory

To make custom classes available in Deephaven, they must be archived into a JAR file (see "Creating JAR Files" below) and then added to /etc/sysconfig/illumon.d/java_lib.

After restarting the Client Update Service, the files from the server will automatically be downloaded to the client when the Deephaven Launcher runs and be usable in the console.

Adding JAR directories

Instead of copying JARs into /etc/sysconfig/illumon.d/java_lib, the Deephaven installation can also look for JAR files in existing customer directories. This is done by exporting the environment variable CUSTOMER_JAR_DIR with a comma-delimited list of directories in which JAR files may be contained. Any JAR files found in these directories will be added to the classpath. This is best done in the host configuration file. For example:

export CUSTOMER_JAR_DIR=/home/product1/java_lib/,/home/user/user1/java_lib

Class Push List

The class push list method is primarily intended for use by developers who are in the process of creating new classes for the Deephaven installation, or classes that only a small subset of users should have access to. For general, production use of custom classes, creating JAR files (see explanation below) for them and placing those JARs in the customer JAR directory is the recommended approach.

To utilize the class push list, a property must be defined for the console: RemoteQueryClient.defaultClassPushList=<listName>.txt. Upon console-worker connection, the specified text file is searched for on the client classpath. The file should contain fully qualified names of your classes, which should also be on the client classpath. These classes will be pushed to the server and ultimately to the associated worker, which will then be able to utilize custom classes from your client.

Creating JAR files

Compilation

First, compile your Java code:

javac -classpath "/path/to/iris/java_lib/*" -d /destination/path CustomClass.java...

  • -classpath "/path/to/iris/java_lib/*" - ensures that you are compiling your code against Deephaven dependencies.
  • -d /destination/path - generates the appropriate package-associated directories to the destination path upon compilation.
  • CustomClass.java... - the source code files to compile.

Archiving

Then, group your compiled code into Java archive format (.jar) files:

jar cf CustomClasses.jar FileToInclude...

  • cf CustomClasses.jar - creates a JAR file with the specified name.
  • FileToInclude... - can specify directories, which will in turn specify directory contents recursively.

Client Update Service configuration for the Deephaven Launcher

The Deephaven console may be viewed over VNC without any additional client software. However, a more common implementation is performed where users access the Deephaven console through an application called the Deephaven Launcher, which must be installed on individual users' computers.

The Deephaven Launcher uses the "Client Update Service" to access code and configuration settings from your Deephaven server. However, before installing and running the Launcher, the "Client Update Service" must be configured and enabled on the server using M/Monit. Follow these steps to enable the "Client Update Service":

  1. On the Deephaven Server, edit the /var/lib/iris/cus/iris/getdown.txt.pre file: Set the appbase value, replacing WEBHOST with the host address of your Deephaven Service.
    For example: 
    #appbase = http://WEBHOST/iris/
appbase = http://10.20.30.40/iris/
  2. In the M/Monit config folder, remove the .disabled extension from the Client Update Service config file name and run monit reload. This will instruct the M/Monit daemon to reread its configuration and re-initialize. 
    cd /etc/sysconfig/illumon.d/monit
mv 04-cus.conf.disabled 04-cus.conf
monit reload
  3. Check the status of the getdown service: 
    monit status client_update_service
  4. Once the "Client Update Service" is up and running, you can proceed to install and run the Launcher on client desktops. The installers for Windows, Mac and Linux desktops can be downloaded from the "Client Update Service" on your Deephaven Server at: http://<IRIS_SERVER_ADDRESS>:80

Deephaven process ports

Deephaven processes listen on various TCP ports and port ranges. All ports are configurable.

TCP ports can be configured in the iris-common file:

/etc/sysconfig/illumon.d/resources/iris-common.prop

The default Deephaven ports and port ranges follow.

TCP Ports:

  • 22013
  • 22012
  • 8084

Component: Remote Query Dispatcher (RQD)

Process name: db_query_server

Properties:

  • RemoteQueryDispatcherParameters.queryPort=22013
  • RemoteQueryDispatcher.workerPort=22012
  • RemoteQueryDispatcher.webPort=8084

TCP Ports:

  • 30002
  • 30003
  • 8085

Component: Remote Merge Dispatcher (RMD)

Process name: db_merge_server

Properties:

  • RemoteQueryDispatcherParameters.queryPort=30002
  • RemoteQueryDispatcher.workerPort=30003
  • RemoteQueryDispatcher.webPort=8085

TCP Ports: 23000-24999

Component: RQD Workers

Process name: RemoteQueryDispatcher_worker_<number>

Properties:

  • RemoteQueryDispatcher.workerServerPorts=23000-23999
  • RemoteQueryDispatcher.workerServerWebsocketPorts=24000-24999

TCP Ports:

  • 32000-32999
  • 25000-25999

Component: RMD Workers

Process name: RemoteQueryDispatcher_worker_<number>

Properties:

  • RemoteQueryDispatcher.workerServerPorts=32000-32999
  • RemoteQueryDispatcher.workerServerWebsocketPorts=25000-25999

TCP Ports:

  • 22021
  • 22015

Component: Data Import Server

Process name: db_dis

Properties:

routing_service.yml:

dataImportServers:
   db_dis:
     tailerPort: 22021
     tableDataPort: 22015

TCP Ports: 22020

Component: Log Aggregator Server

Process name: log_aggregator_service

Properties:

routing_service.yml:

logAggregatorServers:
  log_aggregator_service:
     port: 22020

TCP Ports: 22014

Component: Local Data Table

Process name: db_ltds

Properties:

routing_service.yml:

 tableDataServices:
   db_ltds:
     port: 22014

TCP Ports: 22016

Component: Table Data Cache Proxy

Process name: db_tdcp

Properties:

routing_service.yml:

tableDataServices:
  db_tdcp:
     port: 22016

TCP Ports: 22023

Component: Configuration Server (Centralized Schema Service, Data Routing Service and Configuration Service)

Process name: configuration_server

Properties: configuration.server.port=22023

TCP Ports:

  • 9030
  • 9031

Component: User Authentication Server

Process name: authentication_server

Properties:

  • authentication.server.port.plaintext=9030
  • authentication.server.port.ssl=9031

TCP Ports:

  • 9040
  • 9041

Component: User Access Control Server

Process name: db_acl_write_server

Properties:

  • dbaclwriter.port=9040
  • dbaclwriter.ssl.port=9041

TCP Ports: 20126

Component: Persistent Query Controller

Process name: iris_controller

Properties: PersistentQueryController.port=20126

TCP Ports: 20021 (outbound)

Component: Log Tailer

Process name: tailer1..tailerN

Properties: This is determined by the db_dis section (see above).

TCP Ports: 80/443

Component: Client Update Server

Process name: client_update_service

Properties:

client-update-service.conf:

  server.port = 80

TCP Ports: 8123

Component: Web API Service

Process name: web_api_service

Properties: Webapi.server.port=8123

TCP Ports: 3306

Component: MySQL Database

Process name: mariadb_server

Properties:

/etc/my.cnf:

  port=3306 # default

TCP Ports: 2812

Component: M/Monit Daemon

Process name: monit

Properties:

/etc/monitrc:

  set httpd port 2812