System administration

This cheat sheet provides quick access to common commands and procedures for system administrators.

Management

Service status and control

Check that all Deephaven services have started and are available.

/usr/illumon/latest/bin/dh_monit up --block

Get a summary of all Deephaven services status:

/usr/illumon/latest/bin/dh_monit summary

Restart all Deephaven services (run on every machine in the cluster; dh_monit only manages the local machine):

/usr/illumon/latest/bin/dh_monit restart all

Restart a specific service:

sudo -u irisadmin monit restart <service_name>
# Example: sudo -u irisadmin monit restart db_query_server

Start a service interactively with debug logging:

sudo su - irisadmin
/usr/illumon/latest/bin/iris --debug start <process_name>

Verify cluster connection

# For password-based authentication
/usr/illumon/latest/bin/check-deephaven-cluster --connection-json https://deephaven-host:8000/iris/connection.json --user YOUR_USERNAME

# For public key authentication
/usr/illumon/latest/bin/check-deephaven-cluster --connection-json https://deephaven-host:8000/iris/connection.json --key-file /path/to/private-key.txt

etcd management

Check etcd endpoint status:

sudo -u irisadmin /usr/illumon/latest/bin/etcdctl.sh endpoint status --write-out table

List etcd roles:

sudo -u irisadmin /usr/illumon/latest/bin/etcdctl.sh role list

Defragment etcd node (to reclaim disk space). This takes the target node offline, reducing the cluster's fault tolerance (or risking quorum loss if another node also fails) until it rejoins, so only run it while Deephaven is down:

# Stop the etcd service on the target node
sudo systemctl stop dh-etcd.service

# Ensure the member/snap subdirectory exists (replace <etcd_cluster_id> with your etcd cluster's
# ID/token, shared by every node -- see: grep cluster-token /etc/etcd/dh/latest/config.yaml)
sudo -u etcd -g etcd mkdir -p /var/lib/etcd/dh/<etcd_cluster_id>/member/snap

# Run defragmentation
sudo -u etcd -g etcd /bin/etcdctl defrag --data-dir=/var/lib/etcd/dh/<etcd_cluster_id>

# Restart the etcd service
sudo systemctl start dh-etcd.service

MariaDB/MySQL management

If MySQL is used for ACLs, these commands are useful:

# Check MariaDB/MySQL service status
sudo systemctl status mariadb

# View databases
sudo mysql -e "show databases"

# View ACL tables
sudo mysql -D dbacl_iris -e "show tables"

# View table ACLs
sudo mysql -D dbacl_iris -e "select * from tableacls"

# Restart MariaDB
sudo systemctl restart mariadb

Certificate management

Inspect certificates

View a certificate's details including validity period and SAN names. For native installations, this depends on whether Envoy is enabled: with Envoy, clients see the certificate at DH_ENVOY_PEM_PATH, which defaults to /etc/deephaven/certs/envoy.pem; without Envoy, they see the per-machine PEM file named after /etc/deephaven/DH_NODE_NAME (see Certificate file input locations for Podman and Kubernetes):

# with Envoy enabled (default native path; see DH_ENVOY_PEM_PATH if customized):
sudo openssl x509 -in /etc/deephaven/certs/envoy.pem -noout -text

# without Envoy:
sudo openssl x509 -in /etc/deephaven/certs/$(< /etc/deephaven/DH_NODE_NAME).pem -noout -text

Check a remote server's certificate:

openssl s_client -connect your.server.com:443 -servername your.server.com | openssl x509 -text -noout

Verify hostname in a certificate:

openssl s_client -connect your.server.com:443 | grep -A1 'subject\|Subject Alternative Name'

Replace certificates

Rotate Deephaven certificates on a native (non-Kubernetes) installation. See Rotate certificates for native installations for full details, and Restarting Envoy for the restart command for your Envoy install mode:

# 1. If you supply your own per-machine certificates, update the certificate inputs in
#    DH_LOCAL_CERT_DIR on the Installation Host (<node>.pem and <node>.key for each machine).
#    Skip this if you rely on Deephaven-generated per-machine certificates (the default
#    when Envoy is enabled) - dh_keygen.sh regenerates them automatically. Update
#    DH_ENVOY_LOCAL_CERT if Envoy is used.

# 2. From the Installation Host, run dh_keygen.sh (generated alongside master_install.sh) to
#    copy/generate the certificates on every machine and rebuild the keystores/truststores
#    under /etc/sysconfig/deephaven/*. This does not require a full reinstall.
./dh_keygen.sh

# 3. Restart affected services; dh_keygen.sh does not restart any Deephaven service
#    (it can conditionally restart MariaDB during first-time ACL bootstrap, unrelated
#    to certificate rotation). Restart Envoy using the command for your install mode.
#    dh_monit only manages the local machine, so run this on every machine in the cluster.
/usr/illumon/latest/bin/dh_monit restart all

Convert a DER certificate to PEM format:

sudo openssl x509 -inform der -in certificate.cer -out tls.crt

Trust certificates in Java

Import a certificate to the Java truststore:

keytool -cacerts -trustcacerts -import -file <certificate_file> -alias <unique_alias>

Create a PKCS12 truststore:

# Create a password file
echo -n "your_password" | base64 > truststore_passphrase

# Import certificate
keytool -importcert -keystore truststore-iris.p12 -storepass $(cat truststore_passphrase | base64 -d) -alias your_server -file your_cert.pem

View certificates in a truststore:

keytool -list -keystore truststore-iris.p12 -storepass $(cat truststore_passphrase | base64 -d)

Users & groups

See ACL management for more information.

Create a user

/usr/illumon/latest/bin/dhconfig acls users add --name <username>

Delete a user

/usr/illumon/latest/bin/dhconfig acls users delete --name <username>

List users

/usr/illumon/latest/bin/dhconfig acls users list

Set a user password

# Set a password interactively
/usr/illumon/latest/bin/dhconfig acls users set-password --name <username>

# Set a password using a pre-hashed value
/usr/illumon/latest/bin/dhconfig acls users set-password --name <username> --hashed-password $(openssl passwd -apr1 'YOUR_PASSWORD')

Create a group

/usr/illumon/latest/bin/dhconfig acls groups add --name <groupname>

Delete a group

/usr/illumon/latest/bin/dhconfig acls groups delete --name <groupname>

List groups

/usr/illumon/latest/bin/dhconfig acls groups list

Add a user to a group

/usr/illumon/latest/bin/dhconfig acls groups add-member --name <username> --group <groupname>

Remove a user from a group

/usr/illumon/latest/bin/dhconfig acls groups remove-member --name <username> --group <groupname>

Backup, restore, & migration

See the backup and restore guide for more information.

Run the backup script

This example creates a full backup in a specified directory and sets a 7-day retention policy for old backups.

/usr/illumon/latest/bin/backup_deephaven --directory /path/to/backup/dir --all --retain 7

Common backup script parameters:

-d, --directory: Specify the backup directory.
-u, --user: Specify the user account under which the backups will be run.
-a, --all: Back up everything (etcd, ACLs, PQs, schemas, properties, routing, workspace data).
-e, --etcd: Back up the etcd database.
-X, --no-etcd: Skip the etcd snapshot when backing up everything with -a/--all.
-A, --ACLs: Back up Access Control Lists.
-p, --persistentQueries: Back up Persistent Queries.
-w, --workspaceData: Back up the WorkspaceData table.
-j, --jvm-arg: Specify an extra JVM argument for the workspace data tool (e.g. -j -Xmx4g). May be repeated.
-s, --schema: Back up schemas.
-P, --propertyFiles: Back up property files.
-r, --routing: Back up the routing file(s).
-R, --retain: Delete backups older than a specified number of days.

Restart all services

After restoring a backup or migrating a system, restart all Deephaven services on all nodes. dh_monit only manages the local machine, so run this on every node:

/usr/illumon/latest/bin/dh_monit restart all

Monitoring & troubleshooting

Check text logs

Standard text logs for Deephaven services are located in sub-directories under /var/log/deephaven.

# View current process log (example for query server)
cat /var/log/deephaven/query_server/RemoteQueryDispatcher.log.current

# List all logs for a service (example for authentication server)
ls -ltr /var/log/deephaven/authentication_server/authentication_server.log.????-??-??

# View process launch logs (contains startup info and command line)
cat /var/log/deephaven/process_name/process_name.log.YYYY-MM-DD

View and search binary logs

The iriscat utility converts binary logs to a readable text format. This is useful when you cannot access a console.

# View a specific binary log file
/usr/illumon/latest/bin/iriscat -l /var/log/deephaven/binlogs/pel/DbInternal.ProcessEventLog.System.deephaven-query-1.YYYY-MM-DD.bin.YYYY-MM-DD.214340.599+0000

# Search all binary logs for a specific ProcessInfoId and page the results
/usr/illumon/latest/bin/iriscat -l /var/log/deephaven/binlogs/pel/DbInternal.ProcessEventLog.System.*.YYYY-MM-DD* | grep 'PROCESS_INFO_ID' | sort | less

Troubleshoot process startup

Check the status of all services managed by M/Monit.

/usr/illumon/latest/bin/dh_monit summary

If a service is failing, you can attempt to start it interactively with debug logging to get more information.

# Run as the irisadmin user
sudo su - irisadmin

# Start the service in debug mode
/usr/illumon/latest/bin/iris --debug start <process_name>

Persistent Query (PQ) management

Determine the active PQ controller

In a multi-controller setup, use this command to find which controller is the current leader.

/usr/illumon/latest/bin/dhconfig pq leader

Reload PQ controller configuration

Dynamically reload the PQ controller's configuration without a restart.

/usr/illumon/latest/bin/dhconfig pq reload

Internal tables

Deephaven uses several internal tables to log system events, metrics, and state. These tables are located in the DbInternal namespace and can be queried for monitoring and troubleshooting.

Key internal tables

AuditEventLog: Records security-related events such as logins, logouts, and permission changes.
ProcessEventLog: Contains detailed log messages from all Deephaven processes and workers. Essential for debugging.
PersistentQueryStateLog: Tracks the state changes of all Persistent Queries (e.g., starting, stopping, failing).
WorkspaceData: Stores saved user details from the web interface, including workspaces, dashboards, draft Persistent Queries, and command history.

Querying internal tables

You can query these tables using Python or Groovy. Here is a basic example to retrieve today's data from a table.

// Replace 'TableName' with the actual table name (e.g., AuditEventLog)
def table = db.liveTable("DbInternal", "TableName").where("Date=today()")

# Replace 'TableName' with the actual table name (e.g., AuditEventLog)
table = db.live_table("DbInternal", "TableName").where("Date=today()")

Table and storage management

Deephaven's data is stored on the filesystem, typically under the /db directory. Understanding the layout is key for administration.

Filesystem layout

The /db directory contains several key subdirectories:

/db/Systems: Stores historical data for system namespaces.
/db/Users: Stores data for user namespaces.
/db/Intraday: Stores real-time, intraday data for system namespaces.

DbInternal, which contains internal system tables used for logging, monitoring, and state management, is a system namespace, not a top-level directory. Its data lives under /db/Systems/DbInternal (historical) and /db/Intraday/DbInternal (intraday), like any other system namespace.

Example directory structure

A simplified view of the directory structure for a namespace. Partitions is shared by every table in the namespace; each table gets its own subdirectory under each partition/date:

/db/Systems/
└── MyNamespace/
    ├── Partitions/
    │   └── 0/
    │       ├── 2023-10-26/
    │       │   └── MyTable/
    │       └── 2023-10-27/
    │           └── MyTable/
    └── WritablePartitions/
        └── 0 -> ../Partitions/0

Table data is broken into partitions, often by date.
Each table has a .tbl metadata file (not shown) that defines its schema.

Performance tuning

JVM memory settings

View the current JVM memory settings for a Deephaven service:

# Find the Java process ID
pgrep -f 'db_query_server'

# View JVM memory settings
jcmd <PID> VM.flags | grep -E 'Xms|Xmx'

Adjust JVM memory settings in the customer-owned hostconfig file (see Process memory configuration for details):

sudo vim /etc/sysconfig/deephaven/illumon.iris.hostconfig
# Add a case clause for the target process, appending to EXTRA_ARGS and
# prefixing each JVM argument with -j. Example: increase db_query_server heap to 8g
db_query_server)
    EXTRA_ARGS="$EXTRA_ARGS -j -Xms8g -j -Xmx8g"
    ;;

System resource monitoring

Monitor system resources used by Deephaven processes:

# CPU and memory usage for all Deephaven processes
top -u irisadmin

# Disk I/O statistics
iostat -xz 1

# Network usage statistics
netstat -tpn | grep java

Networking configuration

For more information on Deephaven networking and proxy setup, see:

Verify network connectivity

Test connectivity to Deephaven services:

# Test basic HTTP connectivity
curl -Ik https://deephaven-host:8000/iris/

# Test WebSocket connectivity (requires wscat tool)
wscat -c wss://deephaven-host:8000/socket

Check open ports and services

# List all open ports and associated services
sudo netstat -tulpn | grep -E ':(8000|9030|9031|2379)'

# Check if specific Deephaven ports are open
sudo ss -tulpn | grep -E ':(8000|9030|9031|2379)'

Configure Envoy as a front proxy

Envoy can simplify network configuration by exposing only one external port for all Deephaven services:

# Check Envoy status if installed (root-managed unit)
sudo systemctl status deephaven_envoy

# For a native/user-managed install, the unit runs as DH_ADMIN_USER (irisadmin by default)
# and requires XDG_RUNTIME_DIR to be set:
sudo -u irisadmin XDG_RUNTIME_DIR=/run/user/$(id -u irisadmin) \
    systemctl --user status deephaven_envoy

# View Envoy configuration
cat /etc/sysconfig/illumon.d/resources/envoy3.yaml

# Restart Envoy after configuration changes (root-managed unit)
sudo systemctl restart deephaven_envoy

# For a native/user-managed install:
sudo -u irisadmin XDG_RUNTIME_DIR=/run/user/$(id -u irisadmin) \
    systemctl --user restart deephaven_envoy

Troubleshoot network issues

# Check firewall status
sudo firewall-cmd --list-all

# Test network latency to Deephaven server
ping -c 5 deephaven-host

# Check DNS resolution
nslookup deephaven-host

Health check commands

System health checks

Quickly check the health of the system and services:

# Check overall system resource usage
top -b -n 1

# Check disk space usage
df -h

# Check memory usage
free -h

# Check load average
uptime

Deephaven service health checks

Verify that Deephaven services are running properly:

# Use monit to check service health
sudo -u irisadmin monit status

# Check if query server is responding
curl -ik https://localhost:8000/iris/connection.json

# Check that the authentication server can authenticate (no dedicated HTTP health endpoint exists)
sudo -u irisadmin /usr/illumon/latest/bin/iris check_auth_server

Log health checks

Monitor logs for errors and warnings:

# Check for errors in query server logs
grep -i error /var/log/deephaven/query_server/RemoteQueryDispatcher.log.current

# Check for warnings in authentication server logs
grep -i warn /var/log/deephaven/authentication_server/AuthenticationServer.log.current

# Monitor logs in real-time
tail -f /var/log/deephaven/query_server/RemoteQueryDispatcher.log.current

Database and storage checks

Verify database and storage systems:

# Check etcd cluster health
sudo -u irisadmin /usr/illumon/latest/bin/etcdctl.sh endpoint health --write-out table

# Check MariaDB status (if used for ACLs)
sudo systemctl status mariadb

# Check filesystem status where Deephaven data is stored
sudo xfs_info /db

Security hardening

File permissions audit

Verify and fix permissions on key Deephaven directories:

# Check permissions on config directories
ls -la /etc/deephaven/
ls -la /etc/sysconfig/illumon.d/

# Fix permissions if needed (installer default is irisadmin:dbmergegrp, mode 775, shared with the
# dbquery/dbmerge service accounts). Do not change this recursively: /etc/deephaven/certs has its
# own tighter irisadmin:irisadmin, mode 700 ownership that a recursive chown would overwrite.
sudo chown irisadmin:dbmergegrp /etc/deephaven/
sudo chmod 775 /etc/deephaven/

Authentication configuration

The current recommended way to manage public-key authentication is to import keys into the ACL store rather than a file:

# <public key file> must be a Deephaven public-key file, not a generic PEM/OpenSSH key:
# each line is "<username> <base64-encoded key>", e.g. the pub-<username>.base64.txt
# file generated by generate-iris-keys
/usr/illumon/latest/bin/dhconfig acls publickeys import --file <public key file>

Deephaven properties are stored in etcd and edited by exporting, editing, and re-importing the properties file (see Edit iris-environment.prop). The authentication.server.authorizedkeys.enabled property controls the legacy, deprecated dsakeys.txt file-based keystore (see Deprecated filesystem-based public keystore); prefer the ACL store above instead. To inspect or change it:

# Export the properties file to inspect/edit
/usr/illumon/latest/bin/dhconfig properties export --directory /tmp iris-environment.prop

# Check if the legacy authorized-keys file is enabled
grep -A2 "authentication.server.authorizedkeys.enabled" /tmp/iris-environment.prop

# Change it (edit /tmp/iris-environment.prop), then import and reload the running Authentication Servers
vi /tmp/iris-environment.prop
sudo -u irisadmin /usr/illumon/latest/bin/dhconfig properties import /tmp/iris-environment.prop
sudo -u irisadmin /usr/illumon/latest/bin/iris auth_server_reload_tool

Network security

Configure firewall to restrict access to Deephaven services:

# Check current firewall rules
sudo firewall-cmd --list-all

# Add rules to restrict access to Deephaven ports
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="trusted-subnet/24" port port="8000" protocol="tcp" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="trusted-subnet/24" port port="9031" protocol="tcp" accept'

# Reload firewall rules
sudo firewall-cmd --reload

System account security

Audit and secure system accounts:

# Check if irisadmin account is locked for direct login
sudo passwd -S irisadmin

# Lock system accounts not needed for direct login
sudo passwd -l irisadmin

# Check sudo permissions
sudo cat /etc/sudoers.d/deephaven