---
title: System administration
---

This cheat sheet provides quick access to common commands and procedures for system administrators.

## Management

### Service status and control

Check that all Deephaven services have started and are available.

```bash
/usr/illumon/latest/bin/dh_monit up --block
```

Get a summary of all Deephaven services status:

```bash
/usr/illumon/latest/bin/dh_monit summary
```

Restart all Deephaven services:

```bash
/usr/illumon/latest/bin/dh_monit restart all
```

Restart a specific service:

```bash
sudo -u irisadmin monit restart <service_name>
# Example: sudo -u irisadmin monit restart db_query_server
```

Start a service interactively with debug logging:

```bash
sudo su - irisadmin
/usr/illumon/latest/bin/iris --debug start <process_name>
```

### Verify cluster connection

```shell
# For password-based authentication
/usr/illumon/latest/bin/check-deephaven-cluster --connection-json https://deephaven-host:8000/iris/connection.json --username YOUR_USERNAME

# For public key authentication
/usr/illumon/latest/bin/check-deephaven-cluster --connection-json https://deephaven-host:8000/iris/connection.json --key-file /path/to/private-key.txt
```

### etcd management

Check [etcd](../sys-admin/core-components/etcd.md) endpoint status:

```bash
sudo -u irisadmin /usr/illumon/latest/bin/etcdctl.sh endpoint status --write-out table
```

List etcd roles:

```bash
sudo -u irisadmin /usr/illumon/latest/bin/etcdctl.sh role list
```

Defragment etcd node (to reclaim disk space):

```bash
# Stop the etcd service on the target node
sudo systemctl stop dh-etcd.service

# Run defragmentation (replace <etcd_data_directory> with your actual etcd data directory)
sudo -u etcd -g etcd /bin/etcdctl defrag --data-dir=/var/lib/etcd/dh/<etcd_data_directory>

# Restart the etcd service
sudo systemctl start dh-etcd.service
```

### MariaDB/MySQL management

If [MySQL](../sys-admin/permissions/admin-user.md) is used for ACLs, these commands are useful:

```bash
# Check MariaDB/MySQL service status
sudo systemctl status mariadb

# View databases
sudo mysql -e "show databases"

# View ACL tables
sudo mysql -D dbacl_iris -e "show tables"

# View table ACLs
sudo mysql -D dbacl_iris -e "select * from tableacls"

# Restart MariaDB
sudo systemctl restart mariadb
```

### Certificate management

#### Inspect certificates

View a [certificate's](../sys-admin/troubleshooting/troubleshooting-certificates.md) details including validity period and SAN names:

```bash
sudo openssl x509 -in /etc/deephaven/cus-tls/tls.crt -noout -text
```

Check a remote server's certificate:

```bash
openssl s_client -connect your.server.com:443 -servername your.server.com | openssl x509 -text -noout
```

Verify hostname in a certificate:

```bash
openssl s_client -connect your.server.com:443 | grep -A1 'subject\|Subject Alternative Name'
```

#### Replace certificates

Replace Deephaven certificates on an infra server (non-Kubernetes):

```bash
# 1. Copy new tls.crt and tls.key files to /etc/deephaven/cus-tls
# 2. Generate new web certificates and package them
/usr/illumon/latest/install/iris_keygen.sh --force-new-web
/usr/illumon/latest/install/config_packager.sh query package

# 3. Create lighttpd.pem file for Envoy
source /usr/illumon/latest/bin/dh_users
sudo -u "${DH_ADMIN_USER}" cat /etc/deephaven/cus-tls/tls.key /etc/deephaven/cus-tls/tls.crt | \
    sudo -u "${DH_ADMIN_USER}" tee /etc/sysconfig/illumon.d/client_update_service/lighttpd.pem > /dev/null

# 4. Restart Web API service and query servers
/usr/illumon/latest/bin/dh_monit restart web_api_service
```

Convert a DER certificate to PEM format:

```bash
openssl x509 -inform der -in certificate.cer -out tls.crt
```

#### Trust certificates in Java

Import a certificate to the Java truststore:

```bash
keytool -cacerts -trustcacerts -import -file <certificate_file> -alias <unique_alias>
```

Create a PKCS12 truststore:

```bash
# Create a password file
echo -n "your_password" | base64 > truststore_passphrase

# Import certificate
keytool -importcert -keystore truststore-iris.p12 -storepass $(cat truststore_passphrase | base64 -d) -alias your_server -file your_cert.pem
```

View certificates in a truststore:

```bash
keytool -list -keystore truststore-iris.p12 -storepass $(cat truststore_passphrase | base64 -d)
```

## Users & groups

See [ACL management](../sys-admin/permissions/permissions-overview.md) for more information.

### Create a user

```bash
/usr/illumon/latest/bin/dhconfig acl users add --name <username>
```

### Delete a user

```bash
/usr/illumon/latest/bin/dhconfig acl users delete --name <username>
```

### List users

```bash
/usr/illumon/latest/bin/dhconfig acl users list
```

### Set a user password

```bash
# Set a password interactively
/usr/illumon/latest/bin/dhconfig acl users set-password --name <username>

# Set a password using a pre-hashed value
/usr/illumon/latest/bin/dhconfig acl users set-password --name <username> --hashed-password $(openssl passwd -apr1 'YOUR_PASSWORD')
```

### Create a group

```bash
/usr/illumon/latest/bin/dhconfig acl groups add --name <groupname>
```

### Delete a group

```bash
/usr/illumon/latest/bin/dhconfig acl groups delete --name <groupname>
```

### List groups

```bash
/usr/illumon/latest/bin/dhconfig acl groups list
```

### Add a user to a group

```bash
/usr/illumon/latest/bin/dhconfig acl groups add-member --name <groupname> --member <username>
```

### Remove a user from a group

```bash
/usr/illumon/latest/bin/dhconfig acl groups remove-member --name <groupname> --member <username>
```

## Backup, restore, & migration

See the [backup and restore guide](../sys-admin/backup-restore-migrate/backup-and-restore.md) for more information.

### Run the backup script

This example creates a full backup in a specified directory and sets a 7-day retention policy for old backups.

```bash
/usr/illumon/latest/bin/backup_deephaven --directory /path/to/backup/dir --all --retain 7
```

**Common backup script parameters:**

- `-d`, `--directory`: Specify the backup directory.
- `-a`, `--all`: Back up everything (etcd, ACLs, PQs, schemas, properties, routing, workspace data).
- `-e`, `--etcd`: Back up the etcd database.
- `-A`, `--ACLs`: Back up Access Control Lists.
- `-p`, `--persistentQueries`: Back up Persistent Queries.
- `-w`, `--workspaceData`: Back up the `WorkspaceData` table.
- `-R`, `--retain`: Delete backups older than a specified number of days.

### Restart all services

After restoring a backup or migrating a system, restart all Deephaven services on all nodes.

```bash
/usr/illumon/latest/bin/dh_monit restart all
```

## Monitoring & troubleshooting

### Check text logs

Standard text logs for Deephaven services are located in sub-directories under `/var/log/deephaven`.

```bash
# View current process log (example for query server)
cat /var/log/deephaven/query_server/db_query_server.log.current

# List all logs for a service (example for authentication server)
ls -ltr /var/log/deephaven/authentication_server/authentication_server.log.????-??-??

# View process launch logs (contains startup info and command line)
cat /var/log/deephaven/process_name/process_name.log.YYYY-MM-DD
```

### View and search binary logs

The `iriscat` utility converts binary logs to a readable text format. This is useful when you cannot access a console.

```bash
# View a specific binary log file
/usr/illumon/latest/bin/iriscat -l /var/log/deephaven/binlogs/pel/DbInternal.ProcessEventLog.System.deephaven-query-1.YYYY-MM-DD.bin.YYYY-MM-DD.214340.599+0000

# Search all binary logs for a specific ProcessInfoId and page the results
/usr/illumon/latest/bin/iriscat -l /var/log/deephaven/binlogs/pel/DbInternal.ProcessEventLog.System.*.YYYY-MM-DD* | grep 'PROCESS_INFO_ID' | sort | less
```

### Troubleshoot process startup

Check the status of all services managed by M/Monit.

```bash
/usr/illumon/latest/bin/dh_monit summary
```

If a service is failing, you can attempt to start it interactively with debug logging to get more information.

```bash
# Run as the irisadmin user
sudo su - irisadmin

# Start the service in debug mode
/usr/illumon/latest/bin/iris --debug start <process_name>
```

## Persistent Query (PQ) management

### Determine the active PQ controller

In a multi-controller setup, use this command to find which controller is the current leader.

```bash
/usr/illumon/latest/bin/dhconfig pq leader
```

### Reload PQ controller configuration

Dynamically reload the PQ controller's configuration without a restart.

```bash
/usr/illumon/latest/bin/dhconfig pq reload
```

## Internal tables

Deephaven uses several [internal tables](../sys-admin/internal-tables/internal-tables.md) to log system events, metrics, and state. These tables are located in the `DbInternal` namespace and can be queried for monitoring and troubleshooting.

### Key internal tables

- [`AuditEventLog`](../sys-admin/internal-tables/audit-event-log.md): Records security-related events such as logins, logouts, and permission changes.
- [`ProcessEventLog`](../sys-admin/internal-tables/process-event-log.md): Contains detailed log messages from all Deephaven processes and workers. Essential for debugging.
- [`PersistentQueryStateLog`](../sys-admin/internal-tables/persistent-query-state-log.md): Tracks the state changes of all Persistent Queries (e.g., starting, stopping, failing).
- [`WorkspaceData`](../sys-admin/internal-tables/workspace-data.md): Stores user-created content, including dashboards, notebooks, and layouts.

### Querying internal tables

You can query these tables using Python or Groovy. Here is a basic example to retrieve today's data from a table.

```python
# Replace 'TableName' with the actual table name (e.g., AuditEventLog)
table = db.live_table("DbInternal", "TableName").where("Date=today()")
```

```groovy
// Replace 'TableName' with the actual table name (e.g., AuditEventLog)
def table = db.liveTable("DbInternal", "TableName").where("Date=today()")
```

## Table and storage management

Deephaven's data is stored on the filesystem, typically under the `/db` directory. Understanding the layout is key for administration.

### Filesystem layout

The `/db` directory contains several key subdirectories:

- **`/db/Systems`**: Stores historical data for system namespaces.
- **`/db/Users`**: Stores data for user namespaces.
- **`/db/Intraday`**: Stores real-time, intraday data for system namespaces.
- **`/db/DbInternal`**: Contains internal system tables used for logging, monitoring, and state management.

### Example directory structure

A simplified view of the directory structure for a partitioned table:

```text
/db/Systems/
└── MyNamespace/
    └── MyTable/
        ├── Partitions/
        │   └── 0/
        │       ├── 2023-10-26/
        │       └── 2023-10-27/
        └── WritablePartitions/
            └── 0 -> ../Partitions/0
```

- Table data is broken into partitions, often by date.
- Each table has a `.tbl` metadata file (not shown) that defines its schema.

## Performance tuning

### JVM memory settings

View the current JVM memory settings for a Deephaven service:

```bash
# Find the Java process ID
pgrep -f 'db_query_server'

# View JVM memory settings
jcmd <PID> VM.flags | grep -E 'Xms|Xmx'
```

Adjust JVM memory settings in the service configuration file:

```bash
sudo vim /etc/sysconfig/illumon.d/db_query_server/irisdbserver.conf
# Edit the JAVA_OPTS line to adjust -Xms (initial heap) and -Xmx (max heap)
# Example: JAVA_OPTS="-Xms4g -Xmx8g ..."
```

### System resource monitoring

Monitor system resources used by Deephaven processes:

```bash
# CPU and memory usage for all Deephaven processes
top -u irisadmin

# Disk I/O statistics
iostat -xz 1

# Network usage statistics
netstat -tpn | grep java
```

## Networking configuration

For more information on Deephaven networking and proxy setup, see:

- [Envoy as a front proxy](../sys-admin/envoy/envoy.md)
- [Configuring Envoy](../sys-admin/envoy/configuring-envoy.md)
- [Web UI troubleshooting](../sys-admin/troubleshooting/web-ui-troubleshooting.md)

### Verify network connectivity

Test connectivity to Deephaven services:

```bash
# Test basic HTTP connectivity
curl -Ik https://deephaven-host:8000/iris/

# Test WebSocket connectivity (requires wscat tool)
wscat -c wss://deephaven-host:8000/ws/iris/iris.api
```

### Check open ports and services

```bash
# List all open ports and associated services
sudo netstat -tulpn | grep -E ':(8000|10000|2379)'

# Check if specific Deephaven ports are open
sudo ss -tulpn | grep -E ':(8000|10000|2379)'
```

### Configure Envoy as a front proxy

Envoy can simplify network configuration by exposing only one external port for all Deephaven services:

```bash
# Check Envoy status if installed
sudo systemctl status envoy

# View Envoy configuration
cat /etc/envoy/envoy.yaml

# Restart Envoy after configuration changes
sudo systemctl restart envoy
```

### Troubleshoot network issues

```bash
# Check firewall status
sudo firewall-cmd --list-all

# Test network latency to Deephaven server
ping -c 5 deephaven-host

# Check DNS resolution
nslookup deephaven-host
```

## Health check commands

See also:

- [Troubleshooting certificates](../sys-admin/troubleshooting/troubleshooting-certificates.md)
- [Cluster configuration guide](../sys-admin/installation/cluster-config-guide.md)

### System health checks

Quickly check the health of the system and services:

```bash
# Check overall system resource usage
top -b -n 1

# Check disk space usage
df -h

# Check memory usage
free -h

# Check load average
uptime
```

### Deephaven service health checks

Verify that Deephaven services are running properly:

```bash
# Use monit to check service health
sudo -u irisadmin monit status

# Check if query server is responding
curl -ik https://localhost:8000/iris/connection.json

# Check if authentication server is healthy
curl -ik https://localhost:10000/auth/health
```

### Log health checks

Monitor logs for errors and warnings:

```bash
# Check for errors in query server logs
grep -i error /var/log/deephaven/query_server/db_query_server.log.current

# Check for warnings in authentication server logs
grep -i warn /var/log/deephaven/authentication_server/authentication_server.log.current

# Monitor logs in real-time
tail -f /var/log/deephaven/query_server/db_query_server.log.current
```

### Database and storage checks

Verify database and storage systems:

```bash
# Check etcd cluster health
sudo -u irisadmin /usr/illumon/latest/bin/etcdctl.sh endpoint health --write-out table

# Check MariaDB status (if used for ACLs)
sudo systemctl status mariadb

# Check filesystem status where Deephaven data is stored
sudo xfs_info /db
```

## Security hardening

See also:

- [NFS mounting and security](../sys-admin/table-storage/nfs.md)
- [MariaDB/MySQL management](#mariadbmysql-management)
- [Certificate management](#certificate-management)

### File permissions audit

Verify and fix permissions on key Deephaven directories:

```bash
# Check permissions on config directories
ls -la /etc/deephaven/
ls -la /etc/sysconfig/illumon.d/

# Fix permissions if needed
sudo chown -R irisadmin:irisadmin /etc/deephaven/
sudo chmod 750 /etc/deephaven/
```

### Authentication configuration

Verify and update authentication settings:

```bash
# Check if public key authentication is enabled
grep -A5 "auth.publickey.enabled" /etc/sysconfig/illumon.d/authentication_server/irisauth.properties

# Enable public key authentication
sudo sed -i 's/auth.publickey.enabled=false/auth.publickey.enabled=true/' /etc/sysconfig/illumon.d/authentication_server/irisauth.properties

# Set minimum password length (default is 8)
sudo sed -i 's/auth.password.min.length=8/auth.password.min.length=12/' /etc/sysconfig/illumon.d/authentication_server/irisauth.properties
```

### Network security

Configure firewall to restrict access to Deephaven services:

```bash
# Check current firewall rules
sudo firewall-cmd --list-all

# Add rules to restrict access to Deephaven ports
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="trusted-subnet/24" port port="8000" protocol="tcp" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="trusted-subnet/24" port port="10000" protocol="tcp" accept'

# Reload firewall rules
sudo firewall-cmd --reload
```

### System account security

Audit and secure system accounts:

```bash
# Check if irisadmin account is locked for direct login
sudo passwd -S irisadmin

# Lock system accounts not needed for direct login
sudo passwd -l irisadmin

# Check sudo permissions
sudo cat /etc/sudoers.d/deephaven
```