ACL Write Server runbook

The Access Control List (ACL) Write Server is a Deephaven infrastructure service that provides the API for modifying ACL data including users, groups, and permissions. It is the only service that can write to the ACL database and requires membership in the iris-superusers group to access.

Impact of ACL Write Server failure

Level	Impact
Sev 2 - Moderate	Administrators will not be able to update user permissions and groups.

Note

ACL failures do not prevent users from working. Existing permissions remain in effect, but administrators cannot modify user access, create new users, or change group memberships until the ACL Write Server is restored.

ACL Write Server purpose

The ACL Write Server provides the administrative interface for managing:

Users — Create, modify, and delete user accounts.
Groups — Create, modify, and delete groups and manage memberships.
Permissions — Grant and revoke table, query, and system permissions.
DSA keys — Manage public key authentication for programmatic access.
Password management — Reset passwords, enforce password policies.

ACL Write Server dependencies

The ACL Write Server requires:

etcd cluster — Must be running and accessible if using etcd for ACL storage.
MySQL/MariaDB — Must be running if using MySQL for ACL storage.
etcd client configuration files — Located at /etc/sysconfig/deephaven/etcd/client.
Authentication Server — Used to validate requesting user has superuser permissions.

Checking ACL Write Server status

Check process is running with monit:

dh_monit status db_acl_write_server

Expected output should show status Running.

Viewing ACL Write Server logs

View application log:

cat /var/log/deephaven/acl_write_server/DbAclWriteServer.log.current

Tail the log to follow in real-time:

tail -f /var/log/deephaven/acl_write_server/DbAclWriteServer.log.current

List historical log files:

ls -ltr /var/log/deephaven/acl_write_server/db_acl_write_server.log.????-??-??

View process stdout/stderr logs:

cat /var/log/deephaven/acl_write_server/db_acl_write_server.log.$(date +%Y-%m-%d)

Restart procedure

Restart the ACL Write Server:

dh_monit restart db_acl_write_server

Impact: Restarting the ACL Write Server temporarily prevents ACL modifications but does not affect existing user sessions or permissions.

Verify the restart was successful:

dh_monit status db_acl_write_server

Monitor the log during startup:

tail -f /var/log/deephaven/acl_write_server/DbAclWriteServer.log.current

Expected startup messages:

Successful connection to ACL backend (etcd or MySQL).
Server listening on configured port.
No authentication backend errors.

Checking dependencies

Check etcd status

If using etcd for ACLs:

sudo -u irisadmin /usr/illumon/latest/bin/etcdctl.sh endpoint status --write-out table

Check MySQL status

If using MySQL for ACLs:

sudo systemctl status mariadb

Test MySQL connectivity and query ACL tables:

sudo mysql -D dbacl_iris -e "SELECT COUNT(*) FROM users"
sudo mysql -D dbacl_iris -e "SELECT COUNT(*) FROM groups"
sudo mysql -D dbacl_iris -e "SELECT COUNT(*) FROM tableacls"

Managing ACLs with dhconfig

The primary interface for ACL management is the dhconfig acls command.

List users

sudo /usr/illumon/latest/bin/dhconfig acls users list

Create a user

sudo /usr/illumon/latest/bin/dhconfig acls users add --name newuser --hashed-password '<hashed-password>'

Modify user permissions

System-level permissions (such as superuser) are managed through ACL import/export. See the dhconfig acls command reference for the import and export subcommands with --type systemuser.

List groups

sudo /usr/illumon/latest/bin/dhconfig acls groups list

Create a group

Groups are created implicitly when members are added, or via ACL import. There is no standalone group-creation command. To create a group and add a member in one step:

sudo /usr/illumon/latest/bin/dhconfig acls groups add-member --name someuser --group analysts

Add user to group

sudo /usr/illumon/latest/bin/dhconfig acls groups add-member --name someuser --group analysts

Grant table permissions

Table ACLs are managed through ACL import/export. See the dhconfig acls command reference for the import and export subcommands with --type tableacls.

Manage public keys

# Import a public key for programmatic access
sudo /usr/illumon/latest/bin/dhconfig acls publickeys import --file /path/to/public_key.pem

See dhconfig acls command reference for complete documentation.

ACL storage backends

The ACL Write Server supports two storage backends:

etcd backend (recommended)

Advantages:

Integrated with other Deephaven configuration.
Better consistency with cluster configuration.
Simpler backup/restore with etcd snapshots.
No separate database to manage.

Configuration:

acls.dataSource=etcd

MySQL backend (Legacy)

Advantages:

May be required for Legacy migrations.
Familiar SQL interface for direct queries.
Existing MySQL infrastructure can be reused.

Configuration:

acls.dataSource=mysql
db.url=jdbc:mariadb://localhost:3306/dbacl_iris
db.user=iris
db.password=<encrypted-password>

Backup and recovery

Backup ACL data

For etcd-based ACLs:

ACLs are backed up as part of etcd snapshots:

sudo -u irisadmin /usr/illumon/latest/bin/etcdctl.sh snapshot save /backup/etcd-acls-$(date +%Y%m%d).db

For MySQL-based ACLs:

Create MySQL backup:

sudo mysqldump dbacl_iris > /backup/acls-$(date +%Y%m%d).sql

Restore ACL data

For etcd: Restore from etcd snapshot (see etcd runbook)

For MySQL:

sudo mysql dbacl_iris < /backup/acls-20250101.sql

Migration from MySQL to etcd

To migrate ACLs from MySQL to etcd:

See Migrating ACLs to etcd for detailed procedure.

Configuration files and locations

monit configuration: /etc/sysconfig/illumon.d/monit/db_acl_write_server.conf

Property files:

/etc/sysconfig/illumon.d/resources/iris-common.prop
/etc/sysconfig/illumon.d/resources/db_acl_write_server.prop

MySQL database: dbacl_iris (if using MySQL backend)

MySQL configuration: /etc/my.cnf

Log directory: /var/log/deephaven/acl_write_server/