Uninstall and reinstall Deephaven

This guide describes how to completely remove a native (bare-metal or VM) Deephaven installation so that you can reinstall it from scratch on the same hosts. If your deployment uses Kubernetes, see Reinstalling a broken etcd cluster instead.

Important

You do not need to uninstall Deephaven to upgrade it, even to a newer patch release. You upgrade in place by changing DH_VERSION and re-running the installer. See Basic upgrade. Only follow this guide when you intend to discard the existing installation entirely.

The most important part of a teardown is etcd. The installer checks the etcd servers for an existing etcd configuration and every node for existing etcd client credentials, and reuses whatever it finds. etcd itself reuses any database it finds for the same cluster token. A partial teardown leaves the reinstall trying to talk to an etcd cluster with the wrong keys, the wrong members, or the wrong passwords, and the resulting errors do not point clearly at the leftover files.

If a reinstall has already failed, see Symptoms of an incomplete teardown to match the error to the leftover item.

Overview

Each step applies to a specific set of hosts. Skipping a step on even one host leaves exactly the partial state this guide is meant to prevent.

Step	Where to run it
Stop all Deephaven services	Every Deephaven node
Remove the etcd servers' state	Every etcd server
Remove etcd client credentials	Every Deephaven node
Remove Deephaven files	Every Deephaven node
Clean up the installer host	The installer host
Verify the teardown	Every Deephaven node and the installer host
Restore prepared directories (non-root installs)	Every Deephaven node, if you install without root

Before you begin

Run the commands in this guide as root (or with sudo).

Take a backup

Removing an installation deletes your configuration, ACLs, schemas, Persistent Query definitions, and etcd data. Take a complete backup first, including the etcd data directory. See Backup and restoration.

Decide what to keep

Some items are safe to keep, and keeping them makes the reinstall easier:

Historical and intraday data in /db. /db/Intraday, /db/IntradayUser, /db/Systems, and /db/Users hold your table data. Keep them if you intend to use the same data after the reinstall. Always keep /db/TempFiles, which contains the home directories of the Deephaven service accounts.
Your cluster configuration. Keep your cluster.cnf file on the installer host so that you can reinstall with the same settings.
Your TLS certificates. Keep the certificates you supplied in DH_LOCAL_CERT_DIR if they are still valid.
OS-level prerequisites. The installer reuses all of the following if they are present, so you do not need to remove them:
- Service accounts and groups (irisadmin, dbquery, dbmerge, etcd, and so on).
- Sudoers rules.
- The Java and monit packages, the include lines the installer added to your monit control file, and /etc/systemd/system/monit.service.d/override.conf.
- The etcd and etcdctl binaries in DH_DIR_ETCD_BIN.

Identify your nodes

Note which hosts in your cluster.cnf are etcd servers (the nodes with DH_NODE_N_ROLE_ETCD=true). If some etcd servers keep their old data and others do not, the reinstalled cluster cannot form.

Stop all Deephaven services

On every Deephaven node, stop the Deephaven processes and wait for them to exit, then stop monit so that it does not restart them:

/usr/illumon/latest/bin/dh_monit down --block
systemctl stop monit
systemctl disable monit

If monit was started directly by the Deephaven admin user rather than by systemd, systemctl stop monit does not stop it. Check with pgrep -a monit, and if a monit process is still running, stop it with monit quit as the user that owns the process.

If you use Envoy, stop and remove it as described in Uninstall Envoy. Do this before you remove /usr/illumon, because some of the Envoy removal steps use tools installed there.

Remove the etcd servers' state

On every etcd server, after Deephaven is stopped on all nodes, stop etcd and delete its configuration and data.

Stop and disable the `dh-etcd` service

Deephaven runs etcd as a dedicated systemd service named dh-etcd. The installer enables this service from a unit file inside the etcd configuration directory, so systemd keeps links to that file under /etc/systemd/system.

Warning

If a dh-etcd unit is still enabled when you reinstall, the installer does not replace it. The old unit points to the previous cluster's configuration directory, so if the new installation uses a different cluster token, etcd fails to start or starts with the old configuration.

Stop the service, disable it, and remove any links that remain:

systemctl stop dh-etcd
systemctl disable dh-etcd
rm -f /etc/systemd/system/dh-etcd.service /etc/systemd/system/multi-user.target.wants/dh-etcd.service
systemctl daemon-reload
systemctl reset-failed dh-etcd 2>/dev/null || true

Confirm that systemd no longer knows about the service:

systemctl list-unit-files dh-etcd.service

This command should report 0 unit files listed.

Delete etcd configuration and data

/etc/etcd/dh/<cluster-token>/ contains the server configuration, peer and server TLS certificates, and the dh-etcd.service unit file. /etc/etcd/dh/latest links to it.
/var/lib/etcd/dh/<cluster-token>/ contains the etcd database.

Note

Remove only /var/lib/etcd/dh, not /var/lib/etcd. On RHEL-family systems, and on any system where you created the etcd user by hand, /var/lib/etcd is also that user's home directory.

Remove both directories entirely:

rm -rf /etc/etcd/dh
rm -rf /var/lib/etcd/dh

Do not delete only the latest link or config.yaml. If the database remains and the reinstall uses the same cluster token (for example, because DH_ETCD_TOKEN is set in cluster.cnf), etcd ignores the new cluster settings and starts with the old database. That database includes the old users and passwords, which no longer match the credentials the installer just generated.

Remove etcd client credentials

On every Deephaven node — not only the etcd servers — remove the etcd client credentials:

rm -rf /etc/sysconfig/deephaven/etcd

/etc/sysconfig/deephaven/etcd/client/ holds the user names, passwords, and CA certificate every Deephaven process uses to connect to etcd. If these files remain, the installer assumes etcd is already configured and tries to connect with the old passwords before the new etcd cluster exists.

The node that generated the etcd keys may also have /tmp/dh_etcd_configure.* directories containing working copies of the previous cluster's keys. They do not affect the reinstall, but remove them so that old secrets are not left over:

rm -rf /tmp/dh_etcd_configure.*

Remove Deephaven files

Perform these steps on every Deephaven node.

Remove installation files

rm -rf /usr/illumon
rm -rf /etc/sysconfig/deephaven
rm -rf /etc/sysconfig/illumon /etc/sysconfig/illumon.d /etc/sysconfig/illumon.confs
rm -rf /etc/deephaven
rm -rf /var/lib/deephaven /var/lib/illumon
rm -rf /var/log/deephaven
rm -rf /etc/monit.d/deephaven
rm -f /etc/cron.d/illumonRestart.cron

Path	Contents
`/usr/illumon`	Deephaven and Core+ binaries, Python virtual environments, and the `latest` link.
`/etc/sysconfig/deephaven`	Configuration files, keystores, truststores, the installed copy of `cluster.cnf`, and backups.
`/etc/sysconfig/illumon`, `illumon.d`, `illumon.confs`	Links into `/etc/sysconfig/deephaven`. On installations originally created by older releases, `illumon.d` may be a directory.
`/etc/deephaven`	Node identity and version markers, generated certificates, and PID files.
`/var/lib/deephaven`, `/var/lib/illumon`	Installation media and runtime state.
`/var/log/deephaven`	Process and installation logs.
`/etc/monit.d/deephaven`	The monit configuration for Deephaven processes.

Caution

When you reinstall the same version, the installer does not overwrite configuration files in /etc/sysconfig/deephaven; it only adds files that are missing. The installer silently reuses any configuration left behind.

Remove table data (optional)

Skip this step if you are keeping your data (see Decide what to keep). Otherwise, remove the table data directories and the Python virtual environments used by Legacy workers:

rm -rf /db/Intraday /db/IntradayUser /db/Systems /db/Users /db/VEnvs

Do not remove /db/TempFiles.

Caution

/db/Systems and /db/Users are often shared across nodes over NFS or similar shared storage. Removing them on one node removes them for every node that mounts the same storage, including nodes outside this cluster.

Remove the remote installation directory

Remove the installer's working directory, DH_REMOTE_DIR. Substitute the value from your cluster.cnf:

rm -rf /path/to/DH_REMOTE_DIR

If DH_REMOTE_DIR is the same directory as DH_LOCAL_DIR on the installer host, skip this step on that host and follow Clean up the installer host instead, so that you keep your cluster.cnf.

Clean up the installer host

On the installer host, clear the generated files from DH_LOCAL_DIR, keeping cluster.cnf, your installation media, and your certificates:

cd /path/to/DH_LOCAL_DIR
rm -rf keys logs previous_logs work
rm -f master_install.sh dh_*.sh etcd_configure.sh

keys/dh_etcd_config.tgz holds the previous etcd cluster's keys and passwords. If DH_LOCAL_DIR is also DH_REMOTE_DIR for this host, remove lib/secure as well, since it holds another copy:

rm -rf lib/secure

Verify the teardown

Before reinstalling, confirm on every Deephaven node that nothing is left behind. Substitute your DH_REMOTE_DIR value:

systemctl is-active dh-etcd monit
systemctl list-unit-files dh-etcd.service
ls -d /etc/etcd/dh /var/lib/etcd/dh /etc/sysconfig/deephaven /usr/illumon /etc/deephaven \
  /etc/systemd/system/dh-etcd.service /path/to/DH_REMOTE_DIR 2>&1

The services should report inactive, no dh-etcd unit files should be listed, and ls should report No such file or directory for every path.

On the installer host, confirm that the old etcd keys are gone:

ls -d /path/to/DH_LOCAL_DIR/keys 2>&1

Restore prepared directories (non-root installs)

Skip this step if the installer can run commands as root through passwordless sudo, because the installer recreates everything this guide removed.

If you install without root privileges, you created several system directories and links by hand before your original installation. The teardown removes them, and the installer cannot recreate them without root. Before reinstalling, repeat these steps from Appendix A: Elevated permissions:

Create Deephaven directories and permissions
Create necessary soft links
Restart system services, which re-enables monit and sets up the dh-etcd service that this guide removed. See Appendix E: dh-etcd service unit file for the unit file, and set DH_ETCD_TOKEN in cluster.cnf to the token it uses.

Reinstall

Reinstall by following the Basic install guide from Generate the installation scripts onward, using your saved cluster.cnf.

The installer tries to re-enable the monit service that you disabled in Stop all Deephaven services. Without root privileges, that attempt fails silently, so have a privileged user run sudo systemctl enable monit after the installation to make sure monit starts at boot.

Once the installation finishes, verify the installation and restore any configuration you need from your backup.

Symptoms of an incomplete teardown

If the reinstall fails, the installer logs usually contain one of the following messages. Each one points to a specific leftover item.

Log message	Most likely cause	Fix
`If you wish to generate a new cluster, please delete /etc/etcd/dh/latest/config.yaml`	Old etcd server configuration remains on one or more etcd servers, with different member IPs.	Delete etcd configuration and data on all etcd servers.
`systemctl status dh-etcd must report a running process to continue the install.`	Old etcd server configuration remains, but `dh-etcd` is not running, so the installer stops.	Remove the etcd servers' state on all etcd servers.
`Etcd is not healthy and DH_ALLOW_ETCD_OFFLINE != true, failing installation!`	Old etcd client credentials remain in `/etc/sysconfig/deephaven/etcd`, so the installer expects a running etcd.	Remove etcd client credentials on every node.
`ERROR: Max Attempts exceeding for connecting to ectd.`	The etcd database from the previous installation remains, and its users and passwords do not match the new credentials. The same message also appears when etcd cannot be reached at all.	Delete etcd configuration and data on all etcd servers.

For other installation problems, see Installation troubleshooting and Troubleshoot etcd errors.