---
title: Uninstall and reinstall Deephaven
sidebar_label: Uninstall and reinstall
---

This guide describes how to completely remove a native (bare-metal or VM) Deephaven installation so that you can reinstall it from scratch on the same hosts. If your deployment uses Kubernetes, see [Reinstalling a broken etcd cluster](../kubernetes/kubernetes-install-guide.md#reinstalling-a-broken-etcd-cluster) instead.

> [!IMPORTANT]
> You do not need to uninstall Deephaven to upgrade it, even to a newer patch release. You upgrade in place by changing [`DH_VERSION`](./cluster-config-guide.md#dh_version) and re-running the installer. See [Basic upgrade](./basic-upgrade.md). Only follow this guide when you intend to discard the existing installation entirely.

The most important part of a teardown is etcd. The installer checks the etcd servers for an existing etcd configuration and every node for existing etcd client credentials, and reuses whatever it finds. etcd itself reuses any database it finds for the same cluster token. A partial teardown leaves the reinstall trying to talk to an etcd cluster with the wrong keys, the wrong members, or the wrong passwords, and the resulting errors do not point clearly at the leftover files.

If a reinstall has already failed, see [Symptoms of an incomplete teardown](#symptoms-of-an-incomplete-teardown) to match the error to the leftover item.

## Overview

Each step applies to a specific set of hosts. Skipping a step on even one host leaves exactly the partial state this guide is meant to prevent.

| Step                                                                                                | Where to run it                                   |
| --------------------------------------------------------------------------------------------------- | ------------------------------------------------- |
| [Stop all Deephaven services](#stop-all-deephaven-services)                                         | Every Deephaven node                              |
| [Remove the etcd servers' state](#remove-the-etcd-servers-state)                                    | Every etcd server                                 |
| [Remove etcd client credentials](#remove-etcd-client-credentials)                                   | Every Deephaven node                              |
| [Remove Deephaven files](#remove-deephaven-files)                                                   | Every Deephaven node                              |
| [Clean up the installer host](#clean-up-the-installer-host)                                         | The installer host                                |
| [Verify the teardown](#verify-the-teardown)                                                         | Every Deephaven node and the installer host       |
| [Restore prepared directories (non-root installs)](#restore-prepared-directories-non-root-installs) | Every Deephaven node, if you install without root |

## Before you begin

Run the commands in this guide as `root` (or with `sudo`).

### Take a backup

Removing an installation deletes your configuration, ACLs, schemas, Persistent Query definitions, and etcd data. Take a complete backup first, including the etcd data directory. See [Backup and restoration](../backup-restore-migrate/backup-and-restore.md).

### Decide what to keep

Some items are safe to keep, and keeping them makes the reinstall easier:

- **Historical and intraday data in `/db`.** `/db/Intraday`, `/db/IntradayUser`, `/db/Systems`, and `/db/Users` hold your table data. Keep them if you intend to use the same data after the reinstall. Always keep `/db/TempFiles`, which contains the home directories of the Deephaven service accounts.
- **Your cluster configuration.** Keep your `cluster.cnf` file on the installer host so that you can reinstall with the same settings.
- **Your TLS certificates.** Keep the certificates you supplied in [`DH_LOCAL_CERT_DIR`](./cluster-config-guide.md#dh_local_cert_dir) if they are still valid.
- **OS-level prerequisites.** The installer reuses all of the following if they are present, so you do not need to remove them:
  - Service accounts and groups (`irisadmin`, `dbquery`, `dbmerge`, `etcd`, and so on).
  - Sudoers rules.
  - The Java and monit packages, the `include` lines the installer added to your monit control file, and `/etc/systemd/system/monit.service.d/override.conf`.
  - The `etcd` and `etcdctl` binaries in [`DH_DIR_ETCD_BIN`](./cluster-config-guide.md#dh_dir_etcd_bin).

### Identify your nodes

Note which hosts in your `cluster.cnf` are etcd servers (the nodes with `DH_NODE_N_ROLE_ETCD=true`). If some etcd servers keep their old data and others do not, the reinstalled cluster cannot form.

## Stop all Deephaven services

On every Deephaven node, stop the Deephaven processes and wait for them to exit, then stop monit so that it does not restart them:

```bash
/usr/illumon/latest/bin/dh_monit down --block
systemctl stop monit
systemctl disable monit
```

If monit was started directly by the Deephaven admin user rather than by systemd, `systemctl stop monit` does not stop it. Check with `pgrep -a monit`, and if a monit process is still running, stop it with `monit quit` as the user that owns the process.

If you use Envoy, stop and remove it as described in [Uninstall Envoy](../envoy/installing-envoy.md#uninstall-envoy). Do this before you remove `/usr/illumon`, because some of the Envoy removal steps use tools installed there.

## Remove the etcd servers' state

On every etcd server, after Deephaven is stopped on all nodes, stop etcd and delete its configuration and data.

### Stop and disable the `dh-etcd` service

Deephaven runs etcd as a dedicated systemd service named `dh-etcd`. The installer enables this service from a unit file inside the etcd configuration directory, so systemd keeps links to that file under `/etc/systemd/system`.

> [!WARNING]
> If a `dh-etcd` unit is still enabled when you reinstall, the installer does not replace it. The old unit points to the previous cluster's configuration directory, so if the new installation uses a different cluster token, etcd fails to start or starts with the old configuration.

Stop the service, disable it, and remove any links that remain:

```bash
systemctl stop dh-etcd
systemctl disable dh-etcd
rm -f /etc/systemd/system/dh-etcd.service /etc/systemd/system/multi-user.target.wants/dh-etcd.service
systemctl daemon-reload
systemctl reset-failed dh-etcd 2>/dev/null || true
```

Confirm that systemd no longer knows about the service:

```bash
systemctl list-unit-files dh-etcd.service
```

This command should report `0 unit files listed`.

### Delete etcd configuration and data

- `/etc/etcd/dh/<cluster-token>/` contains the server configuration, peer and server TLS certificates, and the `dh-etcd.service` unit file. `/etc/etcd/dh/latest` links to it.
- `/var/lib/etcd/dh/<cluster-token>/` contains the etcd database.

> [!NOTE]
> Remove only `/var/lib/etcd/dh`, not `/var/lib/etcd`. On RHEL-family systems, and on any system where you created the `etcd` user by hand, `/var/lib/etcd` is also that user's home directory.

Remove both directories entirely:

```bash
rm -rf /etc/etcd/dh
rm -rf /var/lib/etcd/dh
```

Do not delete only the `latest` link or `config.yaml`. If the database remains and the reinstall uses the same cluster token (for example, because [`DH_ETCD_TOKEN`](./cluster-config-guide.md#dh_etcd_token) is set in `cluster.cnf`), etcd ignores the new cluster settings and starts with the old database. That database includes the old users and passwords, which no longer match the credentials the installer just generated.

## Remove etcd client credentials

On every Deephaven node — not only the etcd servers — remove the etcd client credentials:

```bash
rm -rf /etc/sysconfig/deephaven/etcd
```

`/etc/sysconfig/deephaven/etcd/client/` holds the user names, passwords, and CA certificate every Deephaven process uses to connect to etcd. If these files remain, the installer assumes etcd is already configured and tries to connect with the old passwords before the new etcd cluster exists.

The node that generated the etcd keys may also have `/tmp/dh_etcd_configure.*` directories containing working copies of the previous cluster's keys. They do not affect the reinstall, but remove them so that old secrets are not left over:

```bash
rm -rf /tmp/dh_etcd_configure.*
```

## Remove Deephaven files

Perform these steps on every Deephaven node.

### Remove installation files

```bash
rm -rf /usr/illumon
rm -rf /etc/sysconfig/deephaven
rm -rf /etc/sysconfig/illumon /etc/sysconfig/illumon.d /etc/sysconfig/illumon.confs
rm -rf /etc/deephaven
rm -rf /var/lib/deephaven /var/lib/illumon
rm -rf /var/log/deephaven
rm -rf /etc/monit.d/deephaven
rm -f /etc/cron.d/illumonRestart.cron
```

| Path                                                   | Contents                                                                                                                      |
| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
| `/usr/illumon`                                         | Deephaven and Core+ binaries, Python virtual environments, and the `latest` link.                                             |
| `/etc/sysconfig/deephaven`                             | Configuration files, keystores, truststores, the installed copy of `cluster.cnf`, and backups.                                |
| `/etc/sysconfig/illumon`, `illumon.d`, `illumon.confs` | Links into `/etc/sysconfig/deephaven`. On installations originally created by older releases, `illumon.d` may be a directory. |
| `/etc/deephaven`                                       | Node identity and version markers, generated certificates, and PID files.                                                     |
| `/var/lib/deephaven`, `/var/lib/illumon`               | Installation media and runtime state.                                                                                         |
| `/var/log/deephaven`                                   | Process and installation logs.                                                                                                |
| `/etc/monit.d/deephaven`                               | The monit configuration for Deephaven processes.                                                                              |

> [!CAUTION]
> When you reinstall the same version, the installer does not overwrite configuration files in `/etc/sysconfig/deephaven`; it only adds files that are missing. The installer silently reuses any configuration left behind.

### Remove table data (optional)

Skip this step if you are keeping your data (see [Decide what to keep](#decide-what-to-keep)). Otherwise, remove the table data directories and the Python virtual environments used by Legacy workers:

```bash
rm -rf /db/Intraday /db/IntradayUser /db/Systems /db/Users /db/VEnvs
```

Do not remove `/db/TempFiles`.

> [!CAUTION]
> `/db/Systems` and `/db/Users` are often shared across nodes over NFS or similar shared storage. Removing them on one node removes them for every node that mounts the same storage, including nodes outside this cluster.

### Remove the remote installation directory

Remove the installer's working directory, [`DH_REMOTE_DIR`](./cluster-config-guide.md#dh_remote_dir). Substitute the value from your `cluster.cnf`:

```bash
rm -rf /path/to/DH_REMOTE_DIR
```

If `DH_REMOTE_DIR` is the same directory as `DH_LOCAL_DIR` on the installer host, skip this step on that host and follow [Clean up the installer host](#clean-up-the-installer-host) instead, so that you keep your `cluster.cnf`.

## Clean up the installer host

On the installer host, clear the generated files from [`DH_LOCAL_DIR`](./cluster-config-guide.md#dh_local_dir), keeping `cluster.cnf`, your installation media, and your certificates:

```bash
cd /path/to/DH_LOCAL_DIR
rm -rf keys logs previous_logs work
rm -f master_install.sh dh_*.sh etcd_configure.sh
```

`keys/dh_etcd_config.tgz` holds the previous etcd cluster's keys and passwords. If `DH_LOCAL_DIR` is also `DH_REMOTE_DIR` for this host, remove `lib/secure` as well, since it holds another copy:

```bash
rm -rf lib/secure
```

## Verify the teardown

Before reinstalling, confirm on every Deephaven node that nothing is left behind. Substitute your `DH_REMOTE_DIR` value:

```bash
systemctl is-active dh-etcd monit
systemctl list-unit-files dh-etcd.service
ls -d /etc/etcd/dh /var/lib/etcd/dh /etc/sysconfig/deephaven /usr/illumon /etc/deephaven \
  /etc/systemd/system/dh-etcd.service /path/to/DH_REMOTE_DIR 2>&1
```

The services should report `inactive`, no `dh-etcd` unit files should be listed, and `ls` should report `No such file or directory` for every path.

On the installer host, confirm that the old etcd keys are gone:

```bash
ls -d /path/to/DH_LOCAL_DIR/keys 2>&1
```

## Restore prepared directories (non-root installs)

Skip this step if the installer can run commands as root through passwordless `sudo`, because the installer recreates everything this guide removed.

If you install without root privileges, you created several system directories and links by hand before your original installation. The teardown removes them, and the installer cannot recreate them without root. Before reinstalling, repeat these steps from [Appendix A: Elevated permissions](./basic-install.md#appendix-a-elevated-permissions):

- [Create Deephaven directories and permissions](./basic-install.md#create-deephaven-directories-and-permissions)
- [Create necessary soft links](./basic-install.md#create-necessary-soft-links)
- [Restart system services](./basic-install.md#restart-system-services), which re-enables monit and sets up the `dh-etcd` service that this guide removed. See [Appendix E: dh-etcd service unit file](./basic-install.md#appendix-e-dh-etcd-service-unit-file) for the unit file, and set [`DH_ETCD_TOKEN`](./cluster-config-guide.md#dh_etcd_token) in `cluster.cnf` to the token it uses.

## Reinstall

Reinstall by following the [Basic install](./basic-install.md) guide from [Generate the installation scripts](./basic-install.md#generate-the-installation-scripts) onward, using your saved `cluster.cnf`.

The installer tries to re-enable the monit service that you disabled in [Stop all Deephaven services](#stop-all-deephaven-services). Without root privileges, that attempt fails silently, so have a privileged user run `sudo systemctl enable monit` after the installation to make sure monit starts at boot.

Once the installation finishes, [verify the installation](./basic-install.md#verify-the-installation) and restore any configuration you need from your backup.

## Symptoms of an incomplete teardown

If the reinstall fails, the installer logs usually contain one of the following messages. Each one points to a specific leftover item.

| Log message                                                                            | Most likely cause                                                                                                                                                                         | Fix                                                                                            |
| -------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| `If you wish to generate a new cluster, please delete /etc/etcd/dh/latest/config.yaml` | Old etcd server configuration remains on one or more etcd servers, with different member IPs.                                                                                             | [Delete etcd configuration and data](#delete-etcd-configuration-and-data) on all etcd servers. |
| `systemctl status dh-etcd must report a running process to continue the install.`      | Old etcd server configuration remains, but `dh-etcd` is not running, so the installer stops.                                                                                              | [Remove the etcd servers' state](#remove-the-etcd-servers-state) on all etcd servers.          |
| `Etcd is not healthy and DH_ALLOW_ETCD_OFFLINE != true, failing installation!`         | Old etcd client credentials remain in `/etc/sysconfig/deephaven/etcd`, so the installer expects a running etcd.                                                                           | [Remove etcd client credentials](#remove-etcd-client-credentials) on every node.               |
| `ERROR: Max Attempts exceeding for connecting to ectd.`                                | The etcd database from the previous installation remains, and its users and passwords do not match the new credentials. The same message also appears when etcd cannot be reached at all. | [Delete etcd configuration and data](#delete-etcd-configuration-and-data) on all etcd servers. |

For other installation problems, see [Installation troubleshooting](../troubleshooting/troubleshooting-installation.md) and [Troubleshoot etcd errors](../troubleshooting/troubleshooting-etcd.md).

## Related documentation

- [Basic install](./basic-install.md)
- [Basic upgrade](./basic-upgrade.md)
- [Cluster configuration guide](./cluster-config-guide.md)
- [Backup and restoration](../backup-restore-migrate/backup-and-restore.md)
- [Installation troubleshooting](../troubleshooting/troubleshooting-installation.md)
- [Troubleshoot etcd errors](../troubleshooting/troubleshooting-etcd.md)