Persistent Query Controller etcd access

If a malformed Persistent Query (PQ) XML is stored in etcd, the controller may fail to start. This guide explains how to access the controller's etcd storage directly to remove the problematic PQ. This procedure should only be used in exceptional circumstances.

Every PQ is uniquely identified by a serial number that is displayed in the exception message. For example, the serial 1660929719014000000 converts to 170ccec78ab80580 in hexadecimal. When interacting with etcd, always use the hexadecimal format with lowercase letters. Update the examples with the converted hexadecimal serial.

All interactions with etcd will use the Deephaven administration system account, typically irisadmin. These examples assume that etcd is configured with its default Deephaven settings, so the appropriate keys are accessible to irisadmin in the directory /etc/sysconfig/illumon.d/etcd/client/controller. Running the commands on the server where the controller operates will ensure this.

To see a list of PQ serials in hexadecimal, use the following command:

sudo -u irisadmin \
DH_ETCD_USER=controller /usr/illumon/latest/bin/etcdctl.sh get \
 --prefix /main/data/persistent-query-v2/ \
 --keys-only

To display the details of a PQ, use the following command. This will print an XML view of the PQ:

sudo -u irisadmin \
DH_ETCD_USER=controller /usr/illumon/latest/bin/etcdctl.sh get \
 /main/data/persistent-query-v2/configuration/170ccec78ab80580 \
 --print-value-only

To delete a PQ, use the following command. This should only be done in emergencies and when the PQ controller is stopped. If there are multiple controllers, ensure all of them are stopped before proceeding.

sudo -u irisadmin \
DH_ETCD_USER=controller /usr/illumon/latest/bin/etcdctl.sh del \
 --prefix /main/data/persistent-query-v2/configuration/170ccec78ab80580

Related documentation

• etcd