How to troubleshoot errors in etcd

This guide describes several error conditions that can be encountered with etcd and provides mitigations for them.

etcdctl Command Overview

The standard command for interacting with etcd is etcdctl. This command requires several settings or options to find and authenticate with the etcd nodes.

In a Deephaven environment, the etcdctl.sh script, located in /usr/illumon/latest/bin, simplifies the process by automatically setting most required options. To connect, you need access to the appropriate configuration files, which are stored in /etc/sysconfig/illumon.d/etcd/client. While this document uses the root configuration for examples, non-root configurations are also available for users without root permissions.

This command will print a table of your etcd configuration:

Running out of space in a node

When etcd exceeds its configured storage space, it can be challenging to identify and resolve the issue. The tools presented in this section can be useful for addressing a variety of issues. However, examples will demonstrate the troubleshooting process with the following error:

Investigate the condition

If one or more nodes in the etcd cluster are out of space, you will see an output similar to this when running the alarm list command:

Output:

Clear the error condition

You can clear the alarms with alarm disarm:

Note that this clears the current alarms but does not address their root cause. The alarm will return unless the underlying issue is resolved.

Check compaction settings

Every change in etcd creates a new revision, which can be used to retrieve previous key values. To prevent storage space from increasing indefinitely, this history needs to be compacted periodically.

The default configuration file includes these lines:

This means that etcd will automatically compact every hour (implied by the periodic mode) and remove all versions older than 168 hours (1 week). If your system frequently exceeds database space, you can shorten this time period or change the mode.

The default configuration file is /etc/etcd/dh/latest/config.yaml on the nodes running etcd. Note that this is a symbolic link to one of several configuration files. You will need to edit all the configuration files, distribute them to all etcd nodes, and restart the etcd processes to apply the changes.

Compact now

You can compact history immediately instead of waiting for the periodic compaction.

Find the current revision:

Use the value of the "Revision" field minus one, and fill it in below.

Compact away all old revisions

Verify the system accepts changes again

This command makes changes, and you should see the revision number increase. You can also verify a change by running any command that modifies etcd.

Increase the maximum database size

The default maximum size is 2 GB. You can increase this by adding the following setting to the configuration file (typically in /etc/etcd/dh/latest/config.yaml):

An 8 GB limit is recommended, but larger values are supported.

The current setting for this value can be found in the metrics that etcd publishes.

After updating the settings, you must restart etcd with systemctl restart dh-etcd.

Use the etcdctl commands mentioned above to find your addresses, then use curl to get the metrics:

Related documentation

• Configuration files
• Defragment etcd nodes
• Introduction to etcd
• Replace an etcd node
• etcdctl