Cleanup etcd2 keys

Define the problem

While using etcd2 for our production key-value store to hold cluster state and some configuration settings there is the need to perform some manual cleanup from time to time to reduce the number of keys, especially if you use keys that have the machine id for example hold configuration for that machine.

The format where we store information in etcd is following the schema in etcd /service/<service_name>/machines/<machine-id>/...

The real problem is who to query the existing machine of the cluster from fleet and delete the etcd keys that we no longer need.

please be careful as if you are not sure what you are doing you can drop your etcd cluster


With the use of simple bash commands and

we will filter what keys we need to remove.

# First get the machine that are active right now
$ fleetctl --endpoint=http://$etcd_host:2379 list-machines -l -no-legend |awk '{ print $1}' > existing_machines

# Get the machines for a service 
$ etcdctl --peers=http://$etcd_host:2379 ls /service/<service_name>/machines |awk '{split($0,a,"/"); print a[5]}' > machines

#filter the 2 files with the keys that exists in the `machine` and on `existing_machines` and the result will be the keys that need to be cleaned up.
$ grep -f existing_machines -v machines >keys_to_drop
$ cat keys_to_drop

Now we can iterate thought the list and remove the keys from etcd.

$ for i in $(cat keys_to_drop ); do
etcdctl --peers=http://$etcd_host:2379 rmdir /service/<service_name>/machines/$i

Now you will not have the extra keys in etcd.

Alexandros Sapranidis

Software engineer, keen on wearing many hat, current Senior Software Engineer @Elastic cloud

Athens, Greece