Etcd on Minikube

Notes on working with etcd and minikube.

etcdctl

hostMount on minikube

$ minikube ssh "ls /var/lib/minikube/certs/etcd"
ca.crt  ca.key  healthcheck-client.crt  healthcheck-client.key  peer.crt  peer.key  server.crt  server.key
$ minikube ssh "sudo ls /var/lib/minikube/etcd"
member

Backup and restore from cli test:

Build etcd (golang executable)

$ git clone https://github.com/etcd-io/etcd.git && cd etcd && make

Create minikube tmp mount and copy keys over

$ minikube mount ./kurtis:/kurtis &

Copy over the etcd certs on minikube host to our machine:

minikube ssh "sudo cp /var/lib/minikube/certs/etcd/{ca.crt,server.crt,server.key} /kurtis/"

minikube port-forward to the etcd-minikube pod in kube-system namespace

$ kubectl port-forward etcd-minikube -n kube-system 2379

Test connection with local etcdctl

$ ETCDCTL_API=3 ./bin/etcdctl --cacert=./kurtis/ca.crt --cert=./kurtis/server.crt --key=./kurtis/server.key version
etcdctl version: 3.6.0-alpha.0
API version: 3.6

Create Secret and Back it up

$ kubectl create secret generic login --from-literal=username='admin' --from-literal=password='abc123'
secret/login created

$ ETCDCTL_API=3 ./bin/etcdctl --cacert=./kurtis/ca.crt --cert=./kurtis/server.crt --key=./kurtis/server.key snapshot save ./etcd-backup.db

Verify Backup:

$ ETCDCTL_API=3 ./bin/etcdctl --cacert=./kurtis/ca.crt --cert=./kurtis/server.crt --key=./kurtis/server.key --write-out=table snapshot status ./etcd-backup.db
Deprecated: Use `etcdutl snapshot status` instead.

+----------+----------+------------+------------+---------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION |
+----------+----------+------------+------------+---------+
| 9c21182d |     4728 |        605 |     1.5 MB |         |
+----------+----------+------------+------------+---------+

Delete Secret

$ kubectl delete secret login
secret "login" deleted

Restore

Move backup to host mount

mv default.etcd kurtis/

Restart etcd:

$ minikube ssh "sudo rm -rf /var/lib/minikube/etcd/member"
$ minikube ssh "sudo mv /kurtis/default.etcd/* /var/lib/minikube/etcd/"
$ kubectl delete pods etcd-minikube                     -n kube-system

View restored secret:

$ kubectl get secret login                -o yaml
apiVersion: v1
data:
  password: YWJjMTIz
  username: YWRtaW4=
kind: Secret
metadata:
  creationTimestamp: "2022-03-16T01:45:39Z"
  name: login
  namespace: default
  resourceVersion: "4660"
  uid: 01180b69-36db-4837-b066-88eb2408af92
type: Opaque