L3. etcd Security: Encryption at Rest and Access Control

Why etcd Is the Crown Jewel

etcd is a distributed key-value store that holds the entire state of your Kubernetes cluster. Every Secret, ConfigMap, RBAC binding, and pod spec lives in etcd. Anyone with direct etcd access can read all secrets, modify RBAC rules, and effectively own the cluster without going through the API server.

Encryption at Rest

By default, Kubernetes stores secrets in etcd as base64-encoded plaintext. This means anyone who can read the etcd data directory can decode every secret. Encryption at rest fixes this.

Create an encryption configuration:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
<ul class="list-disc pl-6 mb-4 space-y-2">
<li class="text-slate-300">resources:</li>
<li class="text-slate-300">secrets</li>
</ul>
    providers:
<ul class="list-disc pl-6 mb-4 space-y-2">
<li class="text-slate-300">aescbc:</li>
</ul>
          keys:
<ul class="list-disc pl-6 mb-4 space-y-2">
<li class="text-slate-300">name: key1</li>
</ul>
              secret: <base64-encoded-32-byte-key>
<ul class="list-disc pl-6 mb-4 space-y-2">
<li class="text-slate-300">identity: {}</li>
</ul>

Pass it to the API server with --encryption-provider-config. The identity provider at the end allows reading unencrypted data written before encryption was enabled. Best practice: Use aescbc or aesgcm for self-managed clusters. On managed clusters (EKS, AKS, GKE), enable the provider's envelope encryption with a KMS key you control.

After enabling encryption, re-encrypt existing secrets:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Network Access Control

etcd should never be reachable from worker nodes or pods. Restrict access with these controls:

1. Firewall rules: Only allow the API server's IP to reach etcd ports (2379/2380)
2. mTLS: Require client certificates for all etcd connections
3. Dedicated network: Run etcd on a separate subnet or VLAN from worker nodes
4. No public exposure: etcd should never have a public IP or be reachable from the internet

Mutual TLS for etcd

Configure etcd to require client certificate authentication:

etcd \
  --client-cert-auth=true \
  --trusted-ca-file=/etc/etcd/ca.crt \
  --cert-file=/etc/etcd/server.crt \
  --key-file=/etc/etcd/server.key \
  --peer-client-cert-auth=true \
  --peer-trusted-ca-file=/etc/etcd/ca.crt

The API server must present a valid client certificate signed by the same CA to connect.

Backup and Recovery

Backup etcd regularly but protect the backups. An etcd snapshot contains every secret in the cluster.

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/ca.crt \
  --cert=/etc/etcd/server.crt \
  --key=/etc/etcd/server.key

Encrypt backups at rest, store them in a separate security boundary, and restrict access to the backup location.