How to encrypt Kubernetes secrets at rest

Kubernetes Secrets look protected, but by default they are stored in etcd as nothing more than base64-encoded strings — not encrypted at all. Anyone with read access to etcd's data directory or its backups can decode every credential, API token, and TLS key in your cluster in seconds. For teams operating under SOC 2, PCI-DSS, or basic internal security policy, that gap is a finding waiting to happen.

This guide walks through how to encrypt Kubernetes secrets at rest, from understanding the default (unsafe) behavior through configuring an EncryptionConfiguration resource, wiring up a KMS provider for production-grade key management, re-encrypting existing data, and verifying the result. By the end, you'll have a working k8s secrets encryption setup that satisfies auditors and closes off one of the most commonly overlooked attack surfaces in self-managed Kubernetes.

Step 1: Confirm You Actually Need to Encrypt Kubernetes Secrets at Rest

Before changing anything, verify your current exposure. If you're running a managed control plane (EKS, GKE, AKS), encryption at rest may already be handled by the cloud provider — though it's worth confirming, since "encrypted disks" and "application-layer secrets encryption" are not the same guarantee. If you self-manage the control plane (kubeadm, kops, on-prem), you almost certainly need to configure this yourself.

Check what's actually sitting in etcd right now:

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-secret | hexdump -C | head

If you see readable base64 fragments or plaintext-looking strings rather than an opaque binary blob prefixed with k8s:enc:, your secrets are stored unencrypted. That prefix is the marker Kubernetes uses once encryption at rest is active — it's the first thing to check when verifying later steps.

Step 2: Choose an Encryption Provider

Kubernetes' EncryptionConfiguration supports several providers, and the choice matters:

• identity — no encryption (the default, and not a real option here)
• aescbc — AES-CBC with PKCS#7 padding, key stored in a local config file
• aesgcm — AES-GCM, requires periodic key rotation to stay safe against nonce collisions
• secretbox — XSalsa20-Poly1305, solid performance, still a local key
• kms (v2) — delegates the actual data-encryption key to an external Key Management Service (AWS KMS, GCP Cloud KMS, Azure Key Vault, HashiCorp Vault)

For anything beyond a lab cluster, use a KMS encryption provider for Kubernetes rather than a locally-stored AES key. Local keys sitting in a file on the control plane node reintroduce the exact problem you're trying to solve — a single compromised host exposes every secret. KMS v2 (stable since Kubernetes 1.29) also performs far better than v1, since it caches data encryption keys locally and only calls out to the remote KMS for key wrapping, not per-object encryption.

Step 3: Write the etcd Encryption Configuration

The core of your k8s secrets encryption setup is the EncryptionConfiguration resource, which tells the API server which resources to encrypt and with what provider. Create it on every control plane node, typically at /etc/kubernetes/enc/encryption-config.yaml:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      - kms:
          apiVersion: v2
          name: my-kms-provider
          endpoint: unix:///var/run/kmsplugin/socket.sock
          timeout: 3s
      - identity: {}

A few details that trip people up:

• Provider order matters. Kubernetes encrypts new writes with the first provider listed and can decrypt with any provider in the list — this is how you migrate safely.
• Always keep identity: {} last during rollout, never first. Listing it first would mean "don't encrypt" wins.
• Restrict which resources you encrypt deliberately. Encrypting secrets is non-negotiable; encrypting configmaps too is common if you store sensitive values there, but it does add overhead.

If you're not ready to stand up a full KMS integration yet, an interim aescbc configuration looks like this — useful for closing the gap immediately while you plan the KMS rollout:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

Generate the key with head -c 32 /dev/urandom | base64, and treat that file with the same care as a root credential — restrict it to 600 permissions, owned by root, and never commit it to version control.

Step 4: Deploy a KMS Plugin (for the KMS Provider Path)

If you chose the KMS route, the API server talks to a local gRPC plugin over a Unix socket, and that plugin talks to your cloud KMS. Each provider ships or documents one:

• AWS: aws-encryption-provider
• GCP: k8s-cloud-kms-plugin
• Azure: kubernetes-kms (Azure Key Vault provider)
• Vault: HashiCorp's vault-kms-plugin

Run it as a static pod or systemd unit on each control plane node so it starts before kube-apiserver does. A minimal static pod manifest for AWS looks like:

apiVersion: v1
kind: Pod
metadata:
  name: aws-encryption-provider
  namespace: kube-system
spec:
  containers:
    - name: aws-encryption-provider
      image: registry.k8s.io/provider-aws/aws-encryption-provider:v0.4.0
      args:
        - --key=arn:aws:kms:us-east-1:111122223333:key/your-key-id
        - --region=us-east-1
        - --listen=/var/run/kmsplugin/socket.sock
      volumeMounts:
        - name: kmssocket
          mountPath: /var/run/kmsplugin
  volumes:
    - name: kmssocket
      hostPath:
        path: /var/run/kmsplugin
        type: DirectoryOrCreate

Scope the KMS key's IAM/IAM-equivalent policy tightly: the plugin needs Encrypt and Decrypt, nothing else. Enable key rotation on the KMS key itself if your provider supports automatic rotation — this happens independently of Kubernetes and adds defense in depth.

Step 5: Point kube-apiserver at the Encryption Config

Add the flag to your API server's static pod manifest (/etc/kubernetes/manifests/kube-apiserver.yaml) and mount the config file in:

spec:
  containers:
    - command:
        - kube-apiserver
        - --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml
        # ...other existing flags
      volumeMounts:
        - name: encryption-config
          mountPath: /etc/kubernetes/enc
          readOnly: true
  volumes:
    - name: encryption-config
      hostPath:
        path: /etc/kubernetes/enc
        type: DirectoryOrCreate

Kubelet watches the static pod manifest directory and will restart kube-apiserver automatically once it detects the change. Repeat this on every control plane node — an inconsistent configuration across nodes causes intermittent decryption failures depending on which API server instance handles a given request.

Step 6: Re-Encrypt Existing Secrets

Enabling encryption only affects new writes. Every secret written before this point is still sitting in etcd unencrypted. Force a rewrite of everything:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

For large clusters, do this namespace by namespace to avoid a thundering herd against the API server, and run it during a low-traffic window:

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  kubectl get secrets -n "$ns" -o json | kubectl replace -f -
done

If you also encrypted configmaps, repeat the same command with configmaps substituted in.

Verification and Troubleshooting

Confirm secrets are actually encrypted. Re-run the etcdctl read from Step 1 against a secret you just rewrote. You should now see the k8s:enc:kms:v2: (or k8s:enc:aescbc:v1:) prefix followed by opaque binary data — no readable fragments.

API server crash-loops after adding the flag. Almost always a path or permissions issue: the encryption config file isn't mounted correctly, or the KMS plugin's socket doesn't exist yet because the plugin hasn't started. Check kubectl logs isn't an option here since the API server itself is down — read the container logs directly via the container runtime (crictl logs) on the node, and check journalctl -u kubelet for manifest errors.

"context deadline exceeded" from the KMS provider. The plugin can't reach your cloud KMS endpoint, or the timeout in EncryptionConfiguration is too aggressive for cross-region calls. Verify network egress from control plane nodes to the KMS service and confirm IAM/role bindings are attached to the node, not just a pod that doesn't yet exist.

Some secrets still show plaintext-looking data after re-encryption. Check that you didn't miss a namespace, and confirm the encryption config was applied consistently across all control plane nodes before you ran the bulk rewrite — a secret written while hitting a not-yet-updated API server instance won't get encrypted.

Key rotation. To rotate an aescbc key, add the new key above the old one in the keys list, restart the API server, then re-run the bulk re-encrypt command so everything moves to the new key before you remove the old one from the list. For KMS-based encryption, rotation is typically handled by the cloud provider's key rotation policy and requires no Kubernetes-side re-encryption unless you're rotating the KMS key ARN itself.

How Safeguard Helps

Getting etcd encryption configuration right is only half the job — the harder part is proving it stayed right, across every cluster, every control plane node, and every upgrade. Safeguard continuously scans your Kubernetes environments as part of its software supply chain security posture checks, flagging clusters where EncryptionConfiguration is missing, misordered, or silently reverted after a control plane upgrade wiped a static pod manifest.

Safeguard also tracks which secrets were written before encryption was enabled and were never re-encrypted, so you're not relying on tribal knowledge or a one-time migration script to catch drift. Combined with Safeguard's broader supply chain visibility — image provenance, dependency risk, and CI/CD pipeline integrity — encrypting Kubernetes secrets at rest becomes one verified control in a continuously monitored chain, rather than a checkbox you hope nobody unchecks during the next cluster rebuild.