Nimendra's Notes 🪴

06 - Secrets Encryption at Rest & KMS v2

5 min read

kubernetes/networking kubernetes/security

Core Idea

Secrets sit in etcd and the API server writes them unencrypted by default (base64 is not encryption), so turn on at-rest encryption: AES-CBC for a quick win, KMS v2 for production.

  • The threat: read access to etcd, a snapshot backup, or root on the master exposes every Secret, including tokens, credentials, and certificates.
  • Provider configuration hierarchy, with a worked example enabling AES-CBC Secrets encryption end to end.
  • KMS v2 integration as the production standard, then verifying encryption in etcd and re-encrypting existing Secrets.

In Kubernetes, standard resources (like Pods, Deployments) and sensitive resources (like Secrets) are all stored in the etcd database. By default, the Kube API server writes Secrets into etcd in plaintext (or rather, unencrypted base64-encoded text).

If an attacker gains read access to the etcd server, reads a database snapshot backup, or gets root access to the master node, they can extract all Secrets (including API tokens, database credentials, and certificates).

To understand basic cryptographic concepts before proceeding, see 01 - Cryptography and PKI Foundations.

1. Why Base64 is NOT Encryption

Many beginners confuse encoding with encryption:

  • Base64 Encoding: A public, standard algorithm that translates binary data into ASCII text. Its purpose is compatibility (transmitting data over channels that only support text). It requires no key to decode. Anyone can run echo "dGVzdA==" | base64 --decode and read the data.
  • Encryption: A cryptographic process that uses a secret key to render data unreadable. It is mathematically impossible to decrypt the data without the key.

Therefore, Kubernetes Secrets are not secure by default. We must configure the API server to perform encryption before writing data to etcd.

2. Encryption Provider Configuration Hierarchy

Kubernetes supports multiple encryption providers. They are evaluated in order inside an EncryptionConfiguration file. The primary providers are:

  1. kms (Recommended for Production - Stable in v1.29+): Uses an external Key Management Service (KMS) provider (such as AWS KMS, Azure Key Vault, Google Cloud KMS, or HashiCorp Vault) via a gRPC plugin.
  2. aescbc: AES-CBC with a 256-bit key. High strength, uses standard PKCS#7 padding.
  3. secretbox: XSalsa20 and Poly1305. A modern, fast symmetric encryption algorithm.
  4. aesgcm: AES-GCM with a 256-bit key. High speed, authenticated encryption.
  5. identity (Default): Writes plaintext to etcd. If placed first, no encryption occurs.

Worked Example 1: Enabling AES-CBC Secrets Encryption

This example demonstrates how to configure the API server to encrypt all Secrets written to etcd using the aescbc provider.

Step 1: Generate a Secure Key

We need a 32-byte (256-bit) random string, base64-encoded, to act as our symmetric encryption key.

head -c 32 /dev/urandom | base64

Example Output:

c3VwZXJzZWNyZXRra2V5MTIzNDU2Nzg5MGFiY2RlZg==

Step 2: Create the EncryptionConfiguration Manifest

Create a file named /etc/kubernetes/enc/enc-config.yaml on your master node. Ensure the directory exists and permissions are restricted (only root should read this file).

mkdir -p /etc/kubernetes/enc

Write the configuration:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: c3VwZXJzZWNyZXRra2V5MTIzNDU2Nzg5MGFiY2RlZg== # Use your generated key
      - identity: {} # Fallback provider (needed for backwards compatibility)

IMPORTANT

The order of the `providers` list is critical. The first provider is used to write new data. The subsequent providers are only used for reading existing data. Having identity: {} second ensures that the API server can still read old, unencrypted Secrets.

Step 3: Configure Kube-APIServer

The API server must be told to use this configuration.

  1. Open the API server static manifest (typically /etc/kubernetes/manifests/kube-apiserver.yaml).
  2. Add the flag: 
    spec:
  containers:
  - command:
    - kube-apiserver
    # ... other flags ...
    - --encryption-provider-config=/etc/kubernetes/enc/enc-config.yaml
  3. Mount the directory containing your configuration file into the API server container, so the container can access it: 
    # Under volumeMounts:
volumeMounts:
- mountPath: /etc/kubernetes/enc
  name: enc-config
  readOnly: true
 
# Under volumes:
volumes:
- hostPath:
    path: /etc/kubernetes/enc
    type: DirectoryOrCreate
  name: enc-config

When you save the file, the Kubelet on the master node will automatically detect the file change and restart the kube-apiserver static pod.

Worked Example 2: KMS v2 Integration (Production Standard)

Kubernetes KMS v2 is GA and represents the modern enterprise standard for secrets encryption.

Why KMS v2?

  • Envelope Encryption: KMS v2 uses key hierarchy. The API server generates a unique Data Encryption Key (DEK) for each secret, encrypts the secret using AES-GCM, and then sends the DEK to the KMS provider to be encrypted using the KMS’s Key Encryption Key (KEK).
  • Auto-Rotation: KMS v2 supports automatic rotation of KEKs without requiring an API server restart.
  • Performance: KMS v2 caches encrypted DEKs locally, significantly reducing the latency of secret reads/writes by avoiding constant network trips to the external KMS cloud endpoint.

KMS v2 EncryptionConfiguration Example

To set up KMS v2, write an EncryptionConfiguration referencing the Unix socket of your KMS plugin (which runs as a sidecar/daemon on the master node):

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      # KMS v2 provider configuration
      - kms:
          apiVersion: v2
          name: aws-kms-provider
          endpoint: unix:///var/run/kmsplugin/socket.sock # Socket where KMS plugin listens
          timeout: 3s
      - identity: {} # Fallback for reading unencrypted data

Encrypted secrets written to etcd via KMS v2 will have the prefix k8s:enc:kms:v2:.

3. Verify Secret Encryption in etcd

Let’s verify that secrets are indeed written in ciphertext.

  1. Create a new secret:

    kubectl create secret generic test-secret --from-literal=token=super-secret-password

  2. Access etcd directly using etcdctl:
    Connect to etcd using its TLS client keys and query the key /registry/secrets/default/test-secret:

    ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/apiserver-etcd-client.key \
  --endpoints=https://127.0.0.1:2379 \
  get /registry/secrets/default/test-secret

  3. Check the output:

    • If unencrypted, you would see the clear plaintext string super-secret-password inside the output payload.
    • If encrypted, the output will start with the prefix k8s:enc:aescbc:v1:key1 or k8s:enc:kms:v2: followed by binary, unreadable garbage.

4. Encrypt Existing Secrets

Changing the config only encrypts new or updated secrets. Existing secrets in etcd remain unencrypted. To encrypt all existing secrets, trigger a rewrite by updating them:

# This command fetches all Secrets and replaces them with themselves, triggering a write using the new encryption key.
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Now all Secrets in the cluster database are encrypted at rest!

Backlinks

Graph View