Nimendra's Notes 🪴

07 - Certificates Extras

3 min read

kubernetes/security kubernetes/networking

Core Idea

When certificates expire the control plane crashes and kubectl fails with x509 errors; this note is the runbook to diagnose and recover, plus labs to rehearse it.

  • Checking certificate expirations, including the kubeadm built-in check.
  • Renewing expired certificates, with containerd-specific steps.
  • Practical verification labs: client authentication audit, expired cert simulation, and encryption verification.

1. Production Diagnostic & Recovery Commands

When certificates expire in a Kubernetes cluster, the control plane crashes. Nodes show NotReady, and kubectl requests fail with x509: certificate has expired. Below are the operational commands required to diagnose and recover a broken cluster.

A. Checking Certificate Expirations

Method 1: Using Kubeadm

If your cluster was provisioned with kubeadm, run the built-in check command:

kubeadm certs check-expiration

Expected Output: Displays a table of all control-plane certificates, their expiration dates, and the time remaining.

Method 2: Manual Scan (Shell Script)

If you are troubleshooting a custom cluster, run this script on the master node to check all certificates:

for cert in $(find /etc/kubernetes/pki -name "*.crt"); do
    echo "=== Certificate: $cert ==="
    openssl x509 -in "$cert" -noout -enddate -subject
    echo ""
done

B. Renewing Expired Certificates

Step 1: Run Kubeadm Renewal

If certificates have expired (or are close to expiring), renew them using:

sudo kubeadm certs renew all

This updates the certificate files in /etc/kubernetes/pki/ and updates the embedded certificates inside control-plane kubeconfig files (such as admin.conf, controller-manager.conf, and scheduler.conf).

Step 2: Update Your Personal Kubeconfig

The renewal command updates /etc/kubernetes/admin.conf, but it does not automatically copy it to your home directory. You must update your environment:

cp /etc/kubernetes/admin.conf ~/.kube/config
chown $(id -u):$(id -g) ~/.kube/config

Step 3: Restart Control Plane Containers

Because Kubernetes control plane components run as static pods, they cache the certificates in memory. After renewing them on disk, you must restart the containers:

# If using containerd (standard in modern clusters)
sudo crictl ps | grep -E "kube-apiserver|kube-controller-manager|kube-scheduler" | awk '{print $1}' | xargs -I {} sudo crictl stop {}

The local kubelet will automatically detect the stopped containers and start new ones, reloading the renewed certificates.

2. Practical Verification Labs (Hands-on Challenges)

To ensure you have mastered these concepts, try to complete the following self-guided labs:

Challenge 1: The Client Authentication Audit

  1. Access a running cluster.
  2. Locate the static pod manifest for the kube-apiserver (usually /etc/kubernetes/manifests/kube-apiserver.yaml).
  3. Identify the flags --client-ca-file, --tls-cert-file, and --tls-private-key-file.
  4. Run openssl x509 -in <tls-cert-file> -text -noout and identify the Subject Alternative Names (SANs). Are they correct?

Challenge 2: The Expired Cert Simulation

  1. Using the instructions in 03 - Manual Certificate Generation, create a client certificate for a test user, but set the validity to 1 day (-days 1).
  2. Embed the certificate in a kubeconfig file.
  3. Access the cluster.
  4. Wait 24 hours (or adjust system clock) and try to run kubectl get pods. Verify that the connection is rejected with an expired certificate error.

Challenge 3: Encryption Verification

  1. Enable secret encryption using 06 - Secrets Encryption at Rest & KMS v2.
  2. Write a script to fetch the raw database values directly from etcd.
  3. Verify that the values are encrypted and start with the k8s:enc: prefix.

Backlinks

Graph View