mTLS in Kubernetes

SPIFFE
TLS
Mutual TLS (mTLS)
Kubernetes
Sidecar Pattern & Sidecar Containers

Core Idea

A six-phase path to real mTLS in Kubernetes: PKI primer, manual certs with Secrets, automation with cert-manager, then scale-out with a service mesh and SPIFFE/SPIRE before production hardening.

Foundations: what mTLS is (standard vs mutual TLS), why Kubernetes needs it, core concepts, and the architecture overview.

The phases climb deliberately: TLS fundamentals and a hand-built CA, manual mTLS, cert-manager, Istio at scale, identity-centric SPIFFE/SPIRE.

Phase 6 covers production patterns and hardening, with a command quick reference and troubleshooting at the end.

Overview

This guide walks you through understanding and implementing mutual TLS (mTLS) in Kubernetes — from first principles through production-grade patterns. You’ll learn how standard TLS works, why mutual authentication matters in a zero-trust cluster, and how to implement mTLS using multiple approaches: manual certificate management, a service mesh (Istio), and SPIFFE/SPIRE.

What Is mTLS and Why Does It Matter in Kubernetes?

Standard TLS vs Mutual TLS

In standard TLS (what your browser uses with HTTPS), only the server presents a certificate. The client verifies the server’s identity, but the server has no cryptographic proof of who the client is.

Mutual TLS (mTLS) requires both sides to present and verify certificates. This means:

The client verifies the server is who it claims to be
The server verifies the client is who it claims to be
Communication is encrypted end-to-end

Standard TLS:
  Client ──── "Who are you?" ────► Server
  Client ◄─── certificate ──────── Server
  Client ──── (anonymous) ────────► Server
  Client ◄══════ encrypted ══════► Server

Mutual TLS:
  Client ──── "Who are you?" ────► Server
  Client ◄─── certificate ──────── Server
  Client ──── certificate ───────► Server  ← extra step
  Client ◄══════ encrypted ══════► Server
         (both sides verified)

Why Kubernetes Specifically Needs mTLS

In a Kubernetes cluster, the traditional network perimeter doesn’t protect you:

Pods share nodes; a compromised pod can reach any service on its network
IP addresses are ephemeral — you can’t trust “it came from 10.0.1.42”
Service accounts alone don’t authenticate at the transport layer
East-west traffic (pod-to-pod) is often unencrypted by default

mTLS solves this by moving identity into the connection itself rather than relying on network topology.

Core Concepts

Concept	Description
Certificate Authority (CA)	The trusted entity that signs certificates; both sides must trust the same CA
X.509 Certificate	A standard format for public key certificates; carries identity + public key
Private Key	The secret key that proves ownership of a certificate; never leaves the workload
Certificate Signing Request (CSR)	A request sent to a CA to get a certificate issued
TLS Handshake	The negotiation phase where both sides exchange and verify certificates before data flows
mTLS	TLS where the client also presents a certificate and the server verifies it
Certificate TTL	Time-to-live for a certificate; shorter TTLs reduce the blast radius of compromise
Trust Bundle	The set of CA certificates that a workload trusts when validating peers
SVID	SPIFFE Verifiable Identity Document — an X.509 cert whose SAN carries a `spiffe://` URI
PKI	Public Key Infrastructure — the system of CAs, certificates, and policies

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                     Kubernetes Cluster                      │
│                                                             │
│   ┌─────────────────────────────────────────────────────┐   │
│   │              Certificate Authority (CA)             │   │
│   │   (Kubernetes built-in / cert-manager / SPIRE)      │   │
│   └────────────────────┬────────────────────────────────┘   │
│                        │ signs certs                        │
│          ┌─────────────┼─────────────┐                      │
│          │             │             │                      │
│   ┌──────┴──────┐      │      ┌──────┴─────┐                │
│   │  Frontend   │      │      │   Backend  │                │
│   │    Pod      │      │      │    Pod     │                │
│   │             │      │      │            │                │
│   │ cert + key  │      │      │ cert + key │                │
│   └──────┬──────┘      │      └─────┬──────┘                │
│          │             │            │                       │
│          │    mTLS connection       │                       │
│          └─────────────────────────►│                       │
│           (both present & verify    │                       │
│            each other's certs)      │                       │
│                                     │                       │
└─────────────────────────────────────────────────────────────┘

Phase 1 — TLS Fundamentals and PKI Primer

Goal

Build a solid mental model of how certificates and TLS work before applying them in Kubernetes.

1.1 Key Cryptography Concepts

Asymmetric Cryptography

Every certificate involves a key pair:

Private key — kept secret, used to sign/decrypt
Public key — shared freely in the certificate, used to verify/encrypt

Anything encrypted with the public key can only be decrypted by the private key, and vice versa.

Certificate Chain of Trust

Root CA (self-signed, highly trusted)
  └── Intermediate CA (signed by Root CA)
        └── Leaf Certificate (signed by Intermediate CA)
              ← this is what your service presents

When a client receives a leaf certificate, it walks up the chain verifying signatures until it reaches a CA it trusts.

1.2 Hands-On: Create a CA and Issue Certificates with OpenSSL

This exercise builds intuition before using automated tooling.

# 1. Create a Root CA key and self-signed certificate
openssl genrsa -out ca.key 4096
openssl req -new -x509 -key ca.key -sha256 -days 365 \
  -out ca.crt \
  -subj "/CN=My Lab CA/O=Lab"
 
# 2. Create a key and CSR for the backend service
openssl genrsa -out backend.key 2048
openssl req -new -key backend.key \
  -out backend.csr \
  -subj "/CN=backend.default.svc.cluster.local/O=Lab"
 
# 3. Sign the backend CSR with our CA
openssl x509 -req -in backend.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out backend.crt -days 90 -sha256
 
# 4. Create key and CSR for the frontend (client)
openssl genrsa -out frontend.key 2048
openssl req -new -key frontend.key \
  -out frontend.csr \
  -subj "/CN=frontend.default.svc.cluster.local/O=Lab"
 
openssl x509 -req -in frontend.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out frontend.crt -days 90 -sha256
 
# 5. Inspect a certificate
openssl x509 -in backend.crt -noout -text
# Key fields to find:
#   Issuer:       CN=My Lab CA
#   Subject:      CN=backend.default.svc.cluster.local
#   Validity:     90 days
#   Subject Alternative Name (if set)

1.3 Test mTLS with OpenSSL Directly

# Terminal 1: Start a TLS server requiring client certs
openssl s_server \
  -cert backend.crt -key backend.key \
  -CAfile ca.crt \
  -Verify 1 \
  -port 8443
 
# Terminal 2: Connect as a client presenting a cert
openssl s_client \
  -connect localhost:8443 \
  -cert frontend.crt -key frontend.key \
  -CAfile ca.crt
 
# Look for:
# Verify return code: 0 (ok)
# Server certificate: CN=backend.default.svc.cluster.local
# Client certificate: CN=frontend.default.svc.cluster.local

What to Learn Here

Why certificate validation walks up to a trusted CA, not just checking the cert itself
The difference between a CA certificate and a leaf/end-entity certificate
What happens when you connect without a client cert to a server requiring one (SSL alert: certificate required)
How the Subject and Subject Alternative Name fields identify a service

Phase 2 — Manual mTLS in Kubernetes Using Secrets

Goal

Implement mTLS manually by creating TLS secrets and mounting them into pods. This teaches the mechanics before automation.

2.1 Store Certificates as Kubernetes Secrets

# Create a Kubernetes Secret with the CA cert (shared trust)
kubectl create secret generic ca-bundle \
  --from-file=ca.crt=ca.crt
 
# Create TLS secrets for each service
kubectl create secret tls backend-tls \
  --cert=backend.crt \
  --key=backend.key
 
kubectl create secret tls frontend-tls \
  --cert=frontend.crt \
  --key=frontend.key

2.2 Deploy the Backend with TLS

# backend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: backend
          image: nginx:alpine
          ports:
            - containerPort: 8443
          volumeMounts:
            - name: tls-certs
              mountPath: /etc/tls
              readOnly: true
            - name: ca-bundle
              mountPath: /etc/ca
              readOnly: true
            - name: nginx-config
              mountPath: /etc/nginx/conf.d
      volumes:
        - name: tls-certs
          secret:
            secretName: backend-tls
        - name: ca-bundle
          secret:
            secretName: ca-bundle
        - name: nginx-config
          configMap:
            name: backend-nginx-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: backend-nginx-config
data:
  default.conf: |
    server {
      listen 8443 ssl;
      ssl_certificate     /etc/tls/tls.crt;
      ssl_certificate_key /etc/tls/tls.key;
 
      # Require client certificate (mutual TLS)
      ssl_client_certificate /etc/ca/ca.crt;
      ssl_verify_client on;
 
      location / {
        return 200 "Hello from backend. Client: $ssl_client_s_dn\n";
        add_header Content-Type text/plain;
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: backend
spec:
  selector:
    app: backend
  ports:
    - port: 8443
      targetPort: 8443

kubectl apply -f backend-deployment.yaml

2.3 Test from the Frontend Pod

# Deploy a curl pod with the frontend certs mounted
kubectl run frontend --image=curlimages/curl:latest \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "frontend",
        "image": "curlimages/curl:latest",
        "command": ["sleep", "3600"],
        "volumeMounts": [
          {"name": "tls", "mountPath": "/etc/tls"},
          {"name": "ca", "mountPath": "/etc/ca"}
        ]
      }],
      "volumes": [
        {"name": "tls", "secret": {"secretName": "frontend-tls"}},
        {"name": "ca", "secret": {"secretName": "ca-bundle"}}
      ]
    }
  }'
 
# Test mTLS connection
kubectl exec -it frontend -- curl \
  --cert /etc/tls/tls.crt \
  --key /etc/tls/tls.key \
  --cacert /etc/ca/ca.crt \
  https://backend:8443/
 
# Expected: "Hello from backend. Client: CN=frontend.default.svc.cluster.local"
 
# Test without client cert — should fail
kubectl exec -it frontend -- curl \
  --cacert /etc/ca/ca.crt \
  https://backend:8443/
# Expected: SSL error: certificate required

2.4 Understanding the Kubernetes Secret Limitations

Manual certificate management has serious operational problems:

No automatic rotation — certificates expire, causing outages
Secrets visible in etcd — unless etcd encryption at rest is enabled
Manual distribution — you must update every secret when the CA rotates
No revocation — you can’t quickly revoke a compromised certificate

The following phases address these with automated tooling.

What to Learn Here

How tls.crt and tls.key are structured inside a Kubernetes Secret
How nginx (or any server) is configured to require client certificates
Why ssl_client_s_dn (the client’s distinguished name) can be used for application-level authz
The operational pain that drives adoption of cert-manager or a service mesh

Phase 3 — Automated Certificate Management with cert-manager

Goal

Use cert-manager to automate certificate issuance, renewal, and rotation, eliminating manual PKI operations.

3.1 Install cert-manager

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
 
# Wait for [[cert-manager]] pods to be ready
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/instance=cert-manager \
  -n cert-manager \
  --timeout=120s

3.2 Create a ClusterIssuer (Internal CA)

# ca-secret.yaml — store your CA keypair
apiVersion: v1
kind: Secret
metadata:
  name: cluster-ca-keypair
  namespace: cert-manager
type: kubernetes.io/tls
data:
  tls.crt: <base64-encoded ca.crt>
  tls.key: <base64-encoded ca.key>

# Encode and apply
kubectl create secret tls cluster-ca-keypair \
  --cert=ca.crt --key=ca.key \
  -n cert-manager

# cluster-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: cluster-ca-keypair

kubectl apply -f cluster-issuer.yaml
kubectl get clusterissuer internal-ca
# STATUS: True (Ready)

3.3 Issue Certificates with Certificate Resources

# backend-certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: backend-cert
  namespace: default
spec:
  secretName: backend-tls-auto        # cert-manager creates/manages this Secret
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
  duration: 24h                        # short TTL — cert-manager auto-renews
  renewBefore: 8h
  dnsNames:
    - backend
    - backend.default.svc.cluster.local
  usages:
    - server auth
    - client auth
---
# frontend-certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: frontend-cert
  namespace: default
spec:
  secretName: frontend-tls-auto
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
  duration: 24h
  renewBefore: 8h
  dnsNames:
    - frontend
    - frontend.default.svc.cluster.local
  usages:
    - server auth
    - client auth

kubectl apply -f backend-certificate.yaml
kubectl apply -f frontend-certificate.yaml
 
# Watch cert-manager issue the certificates
kubectl get certificate -w
# NAME            READY   SECRET              AGE
# backend-cert    True    backend-tls-auto    10s
# frontend-cert   True    frontend-tls-auto   10s
 
# Inspect the issued certificate
kubectl get secret backend-tls-auto -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | openssl x509 -noout -text | grep -A5 "Validity"

3.4 Automatic Rotation

cert-manager automatically renews certificates when renewBefore is reached. To test rotation behavior:

# Manually trigger renewal
kubectl annotate certificate backend-cert \
  cert-manager.io/issuer-name=internal-ca \
  --overwrite
 
# Watch the secret get updated
kubectl get secret backend-tls-auto -w
 
# Verify your application handles cert reload
# (nginx: HUP signal; Go apps using tls.Config with GetCertificate callback)

3.5 Monitoring Certificate Expiry

# Check all certificate statuses
kubectl get certificates -A
 
# Get time until expiry
kubectl get certificate backend-cert \
  -o jsonpath='{.status.notAfter}'
 
# [[cert-manager]] exposes Prometheus metrics
# cert_manager_certificate_expiration_timestamp_seconds
# cert_manager_certificate_ready_status

What to Learn Here

The Certificate → CertificateRequest → Secret lifecycle in cert-manager
How duration and renewBefore interact to give a renewal window
The difference between ClusterIssuer (cluster-wide) and Issuer (namespace-scoped)
How to use cert-manager with external CAs like Let’s Encrypt or Vault

Phase 4 — mTLS at Scale with a Service Mesh (Istio)

Goal

Use Istio to enforce mTLS transparently for all pod-to-pod communication without modifying application code.

4.1 Install Istio

# Download and install istioctl
curl -L https://istio.io/downloadIstio | sh -
export PATH=$PWD/istio-*/bin:$PATH
 
# Install with the demo profile (suitable for learning)
istioctl install --set profile=demo -y
 
# Verify
kubectl get pods -n istio-system
# istiod-xxx         Running  ← Pilot, CA, config
# istio-ingressgateway-xxx  Running

4.2 How Istio mTLS Works

Istio injects an Envoy sidecar proxy into each pod. The sidecar intercepts all inbound and outbound traffic and handles mTLS automatically:

Without Istio:
  Pod A ──── plain TCP ────► Pod B

With Istio:
  App A ──► Envoy A ════ mTLS ════ Envoy B ──► App B
  (127.0.0.1)           (encrypted)          (127.0.0.1)
              ↑                         ↑
         sidecar                   sidecar
    (handles TLS)             (handles TLS)

The application code in Pod A and Pod B sees plain HTTP — the sidecars handle all TLS operations transparently.

4.3 Enable Sidecar Injection

Istio / Installation Configuration Profiles

# Label the namespace to auto-inject Envoy sidecars
kubectl label namespace default istio-injection=enabled
 
# Deploy a sample application
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.21/samples/bookinfo/platform/kube/bookinfo.yaml
 
# Verify sidecars are injected (each pod should have 2 containers)
kubectl get pods
# NAME                       READY   STATUS
# productpage-xxx            2/2     Running   ← app + envoy
# reviews-xxx                2/2     Running
# details-xxx                2/2     Running

4.4 Enable Strict mTLS with PeerAuthentication

By default, Istio operates in PERMISSIVE mode (accepts both plain text and mTLS). Switch to STRICT to enforce mTLS for all traffic.

# strict-mtls.yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: default
spec:
  mtls:
    mode: STRICT

kubectl apply -f strict-mtls.yaml
 
# Test: traffic from a non-sidecar pod should now be rejected
kubectl run no-sidecar --image=curlimages/curl:latest \
  -n default \
  --command -- sleep 3600
 
# This should fail — no sidecar means no client cert
kubectl exec -it no-sidecar -- curl http://productpage:9080/
# upstream connect error or disconnect/reset before headers
 
# Traffic between sidecar-injected pods should work fine
kubectl exec -it $(kubectl get pod -l app=productpage -o name) \
  -c istio-proxy -- curl http://details:9080/details/0

4.5 Inspect mTLS in Action

# Check Istio's view of mTLS status between services
istioctl x describe pod $(kubectl get pod -l app=productpage -o name | head -1)
 
# View the TLS certificates Envoy is using
kubectl exec -it $(kubectl get pod -l app=productpage -o name | head-1) \
  -c istio-proxy -- \
  openssl s_client \
    -connect details:9080 \
    -showcerts 2>/dev/null | \
  openssl x509 -noout -text | grep -E "Subject:|Issuer:|URI:"
 
# You'll see SPIFFE-style URIs like:
# URI:spiffe://cluster.local/ns/default/sa/bookinfo-productpage

4.6 Authorization Policies

mTLS gives you verified identity; AuthorizationPolicy lets you use that identity for access control:

# Only allow frontend to call backend on port 8080
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: backend-authz
  namespace: default
spec:
  selector:
    matchLabels:
      app: backend
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/default/sa/frontend"
      to:
        - operation:
            ports: ["8080"]
            methods: ["GET"]

kubectl apply -f backend-authz.yaml
 
# Now only the frontend service account can reach backend on GET /
# Verify: curl from productpage to details should still work
# Verify: curl from a different service account should be denied (403)

4.7 Visualize the Service Mesh

# Open Kiali dashboard (comes with demo profile)
istioctl dashboard kiali
 
# Navigate to Graph → Namespace: default
# Green padlock icons = mTLS enforced
# Grey icons = plain text (should be none in STRICT mode)

What to Learn Here

The sidecar proxy pattern and how iptables rules redirect traffic transparently
The difference between STRICT, PERMISSIVE, and DISABLE mTLS modes
How Istio’s internal CA (Istiod) issues SPIFFE-compatible SVIDs to each sidecar
How PeerAuthentication (transport layer) and AuthorizationPolicy (application layer) complement each other
Certificate rotation in Istio: sidecars refresh certs every 24h by default (configurable)

4.8 Do You Need cert-manager if You Use Istio?

**No — not for pod-to-pod mTLS.** Istio ships with its own internal CA called Istiod that automatically issues short-lived, SPIFFE-compatible X.509 SVIDs to every Envoy sidecar and rotates them on a configurable interval (default 24h). For east-west traffic inside the mesh, this is fully self-contained.

However, cert-manager and Istio serve complementary roles and are commonly used together because Istiod’s CA only manages certificates for sidecar proxies — not for anything outside the mesh.

What each tool covers:

Use Case	Istio alone	Needs cert-manager
Pod-to-pod mTLS inside the mesh	✅	—
Ingress Gateway TLS (HTTPS termination)	Manual secret only	✅
Workloads outside the mesh	—	✅
Istiod’s own root CA certificate	Manual	✅
Integrating with an external CA (Vault, AWS PCA)	Limited	✅
Certificates for non-HTTP services	—	✅

The most common real-world reason to use both is Ingress TLS. Istio manages east-west certs automatically, but for north-south HTTPS traffic at the gateway you need cert-manager (or manual rotation) to provision and renew the public-facing certificate:

# [[cert-manager]] issues the cert for your Istio Ingress Gateway
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ingress-gateway-cert
  namespace: istio-system
spec:
  secretName: ingress-gateway-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - api.yourdomain.com

The third option: Istio + an external CA via istio-csr

If you want Istio to use your corporate CA instead of its self-signed root, you can use cert-manager’s istio-csr plugin. This creates a single PKI root of trust where Istiod delegates certificate signing to cert-manager, which in turn issues from Vault, AWS Private CA, or any other configured issuer:

External Root CA (Vault / AWS PCA)
  └── cert-manager issues intermediate CA cert
        └── Istiod uses that intermediate to sign all sidecar SVIDs

# Install istio-csr (bridges Istio's CSR API to cert-manager)
helm install cert-manager-istio-csr \
  jetstack/cert-manager-istio-csr \
  -n cert-manager \
  --set "app.certmanager.issuer.name=vault-issuer" \
  --set "app.certmanager.issuer.kind=ClusterIssuer"

Simple rule of thumb:

Mesh-only workloads, self-signed CA is acceptable → Istio alone is sufficient
You need HTTPS on your Ingress Gateway → add cert-manager
You have a corporate CA that must be the root of trust → add cert-manager + istio-csr
You need workload identity outside the mesh or across clusters → consider SPIRE

Phase 5 — mTLS with SPIFFE/SPIRE (Identity-Centric Approach)

Goal

Use SPIRE to issue cryptographic workload identities and consume them directly in application code for mTLS, without a service mesh proxy.

5.1 When to Use SPIRE Instead of a Service Mesh

Scenario	Prefer Istio/Linkerd	Prefer SPIRE
Transparent mTLS, no code changes	✅	—
Multi-cluster / multi-cloud identity	—	✅
Non-HTTP protocols (gRPC, raw TCP)	✅	✅
mTLS in application code	—	✅
Identity federation across orgs	—	✅
Lightweight (no sidecar overhead)	—	✅

5.2 Deploy SPIRE (Quick Setup)

git clone https://github.com/spiffe/spire-tutorials.git
cd spire-tutorials/k8s/quickstart
kubectl apply -f .
 
kubectl get pods -n spire
# spire-server-0    Running
# spire-agent-xxx   Running  (per node)

5.3 Register Workloads and Issue SVIDs

# Create registration entries
kubectl exec -n spire -it spire-server-0 -- \
  /opt/spire/bin/spire-server entry create \
  -spiffeID spiffe://example.org/ns/default/sa/backend \
  -parentID spiffe://example.org/spire/agent/k8s_psat/cluster/node \
  -selector k8s:ns:default \
  -selector k8s:sa:backend
 
kubectl exec -n spire -it spire-server-0 -- \
  /opt/spire/bin/spire-server entry create \
  -spiffeID spiffe://example.org/ns/default/sa/frontend \
  -parentID spiffe://example.org/spire/agent/k8s_psat/cluster/node \
  -selector k8s:ns:default \
  -selector k8s:sa:frontend

5.4 Use SVIDs for mTLS in Go

package main
 
import (
    "context"
    "net/http"
 
    "github.com/spiffe/go-spiffe/v2/spiffeid"
    "github.com/spiffe/go-spiffe/v2/spiffetls"
    "github.com/spiffe/go-spiffe/v2/spiffetls/tlsconfig"
    "github.com/spiffe/go-spiffe/v2/workloadapi"
)
 
// Backend: HTTP server with mTLS
func runBackend(ctx context.Context) error {
    // Fetch our SVID from the SPIRE agent socket
    source, err := workloadapi.NewX509Source(ctx)
    if err != nil {
        return err
    }
    defer source.Close()
 
    // Only allow clients with this SPIFFE ID
    frontendID := spiffeid.RequireFromString(
        "spiffe://example.org/ns/default/sa/frontend")
 
    listener, err := spiffetls.Listen(ctx, "tcp", ":8443",
        tlsconfig.MTLSServerConfig(source, source,
            tlsconfig.AuthorizeID(frontendID)))
    if err != nil {
        return err
    }
 
    return http.Serve(listener, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Authenticated response from backend\n"))
    }))
}
 
// Frontend: HTTP client with mTLS
func callBackend(ctx context.Context) error {
    source, err := workloadapi.NewX509Source(ctx)
    if err != nil {
        return err
    }
    defer source.Close()
 
    backendID := spiffeid.RequireFromString(
        "spiffe://example.org/ns/default/sa/backend")
 
    tlsCfg := tlsconfig.MTLSClientConfig(source, source,
        tlsconfig.AuthorizeID(backendID))
 
    client := &http.Client{
        Transport: &http.Transport{TLSClientConfig: tlsCfg},
    }
 
    resp, err := client.Get("https://backend:8443/")
    // handle resp...
    return err
}

Key insight: The X509Source automatically refreshes SVIDs as they rotate. Your application always has a valid certificate without any restart.

5.5 Verify SPIFFE Identity in the TLS Connection

# Inspect the SVID a workload receives
kubectl exec -n spire -it spire-agent-xxx -- \
  /opt/spire/bin/spire-agent api fetch x509 \
  -socketPath /run/spire/sockets/agent.sock \
  -write /tmp/
 
openssl x509 -in /tmp/svid.0.pem -noout -text | grep "URI:"
# URI:spiffe://example.org/ns/default/sa/frontend

What to Learn Here

The SPIFFE Workload API (a gRPC socket) and how it streams SVIDs to workloads
How to authorize connections by SPIFFE ID (spiffe://example.org/ns/default/sa/frontend) rather than raw cert attributes
Automatic certificate rotation: workloadapi.X509Source is a live source that always holds the current valid SVID
When application-level mTLS (SPIRE) is preferable to transparent mesh mTLS (Istio)

Phase 6 — Production Patterns and Hardening

6.1 Certificate Lifecycle Best Practices

Practice	Why
Short TTLs (1–24h)	Limits blast radius if a cert is compromised
Automate renewal (cert-manager / SPIRE)	Prevents expiry-related outages
Use separate CAs per environment	Dev CA compromise doesn’t affect prod
Store CA keys in an HSM or KMS	Prevents root CA key extraction
Monitor expiry metrics	Alert before, not after, expiry
Test rotation regularly	Verify apps handle cert swaps gracefully

6.2 Protecting Secrets at Rest

Kubernetes Secrets are base64-encoded, not encrypted, by default. Enable etcd encryption at rest:

# /etc/kubernetes/encryption-config.yaml (on API server)
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded 32-byte key>
      - identity: {}

Alternatively, use External Secrets Operator to sync secrets from AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault.

6.3 Network Policies Alongside mTLS

mTLS verifies identity but doesn’t restrict which pods can attempt connections. Layer NetworkPolicies on top:

# Only allow frontend to reach backend on port 8443
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-ingress
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8443

Defense in depth: NetworkPolicy limits which pods can even initiate a connection; mTLS ensures that even if a pod bypasses the network policy, it can’t authenticate without a valid certificate.

6.4 Observability for mTLS

Metrics to Monitor

# cert-manager
cert_manager_certificate_expiration_timestamp_seconds
cert_manager_certificate_ready_status

# Istio (via Prometheus)
istio_requests_total{connection_security_policy="mutual_tls"}
pilot_xds_push_errors

# SPIRE Server
spire_server_svid_rotations_total
spire_agent_attestations_total

Useful Debugging Commands

# Check TLS negotiation details
openssl s_client -connect service:8443 -showcerts 2>&1 | \
  grep -E "subject=|issuer=|Verify return code"
 
# Istio: view TLS mode for each service
istioctl x describe svc backend
 
# Istio: check if mTLS is being enforced on a pod
istioctl x describe pod backend-xxx
 
# cert-manager: view certificate events
kubectl describe certificate backend-cert
 
# Decode and inspect a TLS secret
kubectl get secret backend-tls -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | openssl x509 -noout -text
 
# Test mTLS connectivity from inside the cluster
kubectl run mtls-debug --rm -it --image=nicolaka/netshoot -- \
  openssl s_client -connect backend:8443 \
    -cert /path/to/client.crt -key /path/to/client.key \
    -CAfile /path/to/ca.crt

6.5 Production Hardening Checklist

Approach Comparison

	Manual (OpenSSL + Secrets)	cert-manager	Istio alone	Istio + cert-manager	SPIRE
Code changes needed	Yes	Yes	No	No	Yes
Auto cert rotation	No	Yes	Yes	Yes	Yes
Sidecar overhead	No	No	Yes	Yes	No
Ingress TLS automation	No	Yes	No	Yes	No
External / corporate CA	No	Yes	No	Yes (istio-csr)	Yes
Multi-cluster identity	No	Limited	Limited	Limited	Yes
Good for learning	Best	Good	Good	Good	Advanced
Good for production	No	Yes	Yes	Best	Yes
Protocol support	Any	Any	HTTP/gRPC	HTTP/gRPC	Any

Most production clusters run Istio + cert-manager together: Istiod handles east-west mTLS transparently while cert-manager automates Ingress Gateway TLS and optionally acts as the root CA via istio-csr.

Quick Reference: Common Commands

# Inspect any certificate
openssl x509 -in cert.crt -noout -text
 
# Decode a Kubernetes TLS secret and inspect
kubectl get secret my-tls-secret -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | openssl x509 -noout -text
 
# Test TLS connection
openssl s_client -connect host:port -showcerts
 
# Test mTLS connection
openssl s_client -connect host:port \
  -cert client.crt -key client.key -CAfile ca.crt
 
# Check cert expiry
echo | openssl s_client -connect host:port 2>/dev/null | \
  openssl x509 -noout -enddate
 
# cert-manager: force certificate renewal
kubectl annotate certificate <name> \
  cert-manager.io/issuer-name=<issuer> --overwrite
 
# Istio: check mTLS status
istioctl x describe pod <pod-name>
 
# Istio: enable strict mTLS on a namespace
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: default
spec:
  mtls:
    mode: STRICT
EOF
 
# SPIRE: list registration entries
kubectl exec -n spire spire-server-0 -- \
  /opt/spire/bin/spire-server entry show

Learning Resources

Resource	Type	URL
TLS 1.3 RFC 8446	Specification	https://tools.ietf.org/html/rfc8446
cert-manager Documentation	Official docs	https://cert-manager.io/docs/
Istio Security Concepts	Official docs	https://istio.io/latest/docs/concepts/security/
SPIFFE/SPIRE Guide	See companion guide	—
“Bulletproof TLS and PKI” (Feistás)	Book	ISBN 978-1907117084
Kubernetes Network Policy	Official docs	https://kubernetes.io/docs/concepts/services-networking/network-policies/
External Secrets Operator	GitHub	https://external-secrets.io/
Kiali (Istio dashboard)	Tool	https://kiali.io/

Troubleshooting

Symptom	Likely Cause	Fix
`certificate required` from server	Client didn’t send a cert	Pass `--cert` and `--key` in curl; verify volume mounts
`certificate signed by unknown authority`	Client doesn’t trust the server’s CA	Pass `--cacert` or mount the CA bundle
`certificate has expired`	Cert TTL elapsed	Renew with cert-manager; check `renewBefore` config
`x509: certificate is valid for X, not Y`	SAN doesn’t match the hostname being dialed	Add the correct DNS SAN to the Certificate resource
Istio: connection refused in STRICT mode	Pod has no sidecar	Check namespace label `istio-injection=enabled`
Istio: 403 Forbidden	AuthorizationPolicy denying the request	Check `istioctl x describe pod` for policy analysis
SPIRE: workload gets no SVID	No matching registration entry	Run `entry show`; verify selectors match pod metadata
cert-manager: Certificate stuck in `False`	Issuer misconfigured or CA unreachable	`kubectl describe certificaterequest` for events

Warning

Not Built-in Kubernetes does not natively enforce mTLS between pods. It must be layered on top via a Service Mesh or manual certificate management.

flowchart TD
    subgraph PodA["Pod A"]
        AppA["App Container"]
        SidecarA["Sidecar Proxy(Envoy / Linkerd)"]
    end

    subgraph PodB["Pod B"]
        SidecarB["Sidecar Proxy(Envoy / Linkerd)"]
        AppB["App Container"]
    end

    AppA -- plaintext --> SidecarA
    SidecarA -- mTLS --> SidecarB
    SidecarB -- plaintext --> AppB

    style SidecarA fill:#2d6a4f,color:#fff
    style SidecarB fill:#2d6a4f,color:#fff

The app containers communicate in plaintext internally; the sidecar proxies handle all mTLS transparently.

Option 1: Service Mesh (Recommended)

Service meshes like Istio and Linkerd automatically inject a sidecar proxy into each pod and handle certificate issuance, rotation, and policy enforcement.

Istio

Strict mTLS (namespace-wide):

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: your-namespace
spec:
  mtls:
    mode: STRICT

Permissive mode (useful during mesh migration — accepts both mTLS and plaintext):

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: your-namespace
spec:
  mtls:
    mode: PERMISSIVE

DestinationRule — enforce mTLS on outbound traffic:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: my-service-mtls
  namespace: your-namespace
spec:
  host: my-service
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

ISTIO_MUTUAL uses Istio-managed SPIFFE/SVID certificates. Use MUTUAL if you bring your own certificates.

Linkerd

Linkerd enables mTLS automatically for all meshed pods. Just annotate the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
  annotations:
    linkerd.io/inject: enabled   # Injects the Linkerd proxy sidecar
spec:
  ...

Linkerd uses SPIFFE identities and rotates certificates automatically — zero cert management required.

Option 2: Manual mTLS with Kubernetes Secrets

For advanced use cases only Manual cert management is error-prone. Prefer a service mesh or cert-manager in production.

Step 1 — Generate certificates:

# CA
openssl genrsa -out ca.key 4096
openssl req -new -x509 -days 365 -key ca.key -out ca.crt -subj "/CN=my-ca"
 
# Server cert
openssl genrsa -out server.key 4096
openssl req -new -key server.key -out server.csr -subj "/CN=my-service"
openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt
 
# Client cert
openssl genrsa -out client.key 4096
openssl req -new -key client.key -out client.csr -subj "/CN=my-client"
openssl x509 -req -days 365 -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt

Step 2 — Store as Kubernetes Secrets:

kubectl create secret generic server-tls \
  --from-file=tls.crt=server.crt \
  --from-file=tls.key=server.key \
  --from-file=ca.crt=ca.crt
 
kubectl create secret generic client-tls \
  --from-file=tls.crt=client.crt \
  --from-file=tls.key=client.key \
  --from-file=ca.crt=ca.crt

Step 3 — Mount certs in your Pod:

apiVersion: v1
kind: Pod
metadata:
  name: my-server
spec:
  containers:
    - name: app
      image: my-app:latest
      volumeMounts:
        - name: tls-certs
          mountPath: /etc/tls
          readOnly: true
  volumes:
    - name: tls-certs
      secret:
        secretName: server-tls

Option 3: cert-manager

cert-manager automates certificate issuance and renewal inside Kubernetes.

# Self-signed ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
 
---
# Certificate for a service
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-service-cert
  namespace: default
spec:
  secretName: my-service-tls
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
  dnsNames:
    - my-service.default.svc.cluster.local
  usages:
    - client auth
    - server auth

Nimendra's Notes 🪴

Explorer

mTLS in Kubernetes - A Hands-On Project

Overview

What Is mTLS and Why Does It Matter in Kubernetes?

Standard TLS vs Mutual TLS

Why Kubernetes Specifically Needs mTLS

Core Concepts

Architecture Overview

Phase 1 — TLS Fundamentals and PKI Primer

Goal

1.1 Key Cryptography Concepts

1.2 Hands-On: Create a CA and Issue Certificates with OpenSSL

1.3 Test mTLS with OpenSSL Directly

What to Learn Here

Phase 2 — Manual mTLS in Kubernetes Using Secrets

Goal

2.1 Store Certificates as Kubernetes Secrets

2.2 Deploy the Backend with TLS

2.3 Test from the Frontend Pod

2.4 Understanding the Kubernetes Secret Limitations

What to Learn Here

Phase 3 — Automated Certificate Management with cert-manager

Goal

3.1 Install cert-manager

3.2 Create a ClusterIssuer (Internal CA)

3.3 Issue Certificates with Certificate Resources

3.4 Automatic Rotation

3.5 Monitoring Certificate Expiry

What to Learn Here

Phase 4 — mTLS at Scale with a Service Mesh (Istio)

Goal

4.1 Install Istio

4.2 How Istio mTLS Works

4.3 Enable Sidecar Injection

4.4 Enable Strict mTLS with PeerAuthentication

4.5 Inspect mTLS in Action

4.6 Authorization Policies

4.7 Visualize the Service Mesh

What to Learn Here

4.8 Do You Need cert-manager if You Use Istio?

Phase 5 — mTLS with SPIFFE/SPIRE (Identity-Centric Approach)

Goal

5.1 When to Use SPIRE Instead of a Service Mesh

5.2 Deploy SPIRE (Quick Setup)

5.3 Register Workloads and Issue SVIDs

5.4 Use SVIDs for mTLS in Go

5.5 Verify SPIFFE Identity in the TLS Connection

What to Learn Here

Phase 6 — Production Patterns and Hardening

6.1 Certificate Lifecycle Best Practices

6.2 Protecting Secrets at Rest

6.3 Network Policies Alongside mTLS

6.4 Observability for mTLS

6.5 Production Hardening Checklist

Approach Comparison

Quick Reference: Common Commands

Learning Resources

Troubleshooting