SPIFFE-SPIRE in Kubernetes A Hands-On Project

kubernetes kubernetes/networking kubernetes/security

SPIFFE

Core Idea

This hands-on runs SPIFFE identity end to end in Kubernetes: deploy SPIRE server and agents, register workloads to get SVIDs, then use them for mTLS between services.

Phases: architecture, deploy the server (StatefulSet) and agent (DaemonSet) via quickstart or Helm, then register workloads and fetch SVIDs.

Phase 4 implements mTLS between services with the go-spiffe library and verifies the handshake.

Advanced: automatic registration, federation between trust domains, JWT SVIDs, observability, plus a production hardening checklist, quick reference commands, and troubleshooting.

Architecture Diagram

┌─────────────────────────────────────────────────────────┐
│                   Kubernetes Cluster                    │
│                                                         │
│  ┌─────────────────────────────────────────────────┐   │
│  │              SPIRE Server (Deployment)           │   │
│  │  - CA / Root of Trust                           │   │
│  │  - Registration entries datastore               │   │
│  │  - Issues X.509 SVIDs to Agents                 │   │
│  └──────────────────┬──────────────────────────────┘   │
│                     │ mTLS (node attestation)           │
│     ┌───────────────┼───────────────┐                  │
│     │               │               │                   │
│  ┌──┴──┐         ┌──┴──┐         ┌──┴──┐              │
│  │Agent│         │Agent│         │Agent│  (DaemonSet)  │
│  └──┬──┘         └──┬──┘         └──┬──┘              │
│     │               │               │                   │
│  Workload         Workload        Workload              │
│  API socket       API socket      API socket            │
│     │               │                                   │
│  ┌──┴──────┐   ┌────┴────┐                             │
│  │ Pod A   │   │  Pod B  │  ← fetch SVIDs via socket   │
│  │frontend │   │backend  │                             │
│  └─────────┘   └─────────┘                             │
│                                                         │
└─────────────────────────────────────────────────────────┘

Phase 2 — Deploy SPIRE Server and Agent

Goal

Install a functioning SPIRE deployment using the official Kubernetes manifests or Helm chart.

Option A: Official SPIRE Kubernetes Quickstart

# Clone the SPIRE examples repo
git clone https://github.com/spiffe/spire-tutorials.git
cd spire-tutorials/k8s/quickstart

Apply the manifests in order:

# 1. Namespace and RBAC
kubectl apply -f spire-namespace.yaml
kubectl apply -f server-account.yaml
kubectl apply -f spiffe-csi-driver.yaml
 
# 2. SPIRE Server (StatefulSet)
kubectl apply -f server-cluster-role.yaml
kubectl apply -f server-configmap.yaml
kubectl apply -f server-statefulset.yaml
kubectl apply -f server-service.yaml
 
# 3. SPIRE Agent (DaemonSet)
kubectl apply -f agent-account.yaml
kubectl apply -f agent-cluster-role.yaml
kubectl apply -f agent-configmap.yaml
kubectl apply -f agent-daemonset.yaml

Verify everything is running:

kubectl get pods -n spire
# Expected:
# spire-server-0   Running
# spire-agent-xxxx Running  (one per node)

Option B: Helm Chart

helm repo add spiffe https://spiffe.github.io/helm-charts-hardened/
helm repo update
 
helm install spire spiffe/spire \
  --namespace spire \
  --create-namespace \
  --set global.spiffe.trustDomain=example.org

Key Config Files to Understand

Server ConfigMap (server-configmap.yaml) — defines:

trust_domain — the SPIFFE trust domain
bind_address and bind_port
Node attestor plugin (k8s_psat — Projected Service Account Tokens)
Datastore (SQLite for dev, PostgreSQL for prod)
Key manager (disk or memory)

Agent ConfigMap (agent-configmap.yaml) — defines:

server_address — how the agent finds the server
Node attestor (k8s_psat)
Workload attestor (k8s) — uses pod metadata to match registration entries

What to Learn Here

The difference between node attestation (agent proves it’s a legitimate Kubernetes node) and workload attestation (agent proves a pod matches a registration entry)
How k8s_psat (Projected Service Account Tokens) works as an attestation mechanism
The StatefulSet vs DaemonSet distinction for server vs agent

Phase 3 — Register Workloads and Issue SVIDs

Goal

Create registration entries so that pods receive SVIDs automatically.

Exec into the SPIRE Server

kubectl exec -n spire -it spire-server-0 -- /bin/sh

Create Registration Entries

Every workload that needs an identity must have a registration entry on the SPIRE Server. The entry maps Kubernetes selectors to a SPIFFE ID.

# Register a frontend workload
/opt/spire/bin/spire-server entry create \
  -spiffeID spiffe://example.org/ns/default/sa/frontend \
  -parentID spiffe://example.org/k8s-workload-registrar/spire-agent \
  -selector k8s:ns:default \
  -selector k8s:sa:frontend
 
# Register a backend workload
/opt/spire/bin/spire-server entry create \
  -spiffeID spiffe://example.org/ns/default/sa/backend \
  -parentID spiffe://example.org/k8s-workload-registrar/spire-agent \
  -selector k8s:ns:default \
  -selector k8s:sa:backend

Verify Entries

/opt/spire/bin/spire-server entry show

Check Agent Registration

/opt/spire/bin/spire-server agent list

Fetch an SVID from a Workload Pod

Deploy a test workload with access to the SPIRE Workload API socket, then use spire-agent api fetch:

# Inside an agent pod or a workload pod with the socket mounted:
/opt/spire/bin/spire-agent api fetch x509 \
  -socketPath /run/spire/sockets/agent.sock

You should receive a PEM-encoded X.509 SVID with your SPIFFE ID as the Subject Alternative Name (SAN).

# Inspect the certificate
openssl x509 -in svid.pem -noout -text | grep -A2 "Subject Alternative Name"
# URI:spiffe://example.org/ns/default/sa/frontend

What to Learn Here

How selectors (k8s:ns, k8s:sa, k8s:pod-label) control which pods match an entry
The parent-child relationship between agent entries and workload entries
What an X.509 SVID looks like and how it encodes the SPIFFE ID in the SAN field
Certificate TTLs and automatic rotation

Phase 4 — Implement mTLS Between Services

Goal

Use SVIDs to establish mutually authenticated TLS between two microservices, without any shared secrets or manually managed certificates.

Sample Application

Deploy two services: frontend and backend. Each uses the SPIRE Workload API to fetch its SVID and validates the peer’s SVID during the TLS handshake.

Workload API socket mount (in Pod spec):

volumes:
  - name: spire-agent-socket
    csi:
      driver: "csi.spiffe.io"
      readOnly: true
 
containers:
  - name: frontend
    image: your-app:latest
    volumeMounts:
      - name: spire-agent-socket
        mountPath: /run/spire/sockets
        readOnly: true
    serviceAccountName: frontend

Using the SPIFFE go-spiffe Library

import (
    "context"
    "github.com/spiffe/go-spiffe/v2/spiffetls"
    "github.com/spiffe/go-spiffe/v2/workloadapi"
    "github.com/spiffe/go-spiffe/v2/spiffeid"
)
 
// Backend: listen with mTLS
func runBackend() {
    ctx := context.Background()
    source, _ := workloadapi.NewX509Source(ctx)
    defer source.Close()
 
    backendID := spiffeid.RequireFromString("spiffe://example.org/ns/default/sa/backend")
    
    listener, _ := spiffetls.Listen(ctx, "tcp", ":8443",
        tlsconfig.MTLSServerConfig(source, source,
            tlsconfig.AuthorizeID(backendID)))
    
    // handle connections...
}
 
// Frontend: dial with mTLS
func callBackend() {
    ctx := context.Background()
    source, _ := workloadapi.NewX509Source(ctx)
    defer source.Close()
 
    backendID := spiffeid.RequireFromString("spiffe://example.org/ns/default/sa/backend")
 
    conn, _ := spiffetls.Dial(ctx, "tcp", "backend:8443",
        tlsconfig.MTLSClientConfig(source, source,
            tlsconfig.AuthorizeID(backendID)))
    
    // use conn...
}

Verify mTLS Is Working

# Check TLS handshake and peer certificate
openssl s_client -connect backend-service:8443 \
  -cert frontend-svid.pem \
  -key frontend-key.pem \
  -CAfile bundle.pem \
  -verify_return_error

Look for:

Verify return code: 0 (ok)
Peer certificate SAN containing spiffe://example.org/ns/default/sa/backend

What to Learn Here

The SPIFFE Workload API (gRPC) and how it streams SVIDs + bundles
How certificate rotation is handled transparently (the X509Source auto-refreshes)
The difference between mTLS (both sides authenticate) and standard TLS (server only)
How to authorize connections by SPIFFE ID, not just “is a valid cert”

Phase 5 — Advanced Topics

5.1 Automatic Workload Registration with SPIRE Controller Manager

Manually creating entries doesn’t scale. The SPIRE Controller Manager watches Kubernetes resources and auto-creates registration entries via ClusterSPIFFEID CRDs.

# ClusterSPIFFEID — auto-register all pods in the default namespace
apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
  name: default-workloads
spec:
  spiffeIDTemplate: "spiffe://{{ .TrustDomain }}/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}"
  podSelector:
    matchLabels:
      spiffe.io/spiffe-id: "true"

5.2 Federation Between Trust Domains

SPIFFE federation allows workloads in different trust domains (e.g., different clusters or cloud providers) to establish trust.

# Configure federation on Server A to trust Server B's bundle
/opt/spire/bin/spire-server federation create \
  -bundleEndpointURL https://spire-server-b:8443 \
  -bundleEndpointProfile https_spiffe \
  -endpointSPIFFEID spiffe://cluster-b.example.org/spire/server \
  -trustDomainID spiffe://cluster-b.example.org

5.3 JWT SVIDs

For HTTP APIs that can’t do mTLS, SPIRE also issues JWT SVIDs — short-lived JWT tokens carrying the SPIFFE ID as the sub claim.

# Fetch a JWT SVID for a specific audience
/opt/spire/bin/spire-agent api fetch jwt \
  -socketPath /run/spire/sockets/agent.sock \
  -audience api.backend.example.org

The receiving service validates the JWT using the JWKS endpoint served by the SPIRE Server.

5.4 Observability

# SPIRE Server health check
kubectl exec -n spire spire-server-0 -- \
  /opt/spire/bin/spire-server healthcheck
 
# Prometheus metrics (if enabled)
kubectl port-forward -n spire svc/spire-server 8080:8080
curl http://localhost:8080/metrics | grep spire
 
# Audit logs
kubectl logs -n spire spire-server-0 | grep -i audit

5.5 Production Hardening Checklist

Use PostgreSQL or MySQL as the SPIRE datastore (not SQLite)
Enable HA for the SPIRE Server with multiple replicas + shared DB
Store root CA keys in an HSM or cloud KMS (AWS KMS, GCP Cloud KMS)
Set short SVID TTLs (1h or less) with appropriate clock skew tolerance
Enable Kubernetes audit logging for PSAT attestation forensics
Restrict SPIRE Server API access with Kubernetes NetworkPolicies
Set up bundle endpoint with HTTPS for federation
Use ClusterSPIFFEID with pod label selectors, not open namespace selectors

Learning Resources

Resource	Type	Link
SPIFFE Specification	RFC-style docs	https://github.com/spiffe/spiffe
SPIRE Documentation	Official docs	https://spiffe.io/docs/latest/spire-about/
SPIRE Tutorials Repo	Hands-on examples	https://github.com/spiffe/spire-tutorials
go-spiffe Library	Go SDK	https://github.com/spiffe/go-spiffe
SPIFFE/SPIRE Book (O’Reilly)	In-depth book	”Solving the Bottom Turtle” (free PDF at spiffe.io)
CNCF Slack `#spiffe`	Community	https://slack.cncf.io
SPIRE Controller Manager	Auto-registration	https://github.com/spiffe/spire-controller-manager

Quick Reference: Common Commands

# List all SPIRE agents
kubectl exec -n spire spire-server-0 -- \
  /opt/spire/bin/spire-server agent list
 
# List all registration entries
kubectl exec -n spire spire-server-0 -- \
  /opt/spire/bin/spire-server entry show
 
# Create an entry
kubectl exec -n spire spire-server-0 -- \
  /opt/spire/bin/spire-server entry create \
  -spiffeID spiffe://example.org/myservice \
  -parentID spiffe://example.org/spire/agent/k8s_psat/cluster/node-name \
  -selector k8s:ns:default \
  -selector k8s:sa:myservice
 
# Delete an entry by ID
kubectl exec -n spire spire-server-0 -- \
  /opt/spire/bin/spire-server entry delete -entryID <id>
 
# Fetch X.509 SVID (run from inside a workload or agent pod)
/opt/spire/bin/spire-agent api fetch x509 \
  -socketPath /run/spire/sockets/agent.sock \
  -write /tmp/
 
# Fetch JWT SVID
/opt/spire/bin/spire-agent api fetch jwt \
  -socketPath /run/spire/sockets/agent.sock \
  -audience my-api
 
# Get server bundle (root CA)
kubectl exec -n spire spire-server-0 -- \
  /opt/spire/bin/spire-server bundle show
 
# Check server health
kubectl exec -n spire spire-server-0 -- \
  /opt/spire/bin/spire-server healthcheck -shallow

Troubleshooting

Symptom	Likely Cause	Fix
Agent not attesting	PSAT token not projected into agent pod	Check `serviceAccountToken` volume in agent DaemonSet
Workload gets no SVID	No matching registration entry	Run `entry show` and verify selectors match pod metadata
Certificate expired immediately	Clock skew between nodes	Sync NTP; check `default_x509_svid_ttl` config
Agent can’t reach server	Network policy blocking port 8081	Allow agent → server on port 8081 in NetworkPolicy
mTLS handshake fails	Trust bundle mismatch	Run `bundle show` on both sides; refresh if federation is stale
`no identity issued` from Workload API	Workload socket path wrong	Confirm CSI driver or hostPath mount matches `agent.conf` socket path

Nimendra's Notes 🪴

Explorer

SPIFFE-SPIRE in Kubernetes A Hands-On Project

Architecture Diagram

Phase 2 — Deploy SPIRE Server and Agent

Goal

Option A: Official SPIRE Kubernetes Quickstart

Option B: Helm Chart

Key Config Files to Understand

What to Learn Here

Phase 3 — Register Workloads and Issue SVIDs

Goal

Exec into the SPIRE Server

Create Registration Entries

Verify Entries

Check Agent Registration

Fetch an SVID from a Workload Pod

What to Learn Here

Phase 4 — Implement mTLS Between Services

Goal

Sample Application

Using the SPIFFE go-spiffe Library

Verify mTLS Is Working

What to Learn Here

Phase 5 — Advanced Topics

5.1 Automatic Workload Registration with SPIRE Controller Manager

5.2 Federation Between Trust Domains

5.3 JWT SVIDs

5.4 Observability

5.5 Production Hardening Checklist

Learning Resources

Quick Reference: Common Commands

Troubleshooting

Table of Contents

Graph View