Nimendra's Notes 🪴

Istio

13 min read

kubernetes/networking

Core Idea

Kubernetes runs and connects services but leaves auth, encryption, and access control to you; Istio adds a control plane and Envoy sidecars that intercept traffic to handle them.

  • The gaps in Kubernetes native networking: service authentication, pod-to-pod encryption, and who may call whom.
  • What Istio adds (control plane and data plane), and how traffic enters the sidecar through the istio-init container and iptables rules.
  • A problem-by-problem comparison of Kubernetes native versus Istio, starting with encryption and service identity.

1. What Problem Does Istio Solve?

Kubernetes gives you a way to run and connect services. But it leaves several hard problems to you:

  • How do services authenticate each other?
  • How do you encrypt pod-to-pod traffic?
  • How do you control which service can call which?
  • How do you observe what’s actually happening between services?
  • How do you retry failed calls, add timeouts, or do traffic splitting?

In Kubernetes native networking, you solve all of these yourself - in application code, in config files, scattered across every team. Istio moves all of it into the infrastructure layer, handled uniformly without touching app code.

2. Kubernetes Native Networking - How It Works

Before understanding Istio, understand what Kubernetes gives you by default.

2.1 Pod-to-Pod Communication

Every pod gets a unique IP. Pods can reach each other directly by IP. There is no encryption, no authentication, no access control — just raw IP routing.

graph LR
    A[Pod A 10.0.1.5] -->|plain TCP no encryption no auth| B[Pod B 10.0.1.6]
    A -->|plain TCP| C[Pod C 10.0.1.7]

2.2 Services and ClusterIP

A Kubernetes Service provides a stable virtual IP (ClusterIP) that load-balances across healthy pods. DNS resolves backend.default.svc.cluster.local to that ClusterIP.

graph TD
    Client[Client Pod] -->|DNS lookup backend:8080| DNS[CoreDNS]
    DNS -->|returns ClusterIP 10.96.45.20| Client
    Client -->|TCP to 10.96.45.20:8080| SVC[Service ClusterIP]
    SVC -->|kube-proxy iptables DNAT| P1[Pod 10.0.1.5]
    SVC -->|kube-proxy iptables DNAT| P2[Pod 10.0.1.6]
    SVC -->|kube-proxy iptables DNAT| P3[Pod 10.0.1.7]

kube-proxy maintains iptables rules on every node that rewrite the destination IP from the ClusterIP to a real pod IP (DNAT). The packet is then routed normally through the CNI network.

2.3 The Gaps in Kubernetes Native Networking

ProblemKubernetes Native Answer
Encrypt pod-to-pod traffic❌ None — plain TCP by default
Authenticate which service is calling❌ None at transport layer
Authorize service A to call service B❌ Only coarse NetworkPolicy (IP/port)
Retry failed requests❌ Must code in every app
Timeouts and circuit breaking❌ Must code in every app
Observe request rates, errors, latency❌ Must instrument every app
Traffic splitting (canary, blue/green)❌ Only crude replica-count hacks
Mutual TLS between services❌ Must manage certs in every app
Every one of these gaps requires either custom application code or bespoke infrastructure glued together per team. In a 5-service system that’s manageable. In a 50-service system it becomes a maintenance disaster.

3. What Istio Adds — The Control Plane and Data Plane

Control Plane and Data Plane

Istio has two logical layers:

graph TB
    subgraph Control Plane
        Istiod[Istiod - CA / cert issuer - Config distributor - Service discovery]
    end

    subgraph Data Plane - Node 1
        E1[Envoy sidecar + Pod A]
        E2[Envoy sidecar + Pod B]
    end

    subgraph Data Plane - Node 2
        E3[Envoy sidecar + Pod C]
    end

    Istiod -->|xDS config push mTLS certs| E1
    Istiod -->|xDS config push mTLS certs| E2
    Istiod -->|xDS config push mTLS certs| E3

    E1 <-->|mTLS| E2
    E1 <-->|mTLS| E3

Control Plane (Istiod) — runs once per cluster, manages config and certificates:

  • Issues short-lived SPIFFE X.509 SVIDs to every sidecar
  • Pushes routing rules, policies, and TLS config to sidecars via the xDS API
  • Watches Kubernetes Service/Endpoint resources to know where pods are

Data Plane (Envoy sidecars) — runs as a container in every pod:

  • Intercepts all inbound and outbound traffic
  • Enforces mTLS, retries, timeouts, circuit breaking
  • Emits telemetry (metrics, traces, access logs)

4. The Core Magic — How Traffic Gets Into the Sidecar

Sidecar Pattern & Sidecar Containers
Istio / Installing the Sidecar

This is the most important concept to understand. You call http://backend:8080 and your code doesn’t change — yet Envoy intercepts the call. How?

4.1 The istio-init Container

When a pod is scheduled with Istio sidecar injection enabled, a istio mutating webhook adds two containers before your pod starts:

Sidecars can be automatically added to applicable Kubernetes pods using a mutating webhook admission controller provided by Istio. (Admission Controller)

sequenceDiagram
    participant API as Kubernetes API
    participant Webhook as Istio Mutating Webhook
    participant Node

    API->>Webhook: Pod creation request
    Webhook->>API: Modified pod spec with: 1. istio-init container 2. istio-proxy (Envoy) container
    API->>Node: Schedule modified pod
    Node->>Node: Run istio-init (sets iptables rules)
    Node->>Node: Run istio-proxy (Envoy starts)
    Node->>Node: Run your app container

istio-init runs to completion, writes iptables rules into the pod’s network namespace, then exits. Those rules stay in effect for the pod’s lifetime.

4.2 The iptables Rules

The rules redirect all traffic through Envoy’s ports, while exempting Envoy’s own traffic (by UID) to prevent infinite loops:

flowchart TD
    OUT[Outbound packet from app container] --> OC{Is source UID 1337? Envoy itself}
    OC -->|Yes - skip| NET[Normal network routing]
    OC -->|No - intercept| E15001[Redirect to Envoy port 15001 outbound listener]
    E15001 --> ENVOUT[Envoy processes: - Load balance - Add mTLS - Apply retry policy]
    ENVOUT --> NET

    IN[Inbound packet arriving at pod] --> IC{Is source UID 1337?}
    IC -->|Yes - skip| APP[Delivered to app container]
    IC -->|No - intercept| E15006[Redirect to Envoy port 15006 inbound listener]
    E15006 --> ENVIN[Envoy processes: - Verify mTLS cert - Check AuthzPolicy - Strip TLS]
    ENVIN --> APP

4.3 Full End-to-End Call Flow

A complete picture of what happens when frontend calls http://backend:8080:

sequenceDiagram
    participant App_F as Frontend App
    participant Env_F as Frontend Envoy (port 15001)
    participant Env_B as Backend Envoy (port 15006)
    participant App_B as Backend App (port 8080)

    App_F->>App_F: curl http://backend:8080
    Note over App_F: DNS → ClusterIP 10.96.45.20

    App_F->>Env_F: iptables intercepts redirects to :15001
    Note over Env_F: Looks up "backend" service Selects pod 10.0.1.6 Fetches SVID from Istiod

    Env_F->>Env_B: mTLS handshake both present SVIDs
    Note over Env_F,Env_B: spiffe://cluster.local/ns/default/sa/frontend → spiffe://cluster.local/ns/default/sa/backend

    Env_B->>Env_B: Verify client SVID âś… Check AuthorizationPolicy âś… Terminate TLS

    Env_B->>App_B: Plain HTTP on localhost:8080
    App_B-->>Env_B: HTTP 200 response
    Env_B-->>Env_F: Response over mTLS
    Env_F-->>App_F: Plain HTTP response

    Note over App_F,App_B: App code never changed. No certs in app code. mTLS was fully transparent.

4.4 Why Envoy Doesn’t Intercept Its Own Traffic

flowchart LR
    subgraph Pod network namespace
        APP[App UID 1000] -->|port 8080 outbound| IPT[iptables OUTPUT chain]
        ENV[Envoy UID 1337] -->|outbound connections| IPT

        IPT -->|UID == 1337 ? RETURN - skip| SKIP[Normal routing]
        IPT -->|UID != 1337 ? REDIRECT| PORT[port 15001]
    end

Envoy always runs as UID 1337. The iptables rules have a --uid-owner 1337 -j RETURN rule that fires before the redirect rule, allowing Envoy’s own outbound connections to route normally.

5. Kubernetes Native vs Istio - Problem by Problem

5.1 Encryption of Pod-to-Pod Traffic

Kubernetes native:

graph LR
    A[Service A] -->|plain TCP unencrypted| B[Service B]
    X[Attacker on same node] -.->|can sniff traffic with tcpdump| A

Any process on the same node (compromised container, rogue pod) can sniff east-west traffic with no special privileges needed if they share a network namespace or bridge.

With Istio:

graph LR
    EA[Envoy Service A] -->|TLS 1.3 AES-256-GCM| EB[Envoy Service B]
    X[Attacker] -.->|sees encrypted bytes only| EA
    EA --> A[App A]
    EB --> B[App B]

All pod-to-pod traffic is encrypted at the Envoy layer. The app containers send and receive plain HTTP — encryption happens transparently.

5.2 Service Identity and Authentication

Kubernetes native:

graph TD
    SA[Service A claims to be frontend] -->|"I am frontend"| SB[Service B]
    SB -->|No way to verify the claim| OK[Accepted anyway]
    EVIL[Compromised pod also claims frontend] -->|"I am frontend"| SB
    SB -->|Also accepted| OK

A service’s identity is based on its IP, namespace, or service account — none of which are cryptographically verified at the transport layer.

With Istio:

graph TD
    SA[Service A SVID: spiffe://..../frontend] -->|presents X.509 SVID during TLS handshake| SB[Service B Envoy]
    SB -->|verify cert signature against Istiod CA| CHECK{Valid SVID?}
    CHECK -->|Yes - SPIFFE ID confirmed| ALLOW[Connection allowed]
    CHECK -->|No - self-signed or unknown CA| DENY[Handshake rejected]
    EVIL[Compromised pod no valid SVID] -->|no cert or wrong CA| SB
    SB --> DENY

Each workload gets a short-lived SVID from Istiod (valid 24h, rotated automatically). The SVID is cryptographically tied to the Kubernetes service account — it cannot be forged.

5.3 Access Control — Who Can Call Whom

Kubernetes native (NetworkPolicy):

graph TD
    NP[NetworkPolicy] -->|only controls| IPPort[IP address and port]
    NP -->|cannot control| HTTP[HTTP method path or header]
    NP -->|cannot control| ID[Workload identity or service account]

    A[Any pod in namespace] -->|port 8080 allowed| B[Backend port 8080]
    NOTE[If IP is allowed, ALL requests pass regardless of who the caller is]

NetworkPolicy is coarse — you allow or deny by IP range and port. There’s no concept of “only the checkout service account may call /api/payments”.

With Istio AuthorizationPolicy:

graph TD
    AP[AuthorizationPolicy] -->|controls by| ID[SPIFFE Identity service account]
    AP -->|controls by| HTTP[HTTP method path, header]
    AP -->|controls by| NS[Namespace]

    FRONT[frontend spiffe:.../sa/frontend] -->|GET /api allowed| BACK[backend]
    ADMIN[admin service] -->|GET + POST allowed| BACK
    OTHER[other-service spiffe:.../sa/other] -->|403 Forbidden| BACK
    FRONT -->|POST /admin blocked 403 Forbidden| BACK

AuthorizationPolicy works at the identity + HTTP layer. You can express rules like: “only frontend may call backend on GET /api/*” — and it’s enforced by Envoy before the request reaches your app.

5.4 Retries, Timeouts, and Circuit Breaking

Kubernetes native:

graph LR
    APP[App code] -->|must implement retry logic| R[Retry lib]
    APP -->|must implement timeout logic| T[Timeout lib]
    APP -->|must implement circuit breaker| CB[CB lib]
    R --> SVC[Service]
    NOTE[Every team re-implements this in every language]

Every team writes their own retry/timeout/circuit breaking logic in their own language. It’s inconsistent, frequently misconfigured, and duplicated everywhere.

With Istio VirtualService:

graph LR
    APP[App code] -->|plain HTTP call| ENVOY[Envoy sidecar]
    ENVOY -->|applies policy from VirtualService| SVC[Service]

    subgraph VirtualService config
        R2[retries: 3 retryOn: 5xx]
        T2[timeout: 5s]
        CB2[outlierDetection circuit breaker]
    end

A single YAML file applies retry, timeout, and circuit breaking policy for all callers of a service — in any language, without code changes.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: backend
spec:
  hosts:
    - backend
  http:
    - timeout: 5s
      retries:
        attempts: 3
        retryOn: 5xx,gateway-error
      route:
        - destination:
            host: backend

5.5 Observability — Knowing What’s Happening

Kubernetes native:

graph TD
    A[Service A] -->|must add Prometheus client| METR[Metrics]
    A -->|must add OpenTelemetry SDK| TRACE[Traces]
    A -->|must add structured logging| LOGS[Logs]
    NOTE[Each service manually instrumented per language. Gaps when teams skip it.]

Observability depends entirely on each team correctly instrumenting their service. A service with no instrumentation is a black box.

With Istio:

graph TD
    ENVOY[Envoy sidecar] -->|automatic| METR[Prometheus metrics for every request]
    ENVOY -->|automatic| TRACE[Distributed traces Jaeger / Zipkin]
    ENVOY -->|automatic| LOGS[Access logs every request]

    METR --> DASH[Grafana dashboard]
    TRACE --> TDASH[Jaeger UI]
    LOGS --> KDASH[Kiali graph]

    NOTE[Zero app changes. All services covered automatically.]

Envoy automatically emits metrics, traces, and access logs for every request. Kiali can render a live service graph with traffic rates, error rates, and latency — for every service — without any app instrumentation.

5.6 Traffic Splitting (Canary / Blue-Green Deployments)

Kubernetes native:

graph TD
    SVC[Service] -->|50% traffic| P1[v1 pod]
    SVC -->|50% traffic| P2[v1 pod]
    SVC -->|50% traffic| P3[v2 pod]
    NOTE["Only way to split traffic is replica count ratio. 2 v1 pods + 1 v2 pod = 66/33 split. Cannot do 99/1 without 99 pods."]

With Istio VirtualService + DestinationRule:

graph TD
    VS[VirtualService 99% → v1 1% → v2] --> DR1[DestinationRule subset: v1 label: version=v1]
    VS --> DR2[DestinationRule subset: v2 label: version=v2]
    DR1 --> PV1A[v1 pod]
    DR1 --> PV1B[v1 pod]
    DR2 --> PV2[v2 pod]
    NOTE[Any weight ratio. Independent of replica count. Shift traffic gradually.]
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: backend
spec:
  hosts:
    - backend
  http:
    - route:
        - destination:
            host: backend
            subset: v1
          weight: 99
        - destination:
            host: backend
            subset: v2
          weight: 1   # 1% canary, any ratio possible

6. Istio Component Reference

graph TB
    subgraph Control Plane
        ISTIOD[Istiod]
        ISTIOD --- PILOT[Pilot service discovery routing config]
        ISTIOD --- CITADEL[Citadel certificate CA SVID issuance]
        ISTIOD --- GALLEY[Galley config validation webhook]
    end

    subgraph Data Plane - every pod
        PROXY[Envoy proxy istio-proxy container]
        PROXY --- L1[Inbound listener port 15006]
        PROXY --- L2[Outbound listener port 15001]
        PROXY --- L3[Admin API port 15000]
        PROXY --- L4[Metrics port 15090]
    end

    subgraph Addons
        PROM[Prometheus]
        GRAF[Grafana]
        KIALI[Kiali]
        JAEGER[Jaeger]
    end

    ISTIOD -->|xDS push| PROXY
    PROXY -->|metrics| PROM
    PROM --> GRAF
    PROXY -->|traces| JAEGER
    PROM --> KIALI
    JAEGER --> KIALI

Envoy Port Reference

PortNamePurpose
15001OutboundAll outbound traffic from app is redirected here
15006InboundAll inbound traffic to pod is redirected here
15008HBONEHTTP/2 tunnel for ambient mesh (no sidecar mode)
15020Merged PrometheusCombined Envoy + app metrics
15021Health checkLiveness/readiness probe endpoint
15090Envoy PrometheusRaw Envoy metrics
15000AdminDebug interface — config dump, cluster status

7. Istio CRD Cheat Sheet

graph LR
    subgraph Traffic Management
        VS[VirtualService routing rules retries / timeouts traffic splitting]
        DR[DestinationRule load balancing circuit breaking TLS mode subsets]
        GW[Gateway ingress / egress port and TLS config]
        SE[ServiceEntry register external services]
    end

    subgraph Security
        PA[PeerAuthentication mTLS mode per namespace STRICT / PERMISSIVE]
        AP[AuthorizationPolicy who can call what by identity + HTTP]
        RA[RequestAuthentication JWT validation]
    end

    subgraph Observability
        TE[Telemetry metrics / tracing logging config]
    end
CRDWhat it controls
VirtualServiceHow traffic is routed to a service (retries, timeouts, splits, rewrites)
DestinationRuleHow traffic behaves after routing (LB algorithm, circuit breaking, TLS origination, subsets)
GatewayInbound/outbound traffic at the mesh boundary (what ports, what TLS)
ServiceEntryRegister external services (outside cluster) so Envoy knows about them
PeerAuthenticationSet mTLS mode (STRICT / PERMISSIVE / DISABLE) per namespace or workload
AuthorizationPolicyAllow/deny rules based on SPIFFE identity, namespace, HTTP method/path/header
RequestAuthenticationValidate JWT tokens on incoming requests
TelemetryConfigure metrics, tracing sampling rate, and access log format

8. Summary — Kubernetes Native vs Istio

The core shift: Kubernetes tells you where to run services and how to reach them. Istio takes over how services talk to each other — securing, observing, and controlling every connection in the cluster without touching application code.

CapabilityKubernetes NativeIstio Service Mesh
Encryption❌ None by default✅ Automatic mTLS everywhere
Authentication❌ IP-based only✅ SPIFFE X.509 SVIDs
Authorization⚠️ Port/IP-based NetworkPolicy✅ Identity + HTTP-layer authorization
Retries & Timeouts❌ Implemented in application code by each team✅ Configured via VirtualService YAML
Observability❌ Manual instrumentation required✅ Automatic metrics, traces, and logs
Traffic Splitting⚠️ Limited to replica-count-based distribution✅ Arbitrary weighted traffic splitting via VirtualService
mTLS❌ Manual certificate management in applications✅ Enabled with zero application changes

9. When Not to Use Istio

Istio solves real problems but comes with real costs:

ConcernDetail
ComplexityAdds ~10 CRD types, a control plane, and a sidecar to every pod
Resource overheadEach Envoy sidecar uses ~50–100MB RAM and adds latency (~1–2ms per hop)
Debugging difficultyTraffic flows through an extra layer; misbehaving policy is hard to trace
Overkill for small systemsA 3-service app doesn’t need a service mesh
Learning curvexDS config, Envoy internals, and Istio CRDs take time to master

Consider Istio when you have 10+ services, multiple teams, security/compliance requirements, or need consistent observability without per-team instrumentation effort.

Consider skipping Istio when your system is small, your team is new to Kubernetes, or the operational overhead outweighs the benefits.

How to Set Up Istio in Ambient Mode (Detailed Guide)

Backlinks

Graph View