Kubernetes Networking & Service Mesh Architecture Guide

kubernetes ebpf kubernetes/networking

Istio vs Cilium

Tldr

Use Cilium when you need raw performance and L3/L4 micro-segmentation
Use Istio when you need rich L7 control and observability.
In production, running both together is increasingly common and they complement each other well.

Comparison

	Istio	Cilium
How it works	Sidecar proxy (Envoy) in every pod	eBPF programs loaded into the Linux kernel
Overhead	Higher (~50–100 MB RAM per pod)	Near-zero (runs in kernel)
mTLS	Built-in, automatic	Via Cilium Mesh (less mature)
L7 policies	Excellent (HTTP, gRPC, JWT)	Basic (improving rapidly)
Network policies	Via mesh config	Native, identity-aware
Observability	Best-in-class (Kiali, Jaeger)	Good (Hubble UI)
Kernel requirement	Any	Linux ≥ 5.4

Kubernetes Networking & Service Mesh Architecture Guide

Tldr

• kube-proxy (Legacy L3/L4): Translates virtual Service IPs to Pod IPs using slow, linear kernel iptables rules.
• Cilium (Modern L3/L4): Bypasses iptables and replaces kube-proxy entirely using eBPF in the Linux kernel for instant, constant-speed routing.
• Istio (L7 Control Plane): The “brain” of the service mesh. Manages high-level application traffic rules (routing, retries, security) and identities.
• Envoy (L4/L7 Data Plane): The high-performance “muscle” proxy (used by Istio and Cilium) that actually intercepts, inspects, and forwards HTTP/gRPC packets.
• Ambient Mesh (Sidecarless): The next-gen Istio architecture. It removes resource-heavy sidecars from your pods, splitting responsibilities into node-level L4 security (ztunnel) and on-demand L7 application routing (waypoint).

A comprehensive architectural reference detailing the roles, boundaries, and integration points of kube-proxy, Cilium (eBPF), Istio (Sidecar & Ambient), and Envoy.

1. Core Component Breakdown

┌─────────────────────────────────────────────────────────────────────────┐
│                          APPLICATIONS (L7)                              │
│  [ Istio Control Plane ] ──► [ Envoy Proxies / Waypoints ]              │
├─────────────────────────────────────────────────────────────────────────┤
│                          KERNEL & NETWORK (L3/L4)                       │
│  [ Cilium (eBPF) ]  ◄── (Replaces) ──►  [ kube-proxy (iptables) ]       │
└─────────────────────────────────────────────────────────────────────────┘

• kube-proxy: The traditional Kubernetes network controller. It configures the Linux kernel’s network layer (iptables or IPVS) to route traffic destined for a virtual Service IP to a backend Pod IP. It acts at Layer 3/4 but scales poorly in large clusters. (CNI and Kube proxy)
• Cilium: A modern kube-proxy replacement. It bypasses rigid iptables rules entirely by running sandboxed eBPF programs directly inside the Linux kernel. It provides instant, constant-speed packet routing ($O(1)$ lookup complexity) at the socket level.
• Istio: A Service Mesh that operates at the application layer (Layer 7). It coordinates Microservice communication, managing high-level traffic behavior (canary routing, retries, circuit breaking) and securing communication via cryptographic Mutual TLS (mTLS).
• Envoy: The high-performance, programmable data plane proxy (written in C++) that sits directly in the traffic stream. While Istio acts as the “brain” (Control Plane) making routing policies, Envoy is the “muscle” (Data Plane) that executes them by parsing HTTP/gRPC data.

2. The Next Evolution: Istio Ambient Mesh (Sidecarless)

Traditionally, Istio deployed an Envoy Sidecar proxy inside every single Pod. While powerful, this added high memory overhead and required restarting application pods for proxy upgrades.

Istio Ambient Mesh splits these responsibilities into two distinct, sidecarless layers:

                  ┌──────────────────────────────┐
                  │      WAYPOINT PROXY (L7)     │ <--- On-demand, per namespace
                  └──────────────▲───────────────┘
                                 │ (HBONE)
┌────────────────────────────────┼────────────────────────────────┐
│ NODE 1                         │                     NODE 2     │
│  ┌──────────┐           ┌──────▼──────┐           ┌──────────┐  │
│  │ App Pod  │ ──(TCP)─► │   ztunnel   │ ──(mTLS)─►│ App Pod  │  │
│  └──────────┘           └─────────────┘           └──────────┘  │
│                         (Shared L4 LHB)                         │
└─────────────────────────────────────────────────────────────────┘

A. The Secure Overlay Layer (ztunnel)

• What it is: A shared, high-performance DaemonSet running on each node (written in Rust).
• Responsibility (L4): It handles Mutual TLS (mTLS), transport-layer authorization, and basic TCP metrics.
• Protocol: It wraps TCP traffic in HBONE (HTTP-Based Overlay Network Engine), standardizing secure communication between nodes.
• Impact: Zero application pod restarts, incredibly low CPU/memory overhead, and instant zero-trust security.

B. The L7 Processing Layer (Waypoint Proxies)

• What it is: Shared Envoy-based proxies deployed outside application pods (typically per namespace or service account).
• Responsibility (L7): It handles advanced operations like HTTP routing, traffic splitting, retries, and access control.
• Impact: You only deploy and pay the resource cost for L7 proxies where you actually need complex L7 application logic.

3. How Traffic Flows: Cilium + Istio Ambient Mesh

When you combine Cilium (the fast L3/L4 highway) with Istio Ambient (the sidecarless L4/L7 controller), packet flow is highly optimized:

1. Application Initiation: The application container initiates a connection.
2. Kernel Interception (Cilium): Cilium’s eBPF programs intercept the connection at the socket level. Rather than performing slow DNS/iptables translation, eBPF instantly routes the packets to the node’s local ztunnel.
3. L4 Security (ztunnel): The node’s ztunnel encapsulates the traffic in HBONE (mTLS) and pushes it to the network card.
4. L7 Interception (Waypoint - Optional): If an Istio L7 policy (like path routing or retries) is defined for the target service, Cilium routes the packet through the Waypoint proxy first.
5. Fast Path Delivery (Cilium): Cilium moves the encrypted packets across the host network interfaces straight to the target node’s ztunnel and then to the destination pod.

4. FAQ: Boundaries, Overlaps, & Decisions

Q: Where exactly are the boundaries between Cilium and Istio?

• Cilium’s Boundary: Layer 3 and Layer 4 (Network & Transport). It manages packets, IPs, routing tables, and protocols. It answers: “How do we get bytes from Host A to Host B with absolute minimum latency?”
• Istio’s Boundary: Layer 7 (Application). It manages messages, metadata, headers, cookies, and identity. It answers: “Is this specific HTTP user authorized to access this microservice path, and how should we shape their traffic?”

Q: Don’t both Cilium and Istio use Envoy? Is that an overlap?

Yes.

• Istio relies on Envoy as its primary data plane (running either as a sidecar, inside the Node ztunnel architecture, or as a standalone Waypoint proxy).
• Cilium is primarily an eBPF (kernel) engine. However, because eBPF cannot easily parse complex Layer 7 application protocols (such as parsing a specific HTTP header), Cilium embeds an internal Envoy instance. Cilium only redirects packets to its internal Envoy if you write an explicit L7 Cilium Network Policy.

Q: Does Cilium Service Mesh replace Istio?

It depends on your scale and requirements:

• Use Cilium Service Mesh alone if you want a lightweight solution. By leveraging its embedded Envoy, Cilium can handle basic L7 tasks (like ingress routing, simple path-based redirection, and L7 network policies) without the resource overhead of a dedicated service mesh.
• Use Istio on top of Cilium if you require advanced enterprise service mesh capabilities. This includes complex traffic splitting (canary releases), distributed tracing, rate limiting, circuit breaking, or complex multi-cluster security federations.

Q: If I use Cilium + Istio Ambient Mesh, do I still need an Ingress Controller?

No, you can consolidate. Both platforms leverage Envoy at their boundaries:

• **Cilium** provides its own [[Gateway API]] controller using its internal Envoy instances.
• Istio provides its own Gateway API integration.
  Either can safely act as your ingress, stripping away the need for legacy controllers like standard NGINX Ingress, reducing your overall cluster resource footprint.