Envoy

kubernetes kubernetes/kcna kubernetes/networking

Core Idea

Envoy is the high-performance open-source proxy that powers service meshes and ingress controllers in Kubernetes, usually as a sidecar next to the app.

• Key features: runs as a sidecar proxy in the pod, with modern traffic management capabilities.
• Use cases include the Istio service mesh, and the note explains why Envoy is picked for the job.
• A dedicated section contrasts Envoy with kube-proxy: responsibilities, how each works, key differences, and when to use which.

Envoy is a high-performance, open-source edge and service proxy designed for cloud-native architectures. In Kubernetes, it is widely used as a foundational component in service meshes and Ingress controllers due to its advanced networking capabilities and support for modern traffic management features.

Key Features of Envoy in Kubernetes

1. Service Proxy:
   • Envoy operates as a sidecar proxy, deployed alongside application containers in Pods.
   • It intercepts and manages all inbound and outbound traffic, providing features like load balancing, observability, security (e.g., mTLS), and traffic shaping.
2. Core Component in Service Meshes:
   • Popular service meshes like Istio, AWS App Mesh, and Consul Connect use Envoy as their data plane.
   • It provides capabilities such as:
     • Traffic Management: Routing, retries, circuit breaking, and rate limiting.
     • Observability: Detailed metrics, logs, and distributed tracing.
     • Security: Built-in support for mTLS, authentication, and authorization.
3. Ingress Controller:
   • Envoy can be used as an Ingress controller in Kubernetes, managing HTTP/HTTPS traffic entering the cluster.
   • It handles advanced routing scenarios, including URL-based routing, header matching, and wildcard domains.
4. Extensibility and API-driven Design:
   • Envoy is highly configurable using dynamic APIs, enabling real-time updates without restarts.
   • Supports custom filters for advanced use cases like authentication and custom protocol handling.

Use Cases in Kubernetes

1. Service Mesh:
   • Envoy functions as the data plane in service mesh architectures, managing service-to-service communication across Pods.
2. Ingress and Egress Gateways:
   • Acts as a gateway to route traffic entering (Ingress) or leaving (Egress) a Kubernetes cluster.
3. Traffic Shaping and Canary Deployments:
   • Enables advanced traffic routing scenarios, such as A/B testing, blue-green deployments, and gradual rollouts.
4. Observability:
   • Provides detailed telemetry, such as metrics (via Prometheus), traces (via Jaeger/Zipkin), and logs (structured and customizable).

Why Use Envoy in Kubernetes?

• Performance: Designed for high throughput and low latency.
• Modern Protocol Support: Handles gRPC, HTTP/2, WebSocket, and other modern protocols.
• Resilience: Features like circuit breaking and retry logic enhance application reliability.
• Security: Native support for TLS/mTLS ensures secure communication between services.
• Community and Ecosystem: Backed by the Cloud Native Computing Foundation ( cncf ), with robust support and integrations.
  gRPC

Example: Envoy in Istio Service Mesh

In an Istio-based service mesh:

• Envoy Proxy as Sidecar: Deployed alongside application containers to intercept and manage traffic.
• Control Plane (Istio): Configures Envoy proxies dynamically using Istio’s configuration resources.
• Capabilities: Handles encryption (mTLS), traffic policies, and telemetry collection.

Kube-Proxy and Envoy

kube-proxy and Envoy are both used in Kubernetes networking, but they serve very different purposes and operate at different levels of the network stack. Here’s a detailed comparison:

1. Overview

**kube-proxy:

• Function: A Kubernetes networking component that manages routing and load balancing for service traffic within the cluster.
• Scope: Operates at the IP layer to route traffic between Pods.
• Purpose: Ensures Pods can communicate with each other and external services using Kubernetes Services.

Envoy:

• Function: A high-performance application-layer proxy used for advanced traffic management, service-to-service communication, and observability.
• Scope: Operates at the L7 (application) layer to handle HTTP, gRPC, WebSocket, and other protocols.
• Purpose: Provides advanced traffic shaping, security (e.g., mTLS), and observability, typically in service mesh or Ingress/Egress scenarios.

2. Responsibilities

Feature	kube-proxy	Envoy
Traffic Routing	Routes traffic based on IP and port.	Routes traffic based on HTTP headers, paths, and protocols.
Load Balancing	Simple round-robin or IP-based.	Advanced load balancing with retries, failovers, and circuit breaking.
Protocol Support	TCP/UDP only.	HTTP/1.x, HTTP/2, gRPC, WebSocket, and more.
Security	None (relies on other tools).	Built-in TLS/mTLS for secure communication.
Observability	Minimal (metrics via Prometheus).	Detailed observability: logs, metrics, distributed tracing.
Layer of Operation	Network Layer (L4, IP and port).	Application Layer (L7, protocol-specific).
Use Case	Basic Pod-to-Pod or Pod-to-Service traffic.	Advanced service-to-service communication and Ingress/Egress traffic management.

3. How They Work

kube-proxy:

• Operates at the Node Level: Runs on every Kubernetes node.
• Uses IPTables/IPVS or User-Space:
  • Creates routing rules to direct traffic to the appropriate Pod based on a Service’s label selectors.
  • Does not inspect or modify the traffic; it simply routes it.

Envoy:

• Operates as a Sidecar or Gateway: Typically deployed as a sidecar proxy in service meshes or as an Ingress/Egress controller.
• Application-Aware Proxy:
  • Inspects traffic at the application layer.
  • Applies policies like retries, timeouts, encryption, and more.
• Dynamic Configuration: Controlled by APIs or service mesh control planes like Istio or Linkerd.

4. Use Cases

kube-proxy:

• Default Kubernetes networking component.
• Used for basic Service discovery and connectivity within the cluster.

Envoy:

• Advanced use cases in service meshes (e.g., Istio, Linkerd).
• Complex traffic management:
  • A/B testing, canary deployments, rate limiting.
  • Enforcing mTLS between services.
  • Observability with distributed tracing and metrics.

5. Key Differences

Aspect	kube-proxy	Envoy
Level of Operation	Network-level (L4).	Application-level (L7).
Complexity	Simple, lightweight.	Advanced, adds complexity but more features.
Customization	Minimal.	Highly configurable with rich APIs.
Resource Usage	Lightweight, built into Kubernetes.	Heavier, requires additional resources for sidecars or gateways.
Traffic Inspection	None.	Full traffic inspection and modification.

6. When to Use

Use kube-proxy if:

• You need basic networking for your Kubernetes Services.
• Your application does not require advanced traffic control, observability, or security features.

Use Envoy if:

• You’re implementing a service mesh for advanced service-to-service communication.
• You need mTLS, advanced routing, retries, or traffic shaping.
• You require detailed observability (e.g., distributed tracing, request metrics).