02 - Scalability

systemdesign/concepts systemdesign

Scalability | System Design
08 - ESB & ASG
Acing the System Design Interview, Chapter 3.1

Scalability is the measure of how well a system responds to changes by adding or removing resources to meet demands.

Horizontal and Vertical Scaling

Vertical Scaling (aka scaling up) achieve scalability by adding more power to an existing machine(add more CPU,RAM or Storage).
vertical scaling is simple to implement and easier to manage.
but it has technological limits. also it require stop out host before upgrading therefore it has downtime. also it’s not cost effective.

Horizontal scaling (aka Scaling Out) scale system by Adding multiple logical units of resources (e.g., clustering multiple servers) and making them work as a single unit.It improves the performance of the server by adding more instances to the existing pool of servers, allowing the load to be distributed more evenly.

• Advantages: increased redundancy, better fault tolerance, flexible and easier to upgrade and downgrade.
• Disadvantage: increased complexity, data inconsistency

Load balancer

Horizontally scaled service uses a load balancer, which may be one of the following:

1. Hardware Load Balancer:
   • Physical Device that distribute traffic across multiple hosts
   • too expensive
   • Barracuda Load Balancer, F5 BIG-IP
2. Software Load Balancer:
   • NGINX , HAProxy
3. Shared Load Balancer service:
   • aka LBaaS (Load Balancing as a Service)
   • AWS ESB

Load Balancer and Reverse Proxy

Load Balancing is for scalability, while reverse proxy is a technique to manage client-server communication.
👉 Proxy, Reverse Proxy and Load Balancer

Every load balancer has either,

1. Static IP
2. Static DNS Name

LB hides the # servers that are behind it allowing to add as many server as possible.(Horizontal Scaling)

Load Balancing Algorithms

Static Algorithms

1. Round Robin - Distribute the load iteratively
2. Sticky round robin - This is an improvement of the round-robin algorithm. If Alice’s first request goes to service A, the following requests go to service A as well.
3. Weighted Round Robin - Distributed the load iteratively but as the weights(More request to powerful server). Admin can specify the weight of the server.
4. Hash Based - Hash of some request attribute (ip, user_id) determines which server to pick.

Dynamic Algorithms

1. Least Connections - Pick the server having the least connections from the load balancer.
2. Least response time - A new request is sent to the service instance with the fastest response time.

Layer 7 vs Layer 4 Load Balancers

1204-OSI & TCP-IP
[!video]- Load balancing in Layer 4 vs Layer 7 with HAPROXY Examples

A Layer 4 load balancer operates at the transport layer(TCP). It makes routing decisions based on address information extracted from the first few packets in the TCP stream and does not inspect the contents of the other packets. it can only forward packets.

A Layer 7 load balancer operates at the application layer(http).
It has following capabilities:

• Load balancing/routing decisions: Based on a packet’s contents.
• Authentication: It can return if a specified authentication header is absent
• TLS termination: Security requirements for traffic within a data center may be lower than traffic over the internet, so performing TLS termination.

What is the TLS termination ?

TLS

Reverse proxy handles the TLS/SSL decryption instead of your application servers.
(TLS = Transport Layer Security (successor to SSL))

1. Client sends the HTTPS request.
2. Reverse Proxy decrypt it.(TLS Termination)
3. Sends HTTP request to backend service.
4. App handles the request without worrying about TLS.

Benefits

• Simplify backend processes.
• Better performance
• Central Control

Example

server {
    listen 443 ssl;
    server_name example.com;
 
    ssl_certificate     /etc/nginx/cert.pem;
    ssl_certificate_key /etc/nginx/key.pem;
 
    location / {
        proxy_pass http://localhost:3000;
    }
}

Sticky Sessions

Sticky session ensures that all requests from a specific client are sent to the same backend server during a session(Using load balancer). It’s useful for stateful service where maintaining session data is important.
Load balancer remembers which server a client first connected to.
It use maintain user context like, login state, profile preferences, shopping cart contents.

1. Duration bases sticky session
   • Load balancer issues its own cookie.
   • Cookie contains a timeout duration.
   • All requests with that cookie go to the same backend.
   • When cookie expires → client may be routed to a different server.
2. Application-controlled Sticky Session
   • The application sets its own cookie.
   • Load balancer also sets a cookie that respects the app’s session lifetime.
   • Offers more control, but requires:
     • Deeper integration between app and load balancer
     • Handling of session expiry logic in both

👉 Session Store

Stateful vs Stateless Services

Stateful & Stateless Applications

Stateless Service

A stateless service does not store any session or client data between requests.
Each request is treated as independent and self-contained.

• Think: “I don’t remember anything about the previous request.”

Stateful Service

A stateful service remembers previous interactions or stores session data.
It maintains context about the client between requests.

• Think: “I remember who you are and what you were doing.”

Session Replication

In session replication, writes to a host are copied to several other hosts in the cluster that are assigned to the same session, so reads can be routed to any host with that session. This Improve 03 - Availability.

Advantages of Load Balancer