Nimendra's Notes 🪴

04 - Fault-Tolerance

4 min read

systemdesign systemdesign/concepts distributedsystem/microservice/patterns

03 - Availability
DS12.1 - Fault Tolerance
Release It Design and Deploy Production-Ready Software, Chapter 5: Stability Pattern - Michael T. Nygard

Fault tolerance is the ability of a system to continue operating if some of its components fail and the prevention of permanent harm (such as data loss) should downtime occur.

Graceful Degradation - our system can maintain some functionality when parts of it fail, rather than a complete catastrophic failure.

Self healing mechanisms - Automatically provision replacement components and attach them to our system, so our system can recover without manual intervention and without any noticeable effect on end users.
05. Docker Containers > Self-healing containers with restart policies

Replication and Redundancy

Replication (especially with 3 copies) balances performance and resilience. Strategically place replicas to match your system’s fault tolerance and latency needs.
Replicas can place same data center or Different data centers.

Forward Error Correction and Error Correction Code

Forward error correction (FEC) is a technique to prevent errors in data transmission over noise or unreliable communication channels (unreliable or noisy networks like satellite, wireless or long distance cables) by encoding the messages in a redundant way, such as by using an error correction code(ECC).

How It Works

  1. Sender encodes the original data using an Error Correction Code (ECC).
  2. This encoded message includes extra redundant bits (not part of the original message).
  3. Receiver gets the encoded data.
  4. If there’s a small error, the receiver can detect and correct it automatically — without needing the sender to retransmit.

Example

Let’s say we want to send: 1011

  • The sender applies ECC and sends: 1011 + parity bits = 1011010
  • Suppose 1 bit gets flipped due to noise → 1011110
  • The receiver uses the ECC logic to detect and fix the error, recovering 1011

Circuit Breaker

👉 Circuit Breaker

Exponential Backoff and Retry

It’s similar to a circuit breaker.
When a client receives and error response, it will wait before reattempting the request and exponentially increase the wait duration between retries.

Instead of retrying immediately after a failure, wait… and wait longer… and loooonger…

Jitter

If 1000 clients use exponential backoff without jitter… they’ll all retry at the same time, This causes retry storms — a spike of requests right when the server is recovering.

Jitter adds randomness to the delay to spread out retries, avoiding retry storms.

// Exponential backoff with jitter (Node.js example)
function getDelay(attempt, base = 1000, max = 30000) {
  const expBackoff = Math.min(max, base * 2 ** attempt)
  const jitter = Math.random() * expBackoff
  return jitter
}

In practice, often use exponential backoff and jitter together.

Caching Responses

We can cache the external service’s responses whenever we make successful requests and use these responses when the external service is unavailable.

Dead Letter Queue

👉 Dead Letter Queues

  • The simplest option is that if it is acceptable to miss requests, just drop failed requests.
  • Implement the dead letter queue locally with a try-catch block. Requests will be lost if the host fails.
  • A more complex and reliable option is to use an event-streaming platform like Kafka.

Bulkhead Pattern

The Bulkhead pattern is like a ship’s bulkhead walls, it isolates failures so that one part fails, others keep working.

  • Divide the system into isolated compartments (pools).
  • Each compartment (e.g. a thread pool or connection pool) handles only specific tasks.
  • If one pool is full/failing, others keep working.

Fallback Pattern

The fallback pattern consists of detecting a problem and then executing an alternative code path, such as cached responses or alternative services that are similar to the service the client is trying to get information from.

  • If a remote service call fails (e.g. timeout, error), return a default or cached response.
  • Keeps the system responsive even when parts are broken.

Backlinks

Graph View