distributedsystem/books/think-distributed-systems
Failure = An unwanted but possible state transition that moves a system from a good (legal) state to a bad (illegal) state.
Failure Tolerance = The ability of a system to continue behaving in a well-defined way even when failures occur.
System States
- Legal State (Good State)
- System is working correctly.
- Example: Database is running and data is consistent.
- Illegal State (Bad but Recoverable State)
- Something went wrong.
- Recovery is still possible.
- Example: A server crashes, but another replica can take over.
- Intolerable State (Beyond Recovery)
- The damage is so severe that recovery is impossible or not considered.
- Example: A meteor destroys the entire data center.
Failure Recovery = Sequence of transitions that moves the system from an illegal state back to a legal state.
Fault-tolerant distributed computing focuses on failures that can realistically be handled.
If recovery is impossible, there is no point designing algorithms for it.
Failure Handling
- Failure Detection - Determining that a failure has occured.
- Failure Mitigation - Recovering from or handling the failure.
Failure Classification
Failures are classified along two dimensions:
A. Spatial Dimension (Where?)
Application-Level Failure
- Can only be understood and handled by the application.
- Example:
InsufficientFunds
Platform-Level Failure
- Can be handled by infrastructure/platform.
- Example:
CouldNotConnect
End-to-End Principle:
Handle failures at the lowest layer capable of detecting and resolving them correctly.
B. Temporal Dimension (When?)
1. Transient Failure
- Temporary.
- Resolves quickly by itself.
- Probability of another failure is unchanged.
Example: Router restart.
2. Intermittent Failure
- Comes and goes repeatedly.
- Higher chance of recurring shortly.
Example: Outdated routing table.
3. Permanent Failure
- Persists until repaired.
- Requires manual intervention.
Example: Expired certificate.
Failure Detection
A failure detector is a mechanism that identifies failures.
- Application-Level Detection
- Example: Error codes, Exception Messages, Validation Failures
- Platform-Level Detection
- Example: Connection failures, Service unavailability
Heartbeat-Based Detection
Pull Model
Observer periodically sends requests and expects responses.
Push Model
Observed component periodically sends heartbeats.
Timeout
If no message arrives in time:
- Failure is suspected.