Reed Solomon Encoding - Computerphile
Erasure coding(EC) is a data protection technique used in distributed storage systems to provide fault tolerance with less storage overhead than traditional replication.
Erasure coding,
- Split data into pieces
- Encodes it with parity fragments
- Allows recovery even if some pieces are lost.
Erasure coding solves; High storage cost and Large-scale data durability needs(object storage).
Replication is simple but expensive in terms of storage.(3x replication creates 200% overhead; Erasure coding create ~30-60% overhead).
Erasure coding significantly reduces storage cost, but it is a compute-heavy operation due to encoding, decoding, and reconstruction.
Modern systems balance this trade-off using design and operational strategies.
How Erasure Coding Works (Basic Idea)
Erasure coding is often described as(k, m):
- K -> data blocks
- m -> parity blocks
- Total blocks = k + m
Parity Block
A parity block is redundant data that does not store original data, but is computed from data blocks and is used to reconstruct missing data when failures occur.
Parity = information about data, not the data itself
To do this use reed solomon codes.
Erasure Coding in Action: How Big Tech Saves Petabytes of Data