distributedsystem softwareengineering systemdesign
Tldr
Durable execution is to distributed workflows what transactions are to databases: an abstraction that hides much of the complexity of partial failure behind a simple idea:
Execution survives process failure and resumes from recorded progress instead of starting from scratch.
Durable execution is a programming model where a workflow’s execution state and progress are persisted to durable storage, independently of the process executing it.
Crash → resume, not restart.
Durable Execution is a programming model where a workflow’s execution progress is persisted to durable storage, independent of the process running it, so a crash means resume, not restart.
Core Problem
- Single Machine, Single function -> Crash , just rerun.
- No side effects, no harm (comparing Distributed systems)
- Distributed workflow (multiple services) → crash mid-way is dangerous, because some steps already had real-world effects (charged a card, sent an email) that can’t be undone by re-running code.
failure in distributed systems is partial, not binary. Some steps succeed, some fail — system must reconcile an inconsistent state.
async function signup(user) {
const charge = await Payment.create(user.card, 10); // âś… succeeds
// đź’Ą crash here
const account = await Account.create(user);
await Access.grant(account, charge);
}Rerun from top → double charge. Skip ahead → maybe free access. Both bad.
Questions every distributed workflow must answer
| Question | Why it’s hard |
|---|---|
| Which steps already ran? | No shared memory if process dies |
| Is this step safe to retry? | Idempotency must be enforced manually otherwise |
| How do we undo a step that already happened? | Compensation / saga logic(The Saga Pattern) |
| Where do timers/long waits live if the process restarts? | Can’t rely on in-memory timers |
| Without durable execution, this glue code (retries, idempotency keys, saga compensation, timeouts) often outweighs the actual business logic. |
Solution
Persist workflow progress to durable storage, not to the memory of one running process. (Separate progress from process)
Mechanics (how replay actually works):
- Record history — every step’s input/output is written to durable storage before moving on.
- Replay on recovery — a new process re-runs the code from the start, but skips already-completed steps by reading their recorded results instead of re-executing them.
- Determinism required — workflow logic must produce the same path given the same inputs, or replay can’t rebuild state correctly. Non-deterministic values (random, time, external calls) must be captured as recorded facts.
- Time becomes durable — timers/delays survive crashes natively; no external cron hack needed.
Complexity doesn’t disappear — it moves into the runtime, built once and tested thoroughly instead of reinvented per service.
| Pattern | Manual approach | Durable execution approach |
|---|---|---|
| Retries | Custom retry loops everywhere | Built into runtime, tied to history |
| Sagas / compensation | Hand-written rollback logic | Orchestrated automatically |
| Long timers / cron | External schedulers | Native durable timers |
| State machines | Custom “where am I” persistence | Implicit — history is the state |
| Idempotency | Idempotency keys threaded through APIs | Steps recorded once, can’t double-fire on replay |
 |
Who uses it / what tools implement it
- Used at: Stripe, Netflix, HashiCorp, Datadog (per Temporal’s writeup)
- Engines:
- Temporal - most established, SDK + worker model, full event history
- Restate - lighter, HTTP-native ergonomics
- AWS Step Functions / Azure Durable Functions - state-machine style
- Inngest / Conductor (Orkes) - event-triggered / orchestration platforms
The question today isn’t whether to use durable execution for complex multi-step workflows, but which engine fits your team’s operational maturity and language stack.