ai sitereliabilityengineering casestudies
Understanding “Ironies of Automation”
Lisanne Bainbirdge explains a surprising problem: the more we automate a system, the more difficult and improtant the remaining human resposibilities can become.
Bainbridge argues that automation does not necessarily eliminate human error or make systems safer. Sometimes, it creates entirely new problems for the people who operate them.
Core Idea
Imagine an aircraft controlled by and autopilot,
During a normal flight the autopilot handles most of the work. The pilot mainly observe the system.
But suddenly, the autopilot fails.
The pilot must now take control immediately, understand what has gone wrong and make critical decisions.
The problem is that the pilot may not have been actively controlling the aircraft for hours. Their awareness of the current situation and readiness to control it may have declined.
This is the central irony
automation removes the routine work that helps humans maintain the skills and knowledge they need during emergencies.
Major Ironies of automation
- Automation removes easy tasks but leaves difficult tasks for humans.
- Machines handle predictable, repetitive activities. Humans are left to handle unusual events, system failures, and emergencies—which are often the hardest situations.
- Automation makes human skills deteriorate.
- When people stop performing a task regularly, they lose some of their practical ability. Yet they are expected to perform exceptionally well when automation fails.
- Humans are expected to monitor highly reliable machines.
- Monitoring a system where almost nothing goes wrong is boring and difficult. Human attention naturally decreases, making rare failures harder to detect.
- Humans are expected to check machines that outperform them.
- If a machine makes decisions faster and more accurately than a person, expecting that person to verify every decision in real time may be impossible.
- More successful automation can require more human training.
- The less often a system fails, the fewer opportunities operators have to practise handling failures. Therefore, highly reliable automation may demand substantial simulator training.
What solutions does Bainbridge suggest?
Bainbridge does not argue that we should abandon automation. Instead, she suggests designing systems in which humans and computers work together effectively.
Her suggestions include preserving operators’ practical skills through regular manual control or simulator training, using alarms to help detect rare failures, ensuring important information remains available during emergencies, and giving operators computer support for diagnosis and decision-making.
Do simple tasks really matter?
Think about how you learned Kubernetes.
You probably started with something simple: building a Docker image, creating a Pod, deploying an application, and checking whether it was running.
These tasks seem trivial once you become experienced. But when you were learning, they were essential.
You learned what happens when a container crashes, why a Pod stays in Pending, how networking works, and how different components interact.
Over time, these small experiences helped you build a mental model of the system. That mental model is what allows you to reason about complex problems when things go wrong.
What matters is the learning process: doing something, observing what happens, making mistakes, understanding why something failed, and trying again.
That’s how intuition develops. You gradually move from knowing what commands to run to understanding why the system behaves the way it does.
Now consider what happens when AI automates most of these simple, repetitive tasks.
You describe what you want, and AI generates the manifests, deploys the application, identifies errors, and suggests fixes.
That’s incredibly useful. It saves time and reduces unnecessary work.
But there’s a trade-off.
If automation removes your involvement in the process, it may also remove the feedback loops through which you develop an understanding of the system.
You get the outcome without necessarily experiencing the reasoning, experimentation, and failures that would have helped you understand how that outcome was achieved.
This doesn’t mean automation is bad or that everyone should manually perform every repetitive task forever.
Experienced engineers can use automation to eliminate routine work and focus on deeper problems. Beginners can also use AI to accelerate learning, especially when it explains its reasoning and encourages experimentation.
The real issue is not automation itself. It’s whether automation replaces only the mechanical work or also replaces the thinking and learning that happen alongside it.
And this connects directly to Bainbridge’s Ironies of Automation.
Automation can take away the easy parts of a task while leaving humans responsible for the hardest parts. Yet those easy tasks may have been the very experiences that prepared humans to handle difficult situations.
Perhaps the question we should ask is not just: How much work can we automate?
But also: What opportunities to learn are we removing when we automate that work?
flowchart TD
A["Simple Tasks"] --> B["Practice & Experiment"]
B --> C["Mistakes & Feedback"]
C --> D["Build Intuition"]
D --> E["Solve Complex Problems"]
A --> F["AI Automation"]
F --> G{"Human Involvement?"}
G -->|No| H["Less Learning & Intuition"]
G -->|Yes| B
H --> I["Harder to Solve Complex Problems"]
AI SRE
Tldr
AI may not immediately replace Site Reliability Engineers (SREs), but it could gradually weaken the very skills that make them valuable during complex production incidents.
Bainbridge’s original paper examines what happens when industrial machines automate routine operations but still rely on humans during emergencies.
The current SRE workflow applies the same argument
| Bainbridge (1982) | SRE in 2026 |
|---|---|
| Machines control industrial processes. | AI automates incident response. |
| Operators monitor machines. | SREs monitor AI-driven operations. |
| Manual control skills weaken. | Debugging and troubleshooting skills may weaken. |
| Humans intervene during rare failures. | SREs handle novel, complex outages. |
| Operators lack practice when emergencies occur. | SREs may lack the experience needed for unfamiliar incidents. |
The key connection is that automation changes the nature of human work without necessarily reducing the expertise required to do it well.
Deskilling and Never-Skilling
Deskilling happens when an experienced engineer gradually loses skills through lack of practice. For example, a senior SRE who previously diagnosed Kubernetes networking failures independently now relies heavily on AI-generated root-cause analyses.
Never-skilling happens when a junior engineer never gets enough opportunities to develop those skills in the first place. If AI resolves most everyday incidents, junior SREs may never build the debugging intuition that previous generations developed through experience.
What should SRE teams do?
The article proposes two important approaches, which are closely related to Bainbridge’s recommendations.
- Deliberate practice: Let engineers solve selected low-risk incidents manually, even when AI could resolve them, and use simulations to rehearse more dangerous failures.
- Human-in-the-loop rather than human-on-the-side: Keep engineers actively involved in forming hypotheses, examining evidence, and making decisions instead of merely approving AI recommendations.
- Preserve learning opportunities: Ensure junior engineers participate in real diagnosis and troubleshooting, not just AI-assisted incident review.
- Use AI as a collaborator: Let AI help analyze metrics, traces, logs, and possible causes without obscuring how the system behaves.
Tip
The most interesting part of this argument is that it exposes a tension between short-term productivity and long-term expertise.
An organization can become more efficient at resolving incidents today while quietly reducing its ability to respond to unfamiliar incidents tomorrow.