Eliminating Toil

sitereliabilityengineering sitereliabilityengineering/sre-book sitereliabilityengineering/concepts

Google SRE - What is Toil in SRE: Understanding Its Impact
Pragmatic SRE - Reducing Toil

Core Idea

Toil is the manual, repetitive, automatable, interrupt-driven work of running a service that leaves nothing behind and grows linearly as the service grows. If a human must touch the system during normal operations, you have a bug.

• The six traits: manual, repetitive, automatable, tactical (reactive, not planned), no enduring value, and O(n) with service growth.
• What does not count: overhead like meetings and paperwork, and grungy-but-valuable work like cleaning up alert configuration.
• Eliminating it means pushing automation, infrastructure as code, and CI/CD into incident response, capacity, configuration, backups, deploys, and monitoring.

Quote

If a human operator needs to touch your system during normal operations, you have a bug. The definition of normal changes as your systems grow.
⎯ Carla Geisser

Toil

• Toil is not just “work i don’t like to do”
• Administrative chores (HR Paperworks, team meetings) that have to get done, but should not be categorized as toil: this is overhead.
• Cleaning up the entire alerting configuration for your service and removing clutter may be grungy, but it’s not toil.

Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows

• Manual: Requires human effort, even if it’s just running a script.
• Repetitive: Done repeatedly rather than being a one-time or novel task.
• Automatable: Could be automated or eliminated through better system design.
• Tactical: Reactive and interrupt-driven (e.g., responding to alerts) instead of strategic or planned.
• No enduring value: Doesn’t create lasting improvements; the system is essentially unchanged after the task is completed.
• Scales with growth (O(n)): The amount of work increases linearly as the service, traffic, or user base grows.

Eliminating toil

Eliminating toil means automating low-value, repetitive ops work so SREs can focus on making systems more reliable, scalable, and efficient.

Area	Automation	Infrastructure as Code (IaC)	CI/CD
**[[Incident Management	Incident Response]]**	Automatically detects and attempts to recover from incidents.	Enables quick rollback to stable infrastructure versions.
Capacity Planning & Scaling	Automatically scales resources based on demand.	Uses templates to provision consistent infrastructure.	Deploys new instances with the latest tested software.
Configuration Management	Automatically applies configuration changes across systems.	Stores configurations as code for consistency across environments.	Tests configuration changes before production deployment.
Database Backup & Recovery	Schedules automatic backups and restorations.	Defines backup infrastructure and processes as code.	Tests and deploys backup process improvements continuously.
Software Deployment	Automatically deploys code after merges.	Provisions application infrastructure consistently.	Builds, tests, and deploys code reliably with automated testing.
**[[Observability Primer	Monitoring & Alerting]]**	Continuously monitors systems and triggers alerts or corrective actions.	Configures monitoring tools consistently across infrastructure.