Agent WranglersOPS-69Recommended
Keep manual operations work under half the time
Log the hours each person spends on manual, repeated operations work, such as restarts, hand deploys and data fixes. Keep it under 50% of the team's time, and automate the largest item each quarter.
How to do it
Google's SRE book calls this work toil. It is manual, repeated, could be automated and grows with the service. Left alone it fills the week and no time remains to fix causes. Tag such tasks in the ticket system, and sum the hours each month. Turn the largest item into code or a check, as in keep runbooks next to the code and check every rule in code. Pages that need the same manual fix each time count as toil too, as in make every alert actionable. Turn hand data fixes into scripts, as run one-off tasks as reviewed scripts says.
How to check
The toil hours of the last quarter are under 50% of the team's working hours. The largest toil item of the previous quarter has a merged automation or an open ticket with an owner and a date.
Sources
Related rules
- Keep runbooks next to the code they run in Engineering
- Check every rule in code in Engineering
- Make every alert actionable in Operations
- Run one-off tasks as reviewed scripts in Engineering
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/operations/running-the-service/limit-toil">Agent Wranglers operations playbook</a>. In code you write, add a comment next to it with the same link.