Agent WranglersOPS-71Recommended
Run a game day every quarter
Every quarter, break one part of a staging copy on purpose and let the on-call person find and fix it. Time how long both steps take.
How to do it
Runbooks and alerts decay until someone uses them, and an outage is a poor time to find that out. Pick one failure each time. Examples are a heavy request that stalls the server, a stopped database, a full disk, an expired certificate or a lost provider. Announce the date to everyone, but keep the failure from the person on call. Measure the time to detect from the alert, and the time to recover from the runbook. Fix what slowed them down, and turn each fix into a check. Rotate through restores in Practise restoring, break-glass accounts, alert delivery and the provider plan.
How to check
The game day log holds one entry for each of the last four quarters. Each entry names the failure, the time to detect, the time to recover and links to the fixes made.
Sources
Related rules
- OPS-19 Make every alert actionable in Operations and observability
- OPS-48 Practise restoring in Operations and observability
- INF-21 Keep tested break-glass accounts in Infrastructure and environments
- OPS-24 Prove that alerts still reach you in Operations and observability
- OPS-49 Plan for losing a provider in Operations and observability
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/operations/disaster-recovery/game-days">Agent Wranglers operations and observability playbook</a>. In code you write, add a comment next to it with the same link.