--- title: "Run a game day every quarter ยท Agent Wranglers" url: https://agentwranglers.se/playbook/operations/disaster-recovery/game-days id: OPS-71 level: Recommended updated: 2026-10-22 tokens: 874 licence: "CC BY 4.0" licence_url: https://creativecommons.org/licenses/by/4.0/ --- # Run a game day every quarter For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Disaster recovery](https://agentwranglers.se/playbook/operations/disaster-recovery.md). By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22. - ID: OPS-71 - Level: Recommended - Chapter: [Disaster recovery](https://agentwranglers.se/playbook/operations/disaster-recovery.md) - Rule: Every quarter, break one part of a staging copy on purpose and let the on-call person find and fix it. Time how long both steps take. - Check: The game day log holds one entry for each of the last four quarters. Each entry names the failure, the time to detect, the time to recover and links to the fixes made. - How: Runbooks and alerts decay until someone uses them, and an outage is a poor time to find that out. Pick one failure each time. Examples are a heavy request that stalls the server, a stopped database, a full disk, an expired certificate or a lost provider. Announce the date to everyone, but keep the failure from the person on call. Measure the time to detect from the alert, and the time to recover from the [runbook](https://agentwranglers.se/playbook/operations/alerts-and-on-call/actionable-alerts.md). Fix what slowed them down, and turn each fix into a check. Rotate through restores in [Practise restoring](https://agentwranglers.se/playbook/operations/disaster-recovery/restore-drills.md), [break-glass accounts](https://agentwranglers.se/playbook/infrastructure/identity-and-access/break-glass-access.md), [alert delivery](https://agentwranglers.se/playbook/operations/alerts-and-on-call/alert-pipeline-heartbeat.md) and the [provider plan](https://agentwranglers.se/playbook/operations/disaster-recovery/provider-failure.md). ## Sources - [AWS Well-Architected, REL12-BP05 Conduct game days regularly](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_testing_resiliency_game_days_resiliency.html) ## Related rules - OPS-19 [Make every alert actionable](https://agentwranglers.se/playbook/operations/alerts-and-on-call/actionable-alerts.md) in Operations and observability - OPS-48 [Practise restoring](https://agentwranglers.se/playbook/operations/disaster-recovery/restore-drills.md) in Operations and observability - INF-21 [Keep tested break-glass accounts](https://agentwranglers.se/playbook/infrastructure/identity-and-access/break-glass-access.md) in Infrastructure and environments - OPS-24 [Prove that alerts still reach you](https://agentwranglers.se/playbook/operations/alerts-and-on-call/alert-pipeline-heartbeat.md) in Operations and observability - OPS-49 [Plan for losing a provider](https://agentwranglers.se/playbook/operations/disaster-recovery/provider-failure.md) in Operations and observability **For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `Agent Wranglers operations and observability playbook`. In code you write, add a comment next to it with the same link. - Previous: [Add a second region only when the recovery targets need it](https://agentwranglers.se/playbook/operations/disaster-recovery/second-region.md) - [Disaster recovery](https://agentwranglers.se/playbook/operations/disaster-recovery.md)