Agent WranglersOPS-85Optional
Inject failures in production once staging drills pass
When four staging game days in a row met their recovery targets, run small planned failures in production. Limit each to a share of traffic, with an automatic stop.
How to do it
Staging lacks production's traffic, data size and odd clients, so some failures only show there. Start with one failure the service should survive unseen. Examples are stopping one of several copies, or adding 200 ms to one dependency for 5% of requests. Write the expected result first, such as "the error rate stays under its objective". Set a stop condition on a service level alarm, so the test ends by itself when users suffer. Run it in working hours with the on-call person told. Tools such as AWS Fault Injection Service or Chaos Mesh run and stop the experiment. Staging drills come first, as Run a game day every quarter says.
How to check
Each production experiment has a written expected result, a traffic limit and a stop condition tied to an alarm. The log shows the stop condition ended at least one test run on purpose, to prove it works.
Sources
Related rules
- OPS-71 Run a game day every quarter in Operations and observability
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/operations/disaster-recovery/production-failure-injection">Agent Wranglers operations and observability playbook</a>. In code you write, add a comment next to it with the same link.