--- title: "Inject failures in production once staging ยท Agent Wranglers" url: https://agentwranglers.se/playbook/operations/disaster-recovery/production-failure-injection id: OPS-85 level: Optional updated: 2026-10-27 build: a2a6bdfed830 tokens: 655 licence: "CC BY 4.0" licence_url: https://creativecommons.org/licenses/by/4.0/ --- # Inject failures in production once staging drills pass For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Disaster recovery](https://agentwranglers.se/playbook/operations/disaster-recovery.md). By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27. - ID: OPS-85 - Level: Optional - Chapter: [Disaster recovery](https://agentwranglers.se/playbook/operations/disaster-recovery.md) - Rule: When four staging game days in a row met their recovery targets, run small planned failures in production. Limit each to a share of traffic, with an automatic stop. - Check: Each production experiment has a written expected result, a traffic limit and a stop condition tied to an alarm. The log shows the stop condition ended at least one test run on purpose, to prove it works. - How: Staging lacks production's traffic, data size and odd clients, so some failures only show there. Start with one failure the service should survive unseen. Examples are stopping one of several copies, or adding 200 ms to one dependency for 5% of requests. Write the expected result first, such as "the error rate stays under its objective". Set a stop condition on a service level alarm, so the test ends by itself when users suffer. Run it in working hours with the on-call person told. Tools such as AWS Fault Injection Service or Chaos Mesh run and stop the experiment. Staging drills come first, as [Run a game day every quarter](https://agentwranglers.se/playbook/operations/disaster-recovery/game-days.md) says. ## Sources - [AWS Well-Architected REL12-BP04 Test resiliency using chaos engineering](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_testing_resiliency_failure_injection_resiliency.html) ## Related rules - OPS-71 [Run a game day every quarter](https://agentwranglers.se/playbook/operations/disaster-recovery/game-days.md) in Operations and observability **For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `Agent Wranglers operations and observability playbook`. In code you write, add a comment next to it with the same link. - Previous: [Run a game day every quarter](https://agentwranglers.se/playbook/operations/disaster-recovery/game-days.md) - [Disaster recovery](https://agentwranglers.se/playbook/operations/disaster-recovery.md)