---
title: "Break the services you depend on in a test · Agent Wranglers"
url: https://agentwranglers.se/playbook/testing/security-and-robustness/environment-fault-injection
id: TST-63
level: Recommended
applies: hosted
updated: 2026-10-22
build: a2a6bdfed830
tokens: 1149
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Break the services you depend on in a test environment

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Security and robustness](https://agentwranglers.se/playbook/testing/security-and-robustness.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: TST-63
- Level: Recommended
- Chapter: [Security and robustness](https://agentwranglers.se/playbook/testing/security-and-robustness.md)
- Rule: Before each release, run the critical journeys on staging while a fault proxy slows, cuts or resets the connections to each dependency in turn. Each journey ends in its planned fallback or a clear error, never a hang.
- Check: The release pipeline holds a fault run with one result per dependency and fault type. Each result shows every journey passing or ending in its planned error, with no request open past its timeout. Every journey passes again within 60 seconds after the fault ends.
- Applies when: The project runs a hosted website or online service.
- How: Code-level tests prove one client handles one failure, as [Test how your code handles each outside service failing](https://agentwranglers.se/playbook/testing/integration-and-contract-tests/outside-service-failures.md) says. They miss what the whole system does. Pools run dry, retries pile up and health checks pass while requests hang. Put a fault proxy, such as Toxiproxy, between the app and its database, cache, queue and outside APIs on staging. For each dependency, add latency above its timeout, then drop the connection, then cut it for a minute. Run the [critical journey tests](https://agentwranglers.se/playbook/testing/end-to-end-and-browser-tests/critical-journeys.md) during each fault and check that the app recovers once the fault is removed. Larger teams can use a chaos tool such as Chaos Mesh or a cloud fault injection service. The quarterly drill with people stays in [Run a game day every quarter](https://agentwranglers.se/playbook/operations/disaster-recovery/game-days.md).

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JavaScript example

```js
// Add 3 seconds of latency to the payment API, then run the checkout journey
const TOXIPROXY = 'http://toxiproxy.staging.example.com:8474';
await fetch(`${TOXIPROXY}/proxies/payments/toxics`, {
  method: 'POST',
  body: JSON.stringify({ name: 'slow_payments', type: 'latency', attributes: { latency: 3000 } }),
});
try {
  await runJourney('checkout', { expect: 'payment-delayed-message' });
} finally {
  await fetch(`${TOXIPROXY}/proxies/payments/toxics/slow_payments`, { method: 'DELETE' });
}
```

### Python example

```python
import httpx

TOXIPROXY = "http://toxiproxy.staging.example.com:8474"

def test_checkout_survives_the_database_dropping():
    httpx.post(f"{TOXIPROXY}/proxies/database", json={"enabled": False})
    try:
        assert run_journey("checkout").shows("We could not take your order")
    finally:
        httpx.post(f"{TOXIPROXY}/proxies/database", json={"enabled": True})
    assert run_journey("checkout", within_seconds=60).succeeded()
```

## Sources

- [Azure Well-Architected: reliability testing strategy](https://learn.microsoft.com/en-us/azure/well-architected/reliability/reliability-test)

## Related rules

- TST-16 [Test how your code handles each outside service failing](https://agentwranglers.se/playbook/testing/integration-and-contract-tests/outside-service-failures.md) in Testing
- TST-17 [Test every critical journey end to end](https://agentwranglers.se/playbook/testing/end-to-end-and-browser-tests/critical-journeys.md) in Testing
- OPS-71 [Run a game day every quarter](https://agentwranglers.se/playbook/operations/disaster-recovery/game-days.md) in Operations and observability

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/testing/security-and-robustness/environment-fault-injection">Agent Wranglers testing playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Run the tests with a race detector](https://agentwranglers.se/playbook/testing/security-and-robustness/race-detector.md)
- [Security and robustness](https://agentwranglers.se/playbook/testing/security-and-robustness.md)