Break the services you depend on in a test environment

How to do it

Code-level tests prove one client handles one failure, as Test how your code handles each outside service failing says. They miss what the whole system does. Pools run dry, retries pile up and health checks pass while requests hang. Put a fault proxy, such as Toxiproxy, between the app and its database, cache, queue and outside APIs on staging. For each dependency, add latency above its timeout, then drop the connection, then cut it for a minute. Run the critical journey tests during each fault and check that the app recovers once the fault is removed. Larger teams can use a chaos tool such as Chaos Mesh or a cloud fault injection service. The quarterly drill with people stays in Run a game day every quarter.

Examples

JavaScript example

// Add 3 seconds of latency to the payment API, then run the checkout journey
const TOXIPROXY = 'http://toxiproxy.staging.example.com:8474';
await fetch(`${TOXIPROXY}/proxies/payments/toxics`, {
  method: 'POST',
  body: JSON.stringify({ name: 'slow_payments', type: 'latency', attributes: { latency: 3000 } }),
});
try {
  await runJourney('checkout', { expect: 'payment-delayed-message' });
} finally {
  await fetch(`${TOXIPROXY}/proxies/payments/toxics/slow_payments`, { method: 'DELETE' });
}

Python example

import httpx

TOXIPROXY = "http://toxiproxy.staging.example.com:8474"

def test_checkout_survives_the_database_dropping():
    httpx.post(f"{TOXIPROXY}/proxies/database", json={"enabled": False})
    try:
        assert run_journey("checkout").shows("We could not take your order")
    finally:
        httpx.post(f"{TOXIPROXY}/proxies/database", json={"enabled": True})
    assert run_journey("checkout", within_seconds=60).succeeded()

How to check

The release pipeline holds a fault run with one result per dependency and fault type. Each result shows every journey passing or ending in its planned error, with no request open past its timeout. Every journey passes again within 60 seconds after the fault ends.

Sources

• Azure Well-Architected: reliability testing strategy

Related rules

• TST-16 Test how your code handles each outside service failing in Testing
• TST-17 Test every critical journey end to end in Testing
• OPS-71 Run a game day every quarter in Operations and observability