Agent Wranglers

Required

Test how your code handles each outside service failing

Test what your code does when each outside service hangs, refuses the connection, answers with an error or sends an unexpected body. Nothing is lost or done twice.

Risk if broken

Data leaked or lost

How to do it

Give the fake from Fake only what you do not own a switch for each failure. It can hang, close the connection, answer 429, 500 or 503, or leave out a field. For each failure, check what the person sees and what your data holds afterwards. An order stays unpaid, a job stays queued for a retry, and no email goes out twice. A proxy that injects faults, such as Toxiproxy, does the same for a database or a queue. The time limits come from Set timeouts on every call, and the retries from Retry with backoff and jitter.

Examples

Using these examples: they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

JavaScript example

import { expect, test } from 'vitest';

test('a payment provider that never answers leaves the order unpaid', async () => {
  const neverAnswers = { charge: () => new Promise(() => {}) };
  const shop = createShop({ payments: neverAnswers, paymentTimeoutMs: 50 });
  const order = await shop.placeOrder(makeOrder({ total: 1999 }));

  await expect(shop.pay(order.id)).rejects.toThrow('payment_unavailable');

  expect((await shop.getOrder(order.id)).status).toBe('awaiting_payment');
});

Python example

import httpx
import pytest

def hangs(request): raise httpx.ReadTimeout("no answer", request=request)
def refuses(request): raise httpx.ConnectError("refused", request=request)
def errors(request): return httpx.Response(503)
def missing_field(request): return httpx.Response(200, json={"status": "created"})

@pytest.mark.parametrize("failure", [hangs, refuses, errors, missing_field])
def test_a_label_failure_keeps_the_shipment_queued(failure, db):
    carrier = CarrierClient(httpx.Client(transport=httpx.MockTransport(failure)))
    shipment = ShipmentFactory(status="queued")

    create_label(shipment, carrier)

    assert db.get(Shipment, shipment.id).status == "queued"
    assert db.count(Label) == 0

How to check

List every outside service in the code's settings. Each has four passing tests, for a hang past the time limit, a refused connection, a server error and a missing field.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/testing/integration-and-contract-tests/outside-service-failures">Agent Wranglers testing playbook</a>. In code you write, add a comment next to it with the same link.