{"playbook":"Testing","url":"https://agentwranglers.se/playbook/testing","updated":"2026-10-21","license":"https://creativecommons.org/licenses/by/4.0/","credit":"Required by the licence: credit Agent Wranglers with a link to the rule or playbook you used.","how":"Each rule's How text is on its own page: add .md to its url.","rules":[{"chapter":"Test strategy","name":"Test what would hurt to break","level":"Required","risk":"money","rule":"Put the most tests where a bug would cost money, data or trust. Code that cannot break silently needs few tests.","check":"Every flow on the risk list has at least one test that fails when the flow breaks. Break each flow on purpose on a branch, and a test goes red.","url":"https://agentwranglers.se/playbook/testing/test-strategy/test-by-risk","related":["/playbook/ai-assisted-development/apps-built-with-ai-builders/checks-before-growth","/playbook/adversarial-testing/supply-chain-and-setup/threat-model","/playbook/testing/end-to-end-and-browser-tests/critical-journeys","/playbook/testing/test-quality/mutation-testing"],"sources":[]},{"chapter":"Test strategy","name":"Shape the test suite like a pyramid","level":"Recommended","rule":"Write many small, fast tests and only a few slow, wide ones. Each kind of test checks what only it can check.","check":"Count tests by kind. Unit tests outnumber integration tests, which outnumber browser tests. The full suite runs in under ten minutes.","url":"https://agentwranglers.se/playbook/testing/test-strategy/test-shape","related":["/playbook/testing/flaky-and-slow-tests/suite-time-budget"],"sources":[]},{"chapter":"Test strategy","name":"Ship tests in the same change","level":"Recommended","rule":"Every change that adds or changes behaviour includes the tests for it. A bug fix starts with a test that fails before the fix.","check":"In review, revert the code change but keep the test. The new test fails. Restore the code, and it passes.","url":"https://agentwranglers.se/playbook/testing/test-strategy/tests-with-the-change","related":["/playbook/architecture/dead-code/delete-tests-with-feature","/playbook/adversarial-testing/rules-of-engagement/fix-becomes-check","/playbook/infrastructure/policy-as-code/test-infrastructure-modules"],"sources":[]},{"chapter":"Unit tests","name":"Test behaviour through the public interface","level":"Recommended","rule":"A unit test calls a function or class the way callers do, and checks what it returns or changes.","check":"Refactor the inside of a function without changing its results. No test needs to change. No test calls a private method, for example through reflection, `@ts-expect-error` or a Python `_private` name.","url":"https://agentwranglers.se/playbook/testing/unit-tests/unit-behaviour","related":["/playbook/engineering/build-test-and-deploy/unit-tests-for-the-tricky-parts","/playbook/architecture/coupling/private-internals","/playbook/ai-features/agents-in-products/agent-scenario-tests"],"sources":[]},{"chapter":"Unit tests","name":"Fake the clock and the network in unit tests","level":"Recommended","rule":"Unit tests never depend on the clock, random numbers or the network. They pass on any day, on any machine.","check":"Run the suite in two far-apart time zones, such as `TZ=Pacific/Kiritimati` and `TZ=Pacific/Pago_Pago`. Run it with the date moved a year ahead, such as with `faketime '+365d'`. Then run it with the network off, such as in a container started with `--network none`. All runs pass.","url":"https://agentwranglers.se/playbook/testing/unit-tests/unit-deterministic","related":["/playbook/testing/test-data-and-environments/isolated-test-data","/playbook/testing/unit-tests/fast-unit-tests"],"sources":[]},{"chapter":"Unit tests","name":"Test the edges and the errors","level":"Recommended","rule":"For every rule with a limit, test the values on both sides of it. For every input that can be rejected, test the rejection.","check":"Each limit in the code has a test at the limit and one past it. Moving a limit by one, such as `>=` to `>`, makes a test fail. [Mutation testing](https://agentwranglers.se/playbook/testing/test-quality/mutation-testing) finds this mechanically.","url":"https://agentwranglers.se/playbook/testing/unit-tests/test-the-edges","related":["/playbook/testing/test-quality/mutation-testing","/playbook/testing/unit-tests/decision-tables"],"sources":[]},{"chapter":"Unit tests","name":"Arrange, act and assert","level":"Recommended","rule":"Write each test in three parts. Set up the data, do one action, then check the result.","check":"Pick ten tests at random. Each has setup, then one call to the code under test, then only assertions. No call to the code under test follows an assertion.","url":"https://agentwranglers.se/playbook/testing/unit-tests/arrange-act-assert","related":["/playbook/testing/test-data-and-environments/data-factories"],"sources":[]},{"chapter":"Unit tests","name":"Give each test one reason to fail","level":"Recommended","rule":"Each test checks one behaviour. When it fails, its name tells you what broke.","check":"Break one behaviour on purpose. Exactly one test, or a small group about that behaviour, fails.","url":"https://agentwranglers.se/playbook/testing/unit-tests/one-reason-to-fail","related":["/playbook/testing/test-quality/review-tests"],"sources":[]},{"chapter":"Unit tests","name":"Keep unit tests fast","level":"Recommended","rule":"Each unit test runs in milliseconds, and the whole unit suite runs in seconds.","check":"The slowest unit test takes under 100 milliseconds. The unit suite runs in under 30 seconds.","url":"https://agentwranglers.se/playbook/testing/unit-tests/fast-unit-tests","related":["/playbook/testing/unit-tests/unit-deterministic","/playbook/stack/testing-and-code-quality/pytest"],"sources":[]},{"chapter":"Unit tests","name":"Check rules that hold for any input with property tests","level":"Recommended","rule":"When a rule must hold for every input, such as a round trip or a total, state it once as a property. Let a tool generate hundreds of inputs to test it.","check":"Every function on the risk list that converts, encodes or calculates amounts has at least one property test. Break its rule on purpose, such as rounding one way only. The property test fails and prints a shrunk input.","url":"https://agentwranglers.se/playbook/testing/unit-tests/property-tests","related":["/playbook/payments/amounts-and-orders/allocate-remainders"],"sources":[{"label":"Hypothesis API reference","url":"https://hypothesis.readthedocs.io/en/latest/reference/api.html"},{"label":"fast-check getting started","url":"https://fast-check.dev/docs/introduction/getting-started/"}]},{"chapter":"Unit tests","name":"Test rules with several conditions as a table","level":"Recommended","rule":"When a result depends on two or more conditions, list every combination in a table with its expected result. Run each row as a test.","check":"Multiply the number of values of each condition. The table has that many rows, or merged rows that name the conditions they ignore. Change the result for one combination in the code, and only that row's test fails.","url":"https://agentwranglers.se/playbook/testing/unit-tests/decision-tables","related":["/playbook/testing/unit-tests/test-the-edges","/playbook/accounts/roles-and-permissions/permission-matrix-test"],"sources":[]},{"chapter":"Integration and contract tests","name":"Test against the same database engine","level":"Required","risk":"data","rule":"Test database code against the same database engine and version as production, never against a different engine or a mock.","check":"The test setup starts the production database engine at the same major version as production and runs every migration. A query that only works on the stand-in fails the suite.","url":"https://agentwranglers.se/playbook/testing/integration-and-contract-tests/same-database-engine","related":["/playbook/data/migrations/test-migrations"],"sources":[]},{"chapter":"Integration and contract tests","name":"Fake only what you do not own","level":"Recommended","rule":"Replace only outside services, such as a payment provider or an email API, with test doubles. Run your own code as it runs in production.","check":"Search the integration tests for mocks of your own modules, such as `vi.mock('../`, `jest.mock('./` or `mock.patch(\"shop.`. There are none. Doubles exist only for outside services.","url":"https://agentwranglers.se/playbook/testing/integration-and-contract-tests/doubles-at-the-edge","related":["/playbook/architecture/solid-and-dry/dependency-inversion","/playbook/testing/integration-and-contract-tests/outside-service-failures"],"sources":[]},{"chapter":"Integration and contract tests","name":"Check the contract between services","level":"Recommended","rule":"When two services you own talk to each other, test the shape of their messages on both sides with a shared contract.","check":"Remove a field that a consumer uses from the provider's answer. The provider's build fails on the contract test.","url":"https://agentwranglers.se/playbook/testing/integration-and-contract-tests/contract-tests","related":["/playbook/api-design/contracts-and-docs/contract-tests","/playbook/api-design/changes-over-time/additive-changes","/playbook/api-design/contracts-and-docs/openapi-contract","/playbook/api-design/webhooks-and-events/event-versions"],"sources":[]},{"chapter":"Integration and contract tests","name":"Check that outside services still match your fakes","level":"Optional","rule":"At least once a week, run a small test against each outside service to prove your fakes still behave like it.","check":"The scheduled job ran in the last 7 days and passed, or its failure has an open issue.","url":"https://agentwranglers.se/playbook/testing/integration-and-contract-tests/verify-the-fakes","related":["/playbook/api-design/contracts-and-docs/contract-tests"],"sources":[]},{"chapter":"Integration and contract tests","name":"Test how your code handles each outside service failing","level":"Required","risk":"data","rule":"Test what your code does when each outside service hangs, refuses the connection, answers with an error or sends an unexpected body. Nothing is lost or done twice.","check":"List every outside service in the code's settings. Each has four passing tests, for a hang past the time limit, a refused connection, a server error and a missing field.","url":"https://agentwranglers.se/playbook/testing/integration-and-contract-tests/outside-service-failures","related":["/playbook/testing/integration-and-contract-tests/doubles-at-the-edge","/playbook/api-design/reliability/timeouts","/playbook/api-design/reliability/retry-backoff","/playbook/payments/amounts-and-orders/payment-state-machine"],"sources":[]},{"chapter":"End-to-end and browser tests","name":"Cover only the critical journeys","level":"Required","risk":"outage","rule":"Write end-to-end tests for the few journeys that must never break, such as sign-up, checkout and booking. Leave the details to faster tests.","check":"Every journey on the list has one passing browser test. The browser suite runs in under ten minutes.","url":"https://agentwranglers.se/playbook/testing/end-to-end-and-browser-tests/critical-journeys","related":["/playbook/ai-assisted-development/reviewing-agent-output/run-it-yourself","/playbook/testing/test-strategy/test-by-risk","/playbook/testing/end-to-end-and-browser-tests/test-the-built-app","/playbook/payments/checkout-and-card-data/payment-test-scenarios"],"sources":[]},{"chapter":"End-to-end and browser tests","name":"Find elements the way people do","level":"Recommended","rule":"Find elements by their role, label or visible text. Never by CSS classes or page structure that change with the design.","check":"Search the browser tests for CSS and XPath selectors, such as `locator('.`, `locator('#`, `locator('//` or `xpath=`. There are none, or each has a comment explaining why.","url":"https://agentwranglers.se/playbook/testing/end-to-end-and-browser-tests/stable-selectors","related":["/playbook/website/accessibility/label-in-name","/playbook/app-ui/tests-and-budgets/component-state-tests"],"sources":[{"label":"WAI-ARIA 1.2","url":"https://www.w3.org/TR/wai-aria-1.2/"},{"label":"Accessible Name and Description Computation 1.2","url":"https://www.w3.org/TR/accname-1.2/"}]},{"chapter":"End-to-end and browser tests","name":"Wait for the page to be ready instead of sleeping","level":"Recommended","rule":"In every asynchronous test, wait for a condition, such as a heading appearing or a queue emptying. Never pause for a fixed time.","check":"Search all tests for fixed waits, such as `sleep`, `waitForTimeout` or a bare `setTimeout` used as a pause. There are none outside polling helpers that check a condition each round.","url":"https://agentwranglers.se/playbook/testing/end-to-end-and-browser-tests/no-sleeps","related":["/playbook/testing/flaky-and-slow-tests/repeat-new-tests","/playbook/stack/testing-and-code-quality/playwright"],"sources":[]},{"chapter":"End-to-end and browser tests","name":"Run browser tests against the production build","level":"Required","risk":"outage","rule":"Run end-to-end tests against the same build you deploy, with production settings, not against a development server.","check":"The test job runs the production build and start commands, never a development server such as `vite dev` or `npm run dev`. A bug that only appears in the built app fails the browser suite.","url":"https://agentwranglers.se/playbook/testing/end-to-end-and-browser-tests/test-the-built-app","related":["/playbook/testing/end-to-end-and-browser-tests/critical-journeys","/playbook/website/site-architecture/browser-audits-before-release"],"sources":[]},{"chapter":"End-to-end and browser tests","name":"Test in more than one browser and size","level":"Recommended","rule":"Run the critical journeys in at least two browser engines, at phone and desktop width.","check":"The test report lists each journey passing in two engines and two screen sizes.","url":"https://agentwranglers.se/playbook/testing/end-to-end-and-browser-tests/browsers-and-sizes","related":["/playbook/email/layout-and-accessibility/client-testing","/playbook/website/interface/dynamic-viewport-units","/playbook/website/accessibility/no-sideways-scrolling","/playbook/stack/testing-and-code-quality/playwright"],"sources":[]},{"chapter":"End-to-end and browser tests","name":"Smoke test every deploy","level":"Recommended","rule":"Right after each deploy, run a few fast, read-only tests against the live address. Roll back when one fails.","check":"Deploy a release with a broken key page to staging. The smoke tests fail, and the deploy script rolls back and exits with a code other than zero.","url":"https://agentwranglers.se/playbook/testing/end-to-end-and-browser-tests/smoke-test-every-deploy","related":["/playbook/engineering/build-test-and-deploy/health-check-and-rollback","/playbook/testing/output-comparison/compare-live-before-and-after"],"sources":[]},{"chapter":"End-to-end and browser tests","name":"Keep a trace of every failed browser test","level":"Recommended","rule":"When a browser test fails in continuous integration, keep its trace and screenshot as files of that run. Anyone can then see what happened without running it again.","check":"Make one browser test fail on a branch. The run's artifacts hold that test's trace and screenshot, and the CI report names the failing test.","url":"https://agentwranglers.se/playbook/testing/end-to-end-and-browser-tests/failure-traces","related":["/playbook/testing/test-data-and-environments/production-like-staging"],"sources":[{"label":"Playwright best practices","url":"https://playwright.dev/docs/best-practices"},{"label":"Playwright test use options","url":"https://playwright.dev/docs/test-use-options"}]},{"chapter":"Test data and environments","name":"Build test data with factories","level":"Recommended","rule":"Create test data with small factory functions that fill sensible defaults. Each test sets only the fields it cares about.","check":"Change a default in a factory. Only tests about that field change result.","url":"https://agentwranglers.se/playbook/testing/test-data-and-environments/data-factories","related":["/playbook/testing/unit-tests/arrange-act-assert","/playbook/data/personal-data/masked-copies"],"sources":[]},{"chapter":"Test data and environments","name":"Give each test its own data","level":"Recommended","rule":"Each test creates the data it needs and leaves nothing behind that another test could see.","check":"Run one test alone, then the whole suite in random order. Both pass.","url":"https://agentwranglers.se/playbook/testing/test-data-and-environments/isolated-test-data","related":["/playbook/testing/unit-tests/unit-deterministic","/playbook/payments/checkout-and-card-data/payment-test-scenarios"],"sources":[]},{"chapter":"Test data and environments","name":"Keep a staging copy with fake data","level":"Required","risk":"data","rule":"Keep a staging environment that runs the same build, settings and service versions as production, with fake data.","check":"Staging and production report the same build and runtime versions, such as on a version endpoint or in the deploy log. Staging uses none of production's databases or keys, and its user records hold only generated addresses, such as `@example.com`.","url":"https://agentwranglers.se/playbook/testing/test-data-and-environments/production-like-staging","related":["/playbook/testing/end-to-end-and-browser-tests/failure-traces","/playbook/data/personal-data/masked-copies","/playbook/infrastructure/environments/same-code-every-environment","/playbook/testing/output-comparison/compare-environments","/playbook/adversarial-testing/rules-of-engagement/test-a-copy","/playbook/adversarial-testing/rules-of-engagement/do-no-harm"],"sources":[]},{"chapter":"Flaky and slow tests","name":"Find hidden order dependencies","level":"Recommended","rule":"Run tests in a random order, so tests that depend on each other fail early.","check":"The test command shuffles the order and logs the seed. Running it again with that seed repeats the same order.","url":"https://agentwranglers.se/playbook/testing/flaky-and-slow-tests/random-order","related":["/playbook/architecture/coupling/no-global-state"],"sources":[]},{"chapter":"Flaky and slow tests","name":"Quarantine flaky tests with an owner","level":"Recommended","rule":"Move a test that fails at random out of the blocking suite at once. Give it an owner and a deadline, then fix or delete it.","check":"Every skipped test links to an open issue with an owner and a date. No quarantined test is older than its deadline.","url":"https://agentwranglers.se/playbook/testing/flaky-and-slow-tests/quarantine-flaky-tests","related":["/playbook/engineering/version-control-and-review/keep-main-green"],"sources":[]},{"chapter":"Flaky and slow tests","name":"Find flaky tests before they merge","level":"Recommended","rule":"Run new and changed tests many times in a row before merging, to catch flakiness while the change is fresh.","check":"The pull request job repeats changed tests and fails if any run fails.","url":"https://agentwranglers.se/playbook/testing/flaky-and-slow-tests/repeat-new-tests","related":["/playbook/testing/end-to-end-and-browser-tests/no-sleeps","/playbook/ai-features/evals-and-quality/repeat-eval-runs"],"sources":[]},{"chapter":"Flaky and slow tests","name":"Keep the suite within a time budget","level":"Recommended","rule":"Set a time limit for the whole suite, such as ten minutes, and for each kind of test. Treat going over it as a failure.","check":"The pipeline records each stage's duration and sets a time limit on each, such as `timeout-minutes` in GitHub Actions or `timeout 600` in a script. The run fails when a stage passes its limit.","measured":"On a reference site, the full check went from 12 minutes to about 70 seconds. Steps ran in parallel, and the browser check sampled pages of one template. Its load test, run beside the browser check, failed with a 95th percentile from 42 to 3,874 ms. Run alone, it passed.","url":"https://agentwranglers.se/playbook/testing/flaky-and-slow-tests/suite-time-budget","related":["/playbook/testing/test-strategy/test-shape"],"sources":[]},{"chapter":"Test quality","name":"Prove the tests catch bugs","level":"Recommended","rule":"Run mutation testing on the high-risk modules from the [risk list](https://agentwranglers.se/playbook/testing/test-strategy/test-by-risk). It changes the code on purpose and checks that at least one test fails for each change.","check":"The mutation report for each high-risk module meets the set score, and the build fails when it does not.","url":"https://agentwranglers.se/playbook/testing/test-quality/mutation-testing","related":["/playbook/testing/unit-tests/test-the-edges","/playbook/testing/test-strategy/test-by-risk","/playbook/testing/test-quality/coverage-floor","/playbook/testing/test-quality/tests-can-fail"],"sources":[]},{"chapter":"Test quality","name":"Treat coverage as a floor","level":"Recommended","rule":"Use line and branch coverage to find untested code and to stop it falling. Never treat a high number as proof of good tests.","check":"The test run fails when coverage falls below the set minimum.","url":"https://agentwranglers.se/playbook/testing/test-quality/coverage-floor","related":["/playbook/testing/test-quality/mutation-testing"],"sources":[]},{"chapter":"Test quality","name":"Make every test able to fail","level":"Recommended","rule":"Every test must contain a check that fails when the behaviour breaks. Tests without assertions or with always-true checks are not tests.","check":"The linter reports no test without an assertion. Breaking the code under a new test turns it red.","url":"https://agentwranglers.se/playbook/testing/test-quality/tests-can-fail","related":["/playbook/ai-assisted-development/reviewing-agent-output/no-weakened-checks","/playbook/testing/test-quality/mutation-testing"],"sources":[]},{"chapter":"Test quality","name":"Review tests like code","level":"Recommended","rule":"Review test code with the same care as production code. It must be clear, small and free of copy and paste.","check":"The linter and type checker run on test files with the same settings as production code. A copy-and-paste detector, such as jscpd or PMD CPD, fails the check on duplicated blocks in tests. The [code review questions](https://agentwranglers.se/playbook/architecture/code-review) cover the test files in every pull request.","url":"https://agentwranglers.se/playbook/testing/test-quality/review-tests","related":["/playbook/testing/unit-tests/one-reason-to-fail","/playbook/ai-assisted-development/reviewing-agent-output/no-weakened-checks"],"sources":[]},{"chapter":"Test quality","name":"Keep snapshot tests small and reviewed","level":"Recommended","rule":"Use snapshots only for small, stable output. Review every snapshot change like code, and never accept new snapshots without reading them.","check":"In continuous integration, a missing or changed snapshot fails the run instead of being written, as with Jest's `--ci` flag. No stored text snapshot is longer than 50 lines.","url":"https://agentwranglers.se/playbook/testing/test-quality/small-snapshots","related":["/playbook/testing/output-comparison/refactor-changes-nothing","/playbook/app-ui/tests-and-budgets/visual-regression"],"sources":[]},{"chapter":"Non-functional tests","name":"Test load against a budget","level":"Recommended","rule":"Before a launch or a big change, load test the busiest routes and fail when response times or errors pass a set budget.","check":"The load test exits with an error when a limit is passed. The latest run before the launch passed. The run also fails when memory passes its ceiling.","url":"https://agentwranglers.se/playbook/testing/non-functional-tests/load-tests","related":["/playbook/website/performance/performance-budget","/playbook/adversarial-testing/rules-of-engagement/fix-becomes-check","/playbook/operations/running-the-service/launch-readiness","/playbook/operations/capacity-and-cost/headroom","/playbook/operations/capacity-and-cost/runtime-memory-limits"],"sources":[]},{"chapter":"Non-functional tests","name":"Check accessibility in every build","level":"Required","risk":"law","rule":"Run an automated accessibility check on every page or screen in the test suite, and fail the build on any violation.","check":"Add an image without alt text on a branch. The browser suite fails and names the element.","url":"https://agentwranglers.se/playbook/testing/non-functional-tests/accessibility-in-ci","related":["/playbook/website/site-architecture/browser-audits-before-release","/playbook/website/accessibility/empty-links-buttons","/playbook/compliance/product-and-sector-laws/accessibility-act-duty","/playbook/app-ui/styles-and-components/component-catalogue","/playbook/app-ui/tests-and-budgets/component-state-tests","/playbook/stack/testing-and-code-quality/axe-core"],"sources":[{"label":"WCAG 2.2","url":"https://www.w3.org/TR/WCAG22/"}]},{"chapter":"Non-functional tests","name":"Grade AI features against fixed examples","level":"Recommended","applies":["ai-features"],"rule":"Test every AI feature against a fixed set of graded examples, and fail the build when the score drops.","check":"Change the prompt so one known answer goes wrong. The eval run fails in CI and names the failing example.","url":"https://agentwranglers.se/playbook/testing/non-functional-tests/llm-evals","related":["/playbook/ai-assisted-development/measuring-and-improving/evaluate-agent-setup","/playbook/ai-features/evals-and-quality/evals-from-production","/playbook/operations/running-the-service/pin-model-versions","/playbook/ai-features/deciding-to-use-a-model/define-ai-success","/playbook/ai-features/prompts-as-code/prompt-change-review","/playbook/ai-features/retrieval/retrieval-evals","/playbook/ai-features/evals-and-quality/code-graders-first","/playbook/stack/ai-software/eval-tool-optional"],"sources":[]},{"chapter":"Non-functional tests","name":"Fuzz every parser of untrusted input","level":"Recommended","rule":"Feed generated and random input to every parser of outside data, and turn every crash into a regular test.","check":"The fuzz job runs in CI for its set time and finds no crash. Every past crash input is stored and passes as a unit test.","url":"https://agentwranglers.se/playbook/testing/non-functional-tests/fuzz-parsers","related":["/playbook/architecture/structure/memory-safe-languages","/playbook/adversarial-testing/injection/unsafe-deserialisation","/playbook/adversarial-testing/rules-of-engagement/scan-every-change","/playbook/adversarial-testing/injection/xml-entities"],"sources":[{"label":"NIST SP 800-218: Secure Software Development Framework","url":"https://csrc.nist.gov/pubs/sp/800/218/final"}]},{"chapter":"Output comparison","name":"Build the same input to the same bytes","level":"Recommended","rule":"Build the same commit twice in two clean folders and compare the SHA-256 hash of every output file. Every hash must match.","check":"A job builds the same commit twice, in two empty folders, and compares both hash lists. One file that differs, or one file present in only one build, fails the job.","measured":"On a reference site, two builds of the same commit gave 4,398 identical files.","url":"https://agentwranglers.se/playbook/testing/output-comparison/same-input-same-bytes","related":["/playbook/engineering/running-in-production/reproducible-builds","/playbook/testing/output-comparison/narrow-masks"],"sources":[]},{"chapter":"Output comparison","name":"Write a manifest of the whole output","level":"Recommended","rule":"Every build writes one JSON manifest that lists the path, size in bytes and SHA-256 hash of every output file, sorted by path. Keep it with the release.","check":"Every release folder holds a manifest. A test counts the files in the output and fails when the manifest lists a different number, or when its paths are not sorted.","url":"https://agentwranglers.se/playbook/testing/output-comparison/output-manifest","related":["/playbook/engineering/build-test-and-deploy/atomic-deploys-with-rollback","/playbook/testing/output-comparison/compare-live-before-and-after","/playbook/testing/output-comparison/refactor-changes-nothing"],"sources":[]},{"chapter":"Output comparison","name":"Prove a refactor changes nothing","level":"Recommended","rule":"Compare the manifest from before a change with the one from after it. Any difference fails a change that does not name its expected paths.","check":"The pull request job builds both commits and fails when the manifests differ and the change lists no expected paths. A test change that alters one byte of one page makes the job fail.","url":"https://agentwranglers.se/playbook/testing/output-comparison/refactor-changes-nothing","related":["/playbook/testing/output-comparison/output-manifest","/playbook/testing/test-quality/small-snapshots","/playbook/testing/output-comparison/expected-differences"],"sources":[]},{"chapter":"Output comparison","name":"Explain every difference in an intended change","level":"Recommended","rule":"A change that should alter output lists its expected paths or patterns. The diff report shows changed, added and removed files. Any file outside the list fails the change.","check":"The job fails when any changed, added or removed path matches no pattern in the list. It also fails when a pattern in the list matches no difference, so stale lists do not pass.","url":"https://agentwranglers.se/playbook/testing/output-comparison/expected-differences","related":["/playbook/testing/output-comparison/refactor-changes-nothing","/playbook/testing/output-comparison/compare-live-before-and-after"],"sources":[]},{"chapter":"Output comparison","name":"Mask volatile parts narrowly","level":"Recommended","rule":"Replace each part that must change on every build with a fixed marker before hashing. Use a named list of at most 10 patterns, each with a maximum match count per file.","check":"The mask list has at most 10 entries, and none skips a whole file. A test fails when one mask matches more places in a file than its stated maximum.","url":"https://agentwranglers.se/playbook/testing/output-comparison/narrow-masks","related":["/playbook/testing/output-comparison/same-input-same-bytes","/playbook/testing/output-comparison/compare-live-before-and-after"],"sources":[]},{"chapter":"Output comparison","name":"Compare the live system before and after each deploy","level":"Recommended","rule":"Before and after each deploy, fetch every public address with fixed request headers. Record status, key headers and a body hash. The live diff must match the local diff.","check":"After each deploy, a job compares the two snapshots against the [expected paths](https://agentwranglers.se/playbook/testing/output-comparison/expected-differences). One address that changed and is not listed raises an alert or rolls the deploy back.","url":"https://agentwranglers.se/playbook/testing/output-comparison/compare-live-before-and-after","related":["/playbook/testing/output-comparison/output-manifest","/playbook/testing/output-comparison/narrow-masks","/playbook/testing/end-to-end-and-browser-tests/smoke-test-every-deploy","/playbook/engineering/build-test-and-deploy/atomic-deploys-with-rollback","/playbook/testing/output-comparison/expected-differences","/playbook/testing/output-comparison/compare-environments"],"sources":[]},{"chapter":"Output comparison","name":"Compare environments with the same release","level":"Optional","rule":"The same release on staging and production gives the same bytes at every address. The only exceptions are a named list of environment values.","check":"After each deploy to production, a job compares both snapshots. It fails on any difference that the list of environment values does not explain. The list holds at most 5 values.","url":"https://agentwranglers.se/playbook/testing/output-comparison/compare-environments","related":["/playbook/testing/output-comparison/compare-live-before-and-after","/playbook/testing/test-data-and-environments/production-like-staging"],"sources":[]},{"chapter":"Output comparison","name":"Keep the previous release's manifest","level":"Required","risk":"outage","rule":"Keep the manifest and live snapshot of each of the last five releases. Diff every new release against the one before it, and roll back to a release whose manifest you know.","check":"Each of the last five release folders holds a manifest and a live snapshot. A rollback test deploys, rolls back and fails unless the new live snapshot equals the stored one.","url":"https://agentwranglers.se/playbook/testing/output-comparison/keep-previous-manifest","related":["/playbook/engineering/build-test-and-deploy/atomic-deploys-with-rollback","/playbook/website/site-architecture/keep-the-previous-build-s-assets"],"sources":[]}]}