---
title: "Output comparison · Agent Wranglers"
url: https://agentwranglers.se/playbook/testing/output-comparison
updated: 2026-10-21
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Output comparison

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers). Updated 2026-10-21.

For AI agents: [llms.txt](https://agentwranglers.se/llms.txt) is a short guide that sends you to the rules your problem needs, and [rules.json](https://agentwranglers.se/playbook/testing/rules.json) holds every rule of this playbook as JSON.

> Treat the whole output as one artifact. A refactor changes zero bytes, and an intended change touches only the files and addresses it names.

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/testing/output-comparison">Agent Wranglers testing playbook</a>`. In code you write, add a comment next to it with the same link.

## Required

- [Keep the previous release's manifest](https://agentwranglers.se/playbook/testing/output-comparison/keep-previous-manifest): Keep the manifest and live snapshot of each of the last five releases. Diff every new release against the one before it, and roll back to a release whose manifest you know.

## Recommended

- [Build the same input to the same bytes](https://agentwranglers.se/playbook/testing/output-comparison/same-input-same-bytes): Build the same commit twice in two clean folders and compare the SHA-256 hash of every output file. Every hash must match.
- [Write a manifest of the whole output](https://agentwranglers.se/playbook/testing/output-comparison/output-manifest): Every build writes one JSON manifest that lists the path, size in bytes and SHA-256 hash of every output file, sorted by path. Keep it with the release.
- [Prove a refactor changes nothing](https://agentwranglers.se/playbook/testing/output-comparison/refactor-changes-nothing): Compare the manifest from before a change with the one from after it. Any difference fails a change that does not name its expected paths.
- [Explain every difference in an intended change](https://agentwranglers.se/playbook/testing/output-comparison/expected-differences): A change that should alter output lists its expected paths or patterns. The diff report shows changed, added and removed files. Any file outside the list fails the change.
- [Mask volatile parts narrowly](https://agentwranglers.se/playbook/testing/output-comparison/narrow-masks): Replace each part that must change on every build with a fixed marker before hashing. Use a named list of at most 10 patterns, each with a maximum match count per file.
- [Compare the live system before and after each deploy](https://agentwranglers.se/playbook/testing/output-comparison/compare-live-before-and-after): Before and after each deploy, fetch every public address with fixed request headers. Record status, key headers and a body hash. The live diff must match the local diff.

## Optional

- [Compare environments with the same release](https://agentwranglers.se/playbook/testing/output-comparison/compare-environments): The same release on staging and production gives the same bytes at every address. The only exceptions are a named list of environment values.

- Previous: [Non-functional tests](https://agentwranglers.se/playbook/testing/non-functional-tests)
- [All chapters](https://agentwranglers.se/playbook/testing)