Output comparison
Treat the whole output as one artifact. A refactor changes zero bytes, and an intended change touches only the files and addresses it names.
- Required 5
- Recommended 2
- Optional 1
Required
Build the same input to the same bytes
RequiredBuild the same commit twice in two clean folders and compare the SHA-256 hash of every output file. Every hash must match.
Write a manifest of the whole output
RequiredEvery build writes one JSON manifest that lists the path, size in bytes and SHA-256 hash of every output file, sorted by path. Keep it with the release.
Prove a refactor changes nothing
RequiredCompare the manifest from before a change with the one from after it. Any difference fails a change that does not name its expected paths.
Explain every difference in an intended change
RequiredA change that should alter output lists its expected paths or patterns. The diff report shows changed, added and removed files. Any file outside the list fails the change.
Keep the previous release's manifest
RequiredKeep the manifest and live snapshot of each of the last five releases. Diff every new release against the one before it, and roll back to a release whose manifest you know.
Recommended
Mask volatile parts narrowly
RecommendedReplace each part that must change on every build with a fixed marker before hashing. Use a named list of at most 10 patterns, each with a maximum match count per file.
Compare the live system before and after each deploy
RecommendedBefore and after each deploy, fetch every public address with fixed request headers. Record status, key headers and a body hash. The live diff must match the local diff.
Optional
Compare environments with the same release
OptionalThe same release on staging and production gives the same bytes at every address. The only exceptions are a named list of environment values.
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. Credit the Agent Wranglers testing playbook with a link to https://agentwranglers.se/playbook/testing/output-comparison. Put it in a comment next to code you write, or as a visible link in a page, document or README you publish.