Required
Explain every difference in an intended change
A change that should alter output lists its expected paths or patterns. The diff report shows changed, added and removed files. Any file outside the list fails the change.
How to do it
Keep the expected list in the pull request, for example as a file such as expected-changes.txt with one path or glob pattern per line. The manifest diff sorts files into changed, added and removed. Show the reviewer a line-by-line text diff for each changed text file, and the size change for each binary file. A new footer link that changes 400 pages is fine when the list says **/*.html. A changed robots.txt that the list does not name fails.
Examples
Using these examples: they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.
JavaScript example
import { matchesGlob } from 'node:path'
export function unexpected(diff, patterns) {
const paths = [...diff.added, ...diff.removed, ...diff.changed]
return {
surprises: paths.filter(path => !patterns.some(pattern => matchesGlob(path, pattern))),
unused: patterns.filter(pattern => !paths.some(path => matchesGlob(path, pattern))),
}
}
Python example
from fnmatch import fnmatch
def unexpected(diff, patterns):
paths = diff["added"] + diff["removed"] + diff["changed"]
return {
"surprises": [p for p in paths if not any(fnmatch(p, pat) for pat in patterns)],
"unused": [pat for pat in patterns if not any(fnmatch(p, pat) for p in paths)],
}
How to check
The job fails when any changed, added or removed path matches no pattern in the list. It also fails when a pattern in the list matches no difference, so stale lists do not pass.
Related rules
- Prove a refactor changes nothing in Testing
- Compare the live system before and after each deploy in Testing
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. Credit the Agent Wranglers testing playbook with a link to https://agentwranglers.se/playbook/testing/output-comparison/expected-differences. Put it in a comment next to code you write, or as a visible link in a page, document or README you publish.