Prove a refactor changes nothing

How to do it

Build the main branch and the change, then diff the two manifests. A refactor, a dependency bump with no intended effect, or a moved file must give an empty diff. This is approval testing, also called a golden master, applied to the whole system. It catches what the small snapshot tests never look at. An empty diff proves the output is the same. It does not replace the unit tests that prove the output is right.

Examples

Using these examples: they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

JavaScript example

import { readFileSync } from 'node:fs'

const byPath = file => new Map(JSON.parse(readFileSync(file, 'utf8')).map(entry => [entry.path, entry.sha256]))
export function compare(beforeFile, afterFile) {
  const before = byPath(beforeFile), after = byPath(afterFile)
  const paths = [...new Set([...before.keys(), ...after.keys()])].sort()
  return {
    added: paths.filter(path => !before.has(path)),
    removed: paths.filter(path => !after.has(path)),
    changed: paths.filter(path => before.has(path) && after.has(path) && before.get(path) !== after.get(path)),
  }
}

Python example

import json, sys

def by_path(name):
    return {entry["path"]: entry["sha256"] for entry in json.load(open(name))}

before, after = by_path("before.json"), by_path("after.json")
differing = sorted(p for p in before.keys() | after.keys() if before.get(p) != after.get(p))
if differing:
    print("\n".join(differing))
    sys.exit(1)

How to check

The pull request job builds both commits and fails when the manifests differ and the change lists no expected paths. A test change that alters one byte of one page makes the job fail.

Related rules

• Write a manifest of the whole output in Testing
• Keep snapshot tests small and reviewed in Testing
• Explain every difference in an intended change in Testing