Catch speed regressions on every pull request

How to do it

A load test before launch finds a slow release weeks after the change that caused it. Benchmark the few functions and queries that most requests pass through, such as price calculation, search ranking or page rendering. Run the main branch and the pull request on the same runner, one after the other, so machine noise cancels out. Repeat each benchmark at least 10 times and compare with a statistical tool, such as benchstat for Go or the compare options of pytest-benchmark. Set a margin above the normal noise, such as 10 percent. Count allocations or database queries as well where the tool can, because those numbers do not drift with machine load. Keep the full load test for releases, as Test load against a budget says.

Examples

Shell example

# Go: run both branches on the same runner, then compare
git switch main && go test -run='^$' -bench=. -count=10 ./pricing > old.txt
git switch - && go test -run='^$' -bench=. -count=10 ./pricing > new.txt
benchstat old.txt new.txt

Shell example 2

# Python with pytest-benchmark: save the main branch, then fail the branch on a 10 percent slowdown
git switch main && pytest benchmarks/ --benchmark-autosave
git switch - && pytest benchmarks/ --benchmark-compare --benchmark-compare-fail=mean:10%

Go example

func BenchmarkCartTotal(b *testing.B) {
    cart := sampleCart(50)
    b.ReportAllocs()
    for b.Loop() {
        cart.Total()
    }
}

Python example

def test_cart_total_speed(benchmark):
    cart = sample_cart(items=50)
    benchmark(cart.total)

How to check

The pull request job runs the benchmarks on both branches and posts the comparison. Add a deliberate slowdown, such as a sleep of 20 percent of the run time, to a benchmarked function on a branch. The job fails and names the function.

Sources

• benchstat

Related rules

• TST-36 Test load against a budget in Testing