Agent Wranglers

TST-60Recommended

Fail the run when fewer tests run than before

Fail the test job when it runs no tests, or fewer tests than the main branch ran, unless the change deletes those tests.

By Agent Wranglers. Updated .

How to do it

A renamed folder or a wrong file pattern lets the runner find nothing, and some runners then report success. pytest exits with code 5 when it collects no tests, so keep that code a failure in CI. Turn off settings that pass an empty run, such as passWithNoTests in Vitest and Jest. Save the main branch's test count with each build, and compare every pull request with it. A lower count needs deleted tests in the same diff, as Delete tests with their feature describes. Count skipped tests apart, as Quarantine flaky tests with an owner asks.

Examples

Shell example

# CI: compare this run's test count with the main branch's, from JUnit XML reports
count() { grep -o '<testcase ' "$1" | wc -l; }
now=$(count reports/junit.xml)
before=$(count main-reports/junit.xml)
deleted=$(git diff --diff-filter=D --name-only origin/main...HEAD | grep -cE '(test|spec)')
if [ "$now" -eq 0 ] || { [ "$now" -lt "$before" ] && [ "$deleted" -eq 0 ]; }; then
  echo "FAIL: $now tests ran, main ran $before"; exit 1
fi

How to check

Point the runner at an empty folder, and the job fails. Remove a test folder from the runner settings without deleting its tests, and the count check fails.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/testing/test-quality/test-count-floor">Agent Wranglers testing playbook</a>. In code you write, add a comment next to it with the same link.