---
title: "Fail the run when fewer tests run than · Agent Wranglers"
url: https://agentwranglers.se/playbook/testing/test-quality/test-count-floor
id: TST-60
level: Recommended
updated: 2026-10-22
build: a2a6bdfed830
tokens: 834
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Fail the run when fewer tests run than before

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Test quality](https://agentwranglers.se/playbook/testing/test-quality.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: TST-60
- Level: Recommended
- Chapter: [Test quality](https://agentwranglers.se/playbook/testing/test-quality.md)
- Rule: Fail the test job when it runs no tests, or fewer tests than the main branch ran, unless the change deletes those tests.
- Check: Point the runner at an empty folder, and the job fails. Remove a test folder from the runner settings without deleting its tests, and the count check fails.
- How: A renamed folder or a wrong file pattern lets the runner find nothing, and some runners then report success. pytest exits with code 5 when it collects no tests, so keep that code a failure in CI. Turn off settings that pass an empty run, such as `passWithNoTests` in Vitest and Jest. Save the main branch's test count with each build, and compare every pull request with it. A lower count needs deleted tests in the same diff, as [Delete tests with their feature](https://agentwranglers.se/playbook/architecture/structure/delete-tests-with-feature.md) describes. Count skipped tests apart, as [Quarantine flaky tests with an owner](https://agentwranglers.se/playbook/testing/flaky-and-slow-tests/quarantine-flaky-tests.md) asks.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### Shell example

```sh
# CI: compare this run's test count with the main branch's, from JUnit XML reports
count() { grep -o '<testcase ' "$1" | wc -l; }
now=$(count reports/junit.xml)
before=$(count main-reports/junit.xml)
deleted=$(git diff --diff-filter=D --name-only origin/main...HEAD | grep -cE '(test|spec)')
if [ "$now" -eq 0 ] || { [ "$now" -lt "$before" ] && [ "$deleted" -eq 0 ]; }; then
  echo "FAIL: $now tests ran, main ran $before"; exit 1
fi
```

## Sources

- [pytest: Exit codes](https://docs.pytest.org/en/stable/reference/exit-codes.html)

## Related rules

- ARC-62 [Delete tests with their feature](https://agentwranglers.se/playbook/architecture/structure/delete-tests-with-feature.md) in Clean code and architecture
- TST-28 [Quarantine flaky tests with an owner](https://agentwranglers.se/playbook/testing/flaky-and-slow-tests/quarantine-flaky-tests.md) in Testing

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/testing/test-quality/test-count-floor">Agent Wranglers testing playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Turn deprecation warnings into test failures](https://agentwranglers.se/playbook/testing/test-quality/deprecations-fail-tests.md)
- Next: [Make sure tests an agent wrote do not just repeat the code](https://agentwranglers.se/playbook/testing/test-quality/tautological-tests.md)
- [Test quality](https://agentwranglers.se/playbook/testing/test-quality.md)