Agent WranglersFlaky and slow tests
A test that fails at random teaches people to ignore red builds. Find flaky tests early, quarantine them with an owner, and keep the suite fast.
- Required 1
- Recommended 3
Required
Quarantine flaky tests with an owner
RequiredMove a test that fails at random out of the blocking suite at once. Give it an owner and a deadline, then fix or delete it.
Recommended
Find hidden order dependencies
RecommendedRun tests in a random order, so tests that depend on each other fail early.
Find flaky tests before they merge
RecommendedRun new and changed tests many times in a row before merging, to catch flakiness while the change is fresh.
Keep the suite within a time budget
RecommendedSet a time limit for the whole suite, such as ten minutes, and for each kind of test. Treat going over it as a failure.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers testing playbook: https://agentwranglers.se/playbook/testing/flaky-and-slow-tests.