Agent Wranglers

Flaky and slow tests Recommended

Find flaky tests before they merge

Run new and changed tests many times in a row before merging, to catch flakiness while the change is fresh.

By Erik Bergstedt, Agent Wranglers. Updated .

How to do it

In the pull request build, run only the changed test files several times, for example with Playwright --repeat-each=10. A test that fails once in ten runs is not ready.

Example

npx playwright test tests/e2e/checkout.spec.ts --repeat-each=10

How to check

The pull request job repeats changed tests and fails if any run fails.

For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers testing playbook: https://agentwranglers.se/playbook/testing/flaky-and-slow-tests/repeat-new-tests.