---
title: "Run each case several times · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs
updated: 2026-10-20
licence: "All rights reserved"
---

# Run each case several times

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

- Level: Recommended
- Chapter: [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)
- Rule: Run each eval case at least three times and score the pass rate per case. One run of a model can pass or fail by chance.
- How: Use the same settings as production, including temperature. Mark a case as passed when it passes in every run, or in a share you set, such as two of three. Report cases that pass only sometimes as unstable, and fix the prompt or the schema for them. Mark the cases that test the spec's worst failure as critical. A critical case must pass every run. Compare two versions only when the score gap is larger than the spread between repeated runs of one version.
- Check: The eval report shows each case's pass count out of the number of runs. CI fails when the overall score is under target or a case marked critical fails any run.

Example:

```ts
const RUNS_PER_CASE = 3;
const results = await Promise.all(cases.map(async c => ({ id: c.id, passes: (await Promise.all(Array.from({ length: RUNS_PER_CASE }, () => runCase(c)))).filter(failures => failures.length === 0).length })));
const unstable = results.filter(r => r.passes > 0 && r.passes < RUNS_PER_CASE);
```

```python
RUNS_PER_CASE = 3
results = [{"id": c["id"], "passes": sum(not run_case(c) for _ in range(RUNS_PER_CASE))} for c in cases]
unstable = [r for r in results if 0 < r["passes"] < RUNS_PER_CASE]
```

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers ai features playbook: https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs`.

- Previous: [Grade with code before a grading model](https://agentwranglers.se/playbook/ai-features/evals-and-quality/code-graders-first)
- Next: [Measure cost and speed in every eval run](https://agentwranglers.se/playbook/ai-features/evals-and-quality/eval-cost-and-latency)
- [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)