Run each case several times

How to do it

Use the same settings as production, including temperature. Mark a case as passed when it passes in every run, or in a share you set, such as two of three. Report cases that pass only sometimes as unstable, and fix the prompt or the schema for them. Mark the cases that test the spec's worst failure as critical. A critical case must pass every run. Compare two versions only when the score gap is larger than the spread between repeated runs of one version.

Example

const RUNS_PER_CASE = 3;
const results = await Promise.all(cases.map(async c => ({ id: c.id, passes: (await Promise.all(Array.from({ length: RUNS_PER_CASE }, () => runCase(c)))).filter(failures => failures.length === 0).length })));
const unstable = results.filter(r => r.passes > 0 && r.passes < RUNS_PER_CASE);

RUNS_PER_CASE = 3
results = [{"id": c["id"], "passes": sum(not run_case(c) for _ in range(RUNS_PER_CASE))} for c in cases]
unstable = [r for r in results if 0 < r["passes"] < RUNS_PER_CASE]

How to check

The eval report shows each case's pass count out of the number of runs. CI fails when the overall score is under target or a case marked critical fails any run.