Agent Wranglers

MLO-17Recommended

Report each score with its uncertainty

Report every headline score with a 95% confidence interval. Call a new model better only when the interval of its difference from the current model lies above zero.

By Agent Wranglers. Updated .

How to do it

On 2,000 test rows, a difference of one point is often noise. Bootstrap the test set at least 1,000 times and score both models on the same resampled rows. Take the 2.5th and 97.5th percentiles of the difference. Small groups get wide intervals, which tells you how many rows group coverage needs. The same idea for language model evals is Run each case several times.

Examples

Python example

import numpy as np

BOOTSTRAP_ROUNDS = 1000

def difference_interval(y_true, current_pred, candidate_pred, metric, seed: int = 7) -> tuple[float, float]:
    rng = np.random.default_rng(seed)
    n = len(y_true)
    diffs = []
    for _ in range(BOOTSTRAP_ROUNDS):
        rows = rng.integers(0, n, n)
        diffs.append(metric(y_true[rows], candidate_pred[rows]) - metric(y_true[rows], current_pred[rows]))
    low, high = np.percentile(diffs, [2.5, 97.5])
    return float(low), float(high)

Go example

const bootstrapRounds = 1000

type Metric func(labels []bool, preds []float64) float64

func DifferenceInterval(labels []bool, current, candidate []float64, metric Metric, seed int64) (float64, float64) {
	rng := rand.New(rand.NewSource(seed))
	n := len(labels)
	diffs := make([]float64, bootstrapRounds)
	for round := range diffs {
		l, cur, cand := make([]bool, n), make([]float64, n), make([]float64, n)
		for i := range l {
			row := rng.Intn(n)
			l[i], cur[i], cand[i] = labels[row], current[row], candidate[row]
		}
		diffs[round] = metric(l, cand) - metric(l, cur)
	}
	sort.Float64s(diffs)
	return diffs[bootstrapRounds*25/1000], diffs[bootstrapRounds*975/1000]
}

How to check

Every score in the evaluation report has a lower and an upper bound. The gate rejects a candidate whose difference interval includes zero, unless the release is marked as a change that only has to hold the score.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/machine-learning/evaluation/score-intervals">Agent Wranglers machine learning playbook</a>. In code you write, add a comment next to it with the same link.