Agent WranglersEvaluation
Judge a model with metrics that fit the decision, with their uncertainty and with behaviour tests. Then let an automatic gate decide on each release.
- Recommended 4
- Avoid 1
Recommended
Check calibration when scores are shown as probabilities
MLO-16When a score is shown or used as a probability, check calibration on the test set. Recalibrate when any bin's observed rate differs from its mean score by more than the limit you set, such as 5 percentage points.
Report each score with its uncertainty
MLO-17Report every headline score with a 95% confidence interval. Call a new model better only when the interval of its difference from the current model lies above zero.
Test the model's behaviour case by case
MLO-18Keep a suite of behaviour tests next to the aggregate scores. It holds fixed cases with an expected answer and cases where a change must not move the score. It also holds cases where a change must move it one way.
Gate every release on an offline evaluation
MLO-19Promote a model only through an automatic gate. It scores the candidate and the current production model on the same frozen test set and blocks the release when any limit fails.
Avoid
Judging a model by accuracy alone
MLO-15Never report plain accuracy as the main score when one class holds less than 20% of the rows. Report precision, recall and a cost-based score for the class that matters.
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/machine-learning/evaluation">Agent Wranglers machine learning playbook</a>. In code you write, add a comment next to it with the same link.