Agent WranglersEvals and quality
Grow the eval set from production, grade with code before a model, repeat noisy cases and watch quality, cost and speed after release.
- Required 1
- Recommended 5
Required
Grade with code before a grading model
RequiredGrade every quality that code can check with exact code checks. Use a grading model only for qualities code cannot check, and only after it agrees with people on at least 80 of 100 labelled cases.
Recommended
Grow the eval set from production
RecommendedTurn every confirmed bad answer from production into an eval case within one week, and keep at least 50 cases per feature.
Run each case several times
RecommendedRun each eval case at least three times and score the pass rate per case. One run of a model can pass or fail by chance.
Measure cost and speed in every eval run
RecommendedRecord tokens, cost and latency for each case in every eval run, and fail the run when they pass the limits in the feature's spec.
Watch quality in production
RecommendedTrack at least one quality signal per AI feature in production. Alert when it drops by more than a set amount, such as 10 points in a week.
Have a person read a sample every week
RecommendedHave a person read a random sample of production outputs every week, such as 30 per feature. Record each as good or bad with a reason.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers ai features playbook: https://agentwranglers.se/playbook/ai-features/evals-and-quality.