---
title: "Evals and quality · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/evals-and-quality
updated: 2026-10-20
licence: "All rights reserved"
---

# Evals and quality

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

> Grow the eval set from production, grade with code before a model, repeat noisy cases and watch quality, cost and speed after release.

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers ai features playbook: https://agentwranglers.se/playbook/ai-features/evals-and-quality`.

## Required

- [Grade with code before a grading model](https://agentwranglers.se/playbook/ai-features/evals-and-quality/code-graders-first): Grade every quality that code can check with exact code checks. Use a grading model only for qualities code cannot check, and only after it agrees with people on at least 80 of 100 labelled cases.

## Recommended

- [Grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production): Turn every confirmed bad answer from production into an eval case within one week, and keep at least 50 cases per feature.
- [Run each case several times](https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs): Run each eval case at least three times and score the pass rate per case. One run of a model can pass or fail by chance.
- [Measure cost and speed in every eval run](https://agentwranglers.se/playbook/ai-features/evals-and-quality/eval-cost-and-latency): Record tokens, cost and latency for each case in every eval run, and fail the run when they pass the limits in the feature's spec.
- [Watch quality in production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/production-quality-metrics): Track at least one quality signal per AI feature in production. Alert when it drops by more than a set amount, such as 10 points in a week.
- [Have a person read a sample every week](https://agentwranglers.se/playbook/ai-features/evals-and-quality/weekly-output-review): Have a person read a random sample of production outputs every week, such as 30 per feature. Record each as good or bad with a reason.

- Previous: [Retrieval](https://agentwranglers.se/playbook/ai-features/retrieval)
- Next: [Guardrails and human review](https://agentwranglers.se/playbook/ai-features/guardrails-and-human-review)
- [All chapters](https://agentwranglers.se/playbook/ai-features)