---
title: "Grow the eval set from production · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production
updated: 2026-10-20
licence: "All rights reserved"
---

# Grow the eval set from production

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

- Level: Recommended
- Chapter: [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)
- Rule: Turn every confirmed bad answer from production into an eval case within one week, and keep at least 50 cases per feature.
- How: Start with cases the team writes, covering common questions, edge cases, missing information and attempts to misuse the feature. Then add cases from user reports, thumbs-down feedback and support tickets. Remove personal data before a case enters the set, and store the set in version control. Tag each case with its source and the date. The base method is in [AI evals](https://agentwranglers.se/playbook/testing/non-functional-tests/llm-evals).
- Check: The eval file for each feature has at least 50 cases. Every bad-answer report closed in the last month links to an eval case by ID.

Example:

```jsonl
{"id": "ret-0142", "source": "user-report", "added": "2026-10-12", "input": "Can I return a helmet I wore once?", "expect": {"eligible": "no", "must_mention": ["hygiene"]}}
{"id": "ret-0143", "source": "team", "added": "2026-10-12", "input": "Return policy for order X-99?", "expect": {"eligible": "unknown"}}
```

```ts
const cases = readJsonLines('ai/evals/return-assistant.jsonl');
expect(cases.length).toBeGreaterThanOrEqual(50);
expect(new Set(cases.map(c => c.id)).size).toBe(cases.length);
```

```python
cases = [json.loads(line) for line in open("ai/evals/return-assistant.jsonl")]
assert len(cases) >= 50
assert len({c["id"] for c in cases}) == len(cases)
```

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers ai features playbook: https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production`.

- Next: [Grade with code before a grading model](https://agentwranglers.se/playbook/ai-features/evals-and-quality/code-graders-first)
- [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)