Grow the eval set from production

How to do it

Start with cases the team writes, covering common questions, edge cases, missing information and attempts to misuse the feature. Then add cases from user reports, thumbs-down feedback and support tickets. Remove personal data before a case enters the set, and store the set in version control. Tag each case with its source and the date. The base method is in AI evals.

Example

{"id": "ret-0142", "source": "user-report", "added": "2026-10-12", "input": "Can I return a helmet I wore once?", "expect": {"eligible": "no", "must_mention": ["hygiene"]}}
{"id": "ret-0143", "source": "team", "added": "2026-10-12", "input": "Return policy for order X-99?", "expect": {"eligible": "unknown"}}

const cases = readJsonLines('ai/evals/return-assistant.jsonl');
expect(cases.length).toBeGreaterThanOrEqual(50);
expect(new Set(cases.map(c => c.id)).size).toBe(cases.length);

cases = [json.loads(line) for line in open("ai/evals/return-assistant.jsonl")]
assert len(cases) >= 50
assert len({c["id"] for c in cases}) == len(cases)

How to check

The eval file for each feature has at least 50 cases. Every bad-answer report closed in the last month links to an eval case by ID.