---
title: "Have a person read a sample every week · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/evals-and-quality/weekly-output-review
updated: 2026-10-20
licence: "All rights reserved"
---

# Have a person read a sample every week

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

- Level: Recommended
- Chapter: [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)
- Rule: Have a person read a random sample of production outputs every week, such as 30 per feature. Record each as good or bad with a reason.
- How: Draw the sample at random, not only from complaints, because most bad answers get no report. Hide personal data in the review tool. Use a fixed form with a few reasons, such as wrong fact, missing source, wrong tone and should have refused. Add every bad one to the eval set, as in [grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production).
- Check: The review log has an entry for each AI feature in each of the last four weeks, with at least the set sample size.

Example:

```ts
const sample = await db.query("SELECT id, input_redacted, output_redacted FROM ai_outputs WHERE feature = $1 AND created_at > now() - interval '7 days' ORDER BY random() LIMIT 30", [feature]);
```

```python
sample = db.execute("SELECT id, input_redacted, output_redacted FROM ai_outputs WHERE feature = %s "
                    "AND created_at > now() - interval '7 days' ORDER BY random() LIMIT 30", (feature,)).fetchall()
```

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers ai features playbook: https://agentwranglers.se/playbook/ai-features/evals-and-quality/weekly-output-review`.

- Previous: [Watch quality in production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/production-quality-metrics)
- [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)