---
title: "Watch quality in production · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/evals-and-quality/production-quality-metrics
updated: 2026-10-20
licence: "All rights reserved"
---

# Watch quality in production

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

- Level: Recommended
- Chapter: [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)
- Rule: Track at least one quality signal per AI feature in production. Alert when it drops by more than a set amount, such as 10 points in a week.
- How: Choose signals that follow from what users do. Examples are the share of drafts sent without edits, thumbs-down rate, hand-offs to a person, retries and validation failures. Chart them per prompt version and model version, so a drop points to its cause. Alert the feature's owner, not the on-call engineer, unless the feature is down. Alert setup is in the [operations playbook](https://agentwranglers.se/playbook/operations/alerts-and-on-call).
- Check: The dashboard shows a quality signal per AI feature, split by prompt version. An alert rule exists for each and fires in a test with a simulated drop.

Example:

```sql
SELECT prompt_version, avg(CASE WHEN feedback = 'down' THEN 1.0 ELSE 0 END) AS thumbs_down_rate, count(*)
FROM ai_outputs WHERE feature = 'return-assistant' AND created_at > now() - interval '7 days'
GROUP BY prompt_version;
```

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers ai features playbook: https://agentwranglers.se/playbook/ai-features/evals-and-quality/production-quality-metrics`.

- Previous: [Measure cost and speed in every eval run](https://agentwranglers.se/playbook/ai-features/evals-and-quality/eval-cost-and-latency)
- Next: [Have a person read a sample every week](https://agentwranglers.se/playbook/ai-features/evals-and-quality/weekly-output-review)
- [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)