Watch quality in production

How to do it

Choose signals that follow from what users do. Examples are the share of drafts sent without edits, thumbs-down rate, hand-offs to a person, retries and validation failures. Chart them per prompt version and model version, so a drop points to its cause. Alert the feature's owner, not the on-call engineer, unless the feature is down. Alert setup is in the operations playbook.

Example

SELECT prompt_version, avg(CASE WHEN feedback = 'down' THEN 1.0 ELSE 0 END) AS thumbs_down_rate, count(*)
FROM ai_outputs WHERE feature = 'return-assistant' AND created_at > now() - interval '7 days'
GROUP BY prompt_version;

How to check

The dashboard shows a quality signal per AI feature, split by prompt version. An alert rule exists for each and fires in a test with a simulated drop.