Collect feedback on every answer

How to do it

Keep the control small and visible. On thumbs down, offer a few fixed reasons, such as wrong, unhelpful, unsafe and other, plus a free-text box. Store the feedback with the call ID, so you can see the exact prompt version, model and sources. Also count silent signals, such as an edited draft or a repeated question. Each week, sort new thumbs-down items into bug, missing document, prompt problem or not a problem. Add confirmed bugs to the eval set.

Example

app.post('/api/ai-feedback', async (req, res) => {
  const { callId, rating, reason } = FeedbackInput.parse(req.body);
  await db.aiFeedback.insert({ callId, rating, reason, userId: req.user.id });
  res.status(204).end();
});

@app.post("/api/ai-feedback", status_code=204)
def ai_feedback(body: FeedbackInput, user: User = Depends(current_user)) -> None:
    db.ai_feedback.insert(call_id=body.call_id, rating=body.rating, reason=body.reason, user_id=user.id)

<div class="ai-feedback" data-call-id="c_81f2">
  <button type="button" value="up">Helpful</button>
  <button type="button" value="down">Not helpful</button>
</div>

How to check

A browser test clicks thumbs down on an answer and picks a reason. A row with that call ID and reason appears. The triage log has an entry for each of the last four weeks.