Have a person read a sample every week

How to do it

Draw the sample at random, not only from complaints, because most bad answers get no report. Hide personal data in the review tool. Use a fixed form with a few reasons, such as wrong fact, missing source, wrong tone and should have refused. Add every bad one to the eval set, as in grow the eval set from production.

Example

const sample = await db.query("SELECT id, input_redacted, output_redacted FROM ai_outputs WHERE feature = $1 AND created_at > now() - interval '7 days' ORDER BY random() LIMIT 30", [feature]);

sample = db.execute("SELECT id, input_redacted, output_redacted FROM ai_outputs WHERE feature = %s "
                    "AND created_at > now() - interval '7 days' ORDER BY random() LIMIT 30", (feature,)).fetchall()

How to check

The review log has an entry for each AI feature in each of the last four weeks, with at least the set sample size.