Agent Wranglers

MLO-30Recommended

Alert when the prediction mix shifts

Track the share of each decision and the mean score per hour or per day. Alert when either leaves the range seen in the 28 days after release.

By Agent Wranglers. Updated .

How to do it

The output is the cheapest signal, because it needs no labels. A fraud model that flags 1% of payments and suddenly flags 6% has a problem, even when every input looks normal. Store the range at release, from the shadow and canary steps and the first weeks. Split the check by the groups from group coverage, because a shift in one country can hide in the total.

Examples

SQL example

SELECT date_trunc('hour', at) AS hour, model_version,
       avg(CASE WHEN decision = 'review' THEN 1.0 ELSE 0 END) AS review_share,
       avg(score) AS mean_score
FROM prediction_log
WHERE at > now() - interval '1 day'
GROUP BY 1, 2;
-- Alert when review_share or mean_score is outside the stored release range

How to check

Replay a day of logged requests through a model whose threshold was set too low on purpose. The positive rate leaves the stored range and the alert fires within one check interval.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/machine-learning/monitoring-and-retraining/prediction-drift-alerts">Agent Wranglers machine learning playbook</a>. In code you write, add a comment next to it with the same link.