Agent WranglersMLO-15Avoid
Judging a model by accuracy alone
Never report plain accuracy as the main score when one class holds less than 20% of the rows. Report precision, recall and a cost-based score for the class that matters.
How to do it
When 2% of payments are fraud, a model that says "never fraud" is 98% accurate and catches nothing. Report the confusion matrix at the chosen threshold, recall and precision for the rare class, and the area under the precision-recall curve. Add balanced accuracy when a single number is needed. The cost from Choose the threshold from the cost of errors is the score that decides.
How to check
A script reads each evaluation report. It fails when a class under 20% of rows has no precision and recall, or when accuracy is the only headline score.
Sources
Related rules
- MLO-3 Choose the threshold from the cost of errors in Machine learning
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/machine-learning/evaluation/accuracy-alone">Agent Wranglers machine learning playbook</a>. In code you write, add a comment next to it with the same link.