Choose the threshold from the cost of errors

How to do it

Most libraries turn a score into a yes at 0.5, which fits almost no business. Sweep thresholds on the validation set and compute the cost of each, as false positives times their cost plus false negatives times theirs. Store the chosen threshold with the model version in the model registry, not in serving code. Pick it again with every new model, because scores move between versions.

Examples

Python example

import numpy as np

def cheapest_threshold(y_true: np.ndarray, scores: np.ndarray, fp_cost: float, fn_cost: float) -> tuple[float, float]:
    best = (0.5, float("inf"))
    for t in np.linspace(0.01, 0.99, 99):
        predicted = scores >= t
        cost = fp_cost * np.sum(predicted & (y_true == 0)) + fn_cost * np.sum(~predicted & (y_true == 1))
        best = min(best, (float(t), float(cost)), key=lambda pair: pair[1])
    return best

Go example

func CheapestThreshold(labels []bool, scores []float64, fpCost, fnCost float64) (float64, float64) {
	bestT, bestCost := 0.5, math.Inf(1)
	for step := 1; step < 100; step++ {
		t, cost := float64(step)/100, 0.0
		for i, s := range scores {
			if s >= t && !labels[i] {
				cost += fpCost
			} else if s < t && labels[i] {
				cost += fnCost
			}
		}
		if cost < bestCost {
			bestT, bestCost = t, cost
		}
	}
	return bestT, bestCost
}

How to check

The registry entry of each classifier holds a threshold and the expected cost it gave on validation data. A test compares that cost with the cost at 0.5 and records both in the report.

Sources

• scikit-learn: tuning the decision threshold

Related rules

• MLO-1 Write down the decision and what each error costs in Machine learning
• MLO-21 Keep every model in a registry with versions and aliases in Machine learning
• MLO-15 Judging a model by accuracy alone in Machine learning