Measure every metric for each group of people

How to do it

A good total can hide a model that fails one group, and unequal treatment can break discrimination law. List the groups from the model's purpose and the protected grounds in the law of each market. Pick the measure from the decision. Equal recall suits a model that grants something, and equal false positive rates suit one that flags people. Write the limit, such as 5 percentage points, in the release gate. Repeat the check on live outcomes in Measure live quality once the outcomes arrive. A high-risk system must examine its data for such bias under AI Act Article 10(2). Where you need sensitive traits to measure this, follow Hold sensitive traits for bias checks apart from training.

Examples

Python example

import pandas as pd

GAP_LIMIT = 0.05

def recall_by_group(df: pd.DataFrame, group: str) -> pd.Series:
    positives = df[df["label"] == 1]
    return positives.groupby(group)["predicted"].mean()

def group_gap_failures(df: pd.DataFrame, groups: list[str]) -> list[str]:
    failures = []
    for group in groups:
        recall = recall_by_group(df, group)
        if recall.max() - recall.min() > GAP_LIMIT:
            failures.append(f"{group}: recall gap {recall.max() - recall.min():.3f}")
    return failures

TypeScript example

const GAP_LIMIT = 0.05;
type Row = { label: boolean; predicted: boolean; groups: Record<string, string> };

export function recallGap(rows: Row[], group: string): number {
  const byGroup = new Map<string, { hit: number; total: number }>();
  for (const row of rows.filter(r => r.label)) {
    const key = row.groups[group];
    const entry = byGroup.get(key) ?? { hit: 0, total: 0 };
    entry.total += 1; entry.hit += row.predicted ? 1 : 0;
    byGroup.set(key, entry);
  }
  const recalls = [...byGroup.values()].map(e => e.hit / e.total);
  return Math.max(...recalls) - Math.min(...recalls);
}

export const groupGapFailures = (rows: Row[], groups: string[]) => groups.filter(g => recallGap(rows, g) > GAP_LIMIT);

How to check

Every evaluation report of a model that scores people has a table per listed group with its count, metric, error rates and interval. The gate fails when a gap passes the limit or a group has no row.

Sources

• NIST AI 100-1, AI RMF
• AI Act Article 10
• Google ML Crash Course on bias

Related rules

• MLO-8 Check that every group you serve is in the training data in Machine learning
• MLO-16 Check calibration when scores are shown as probabilities in Machine learning
• MLO-19 Gate every release on an offline evaluation in Machine learning
• MLO-31 Measure live quality once the outcomes arrive in Machine learning
• MLO-36 Hold sensitive traits for bias checks apart from training in Machine learning
• MLO-35 Features that stand in for protected traits in Machine learning