Test that the model does not give away its training records

How to do it

A model can remember the people in its training data. Someone who queries it may learn that a person was in the set, such as a patient in a medical model. They may even pull out the records. The EDPB says a model trained on personal data is anonymous only when both risks are insignificant. It names tests against membership inference, extraction, regurgitation, model inversion and reconstruction. Run an attack that tells training rows from held-out rows by the model's confidence, and record its AUC. The default margin is an AUC of 0.55, where 0.5 is a coin toss. Set a lower margin for sensitive data, such as health records. A model that fails keeps every personal data duty, such as a legal basis, access and erasure. Lower the risk with regularisation, fewer rare features or differential privacy, then test again. Record the result in the model card.

Examples

Python example

import numpy as np
from sklearn.metrics import roc_auc_score

MAX_ATTACK_AUC = 0.55

def membership_attack_auc(model, train_x, train_y, holdout_x, holdout_y) -> float:
    def confidence(x, y):
        return model.predict_proba(x)[np.arange(len(y)), y]
    scores = np.concatenate([confidence(train_x, train_y), confidence(holdout_x, holdout_y)])
    is_member = np.concatenate([np.ones(len(train_y)), np.zeros(len(holdout_y))])
    return roc_auc_score(is_member, scores)

TypeScript example

export const MAX_ATTACK_AUC = 0.55;

export function attackAuc(memberScores: number[], holdoutScores: number[]): number {
  let wins = 0;
  for (const member of memberScores) {
    for (const holdout of holdoutScores) wins += member > holdout ? 1 : member === holdout ? 0.5 : 0;
  }
  return wins / (memberScores.length * holdoutScores.length);
}

How to check

The release gate runs the membership inference test on every model trained on personal data. It fails when the attack AUC is above 0.55 or above the lower margin in the model card. A test with a deliberately overfit model expects that failure.

Sources

• EDPB Opinion 28/2024 on AI models and personal data
• OWASP Secure AI Model Ops Cheat Sheet

Related rules

• MLO-40 Limit what the prediction endpoint gives away in Machine learning
• MLO-22 Write a model card for every model in production in Machine learning
• MLO-44 Remove erased people from every model trained on their data in Machine learning