---
title: "Test that the model does not give away its · Agent Wranglers"
url: https://agentwranglers.se/playbook/machine-learning/people-affected-by-the-model/training-data-extraction-test
id: MLO-41
level: Required
applies: ai-features, eu-users
risk: data
updated: 2026-10-27
build: a2a6bdfed830
tokens: 1237
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Test that the model does not give away its training records

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [People affected by the model](https://agentwranglers.se/playbook/machine-learning/people-affected-by-the-model.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: MLO-41
- Level: Required
- Chapter: [People affected by the model](https://agentwranglers.se/playbook/machine-learning/people-affected-by-the-model.md)
- Rule: Before you ship, share or call anonymous a model trained on personal data, run a membership inference test against it. Treat the model as personal data when the attack's AUC is above 0.55, or above a lower margin set in its model card.
- Check: The release gate runs the membership inference test on every model trained on personal data. It fails when the attack AUC is above 0.55 or above the lower margin in the model card. A test with a deliberately overfit model expects that failure.
- Risk: Data leaked or lost
- Applies when: The project uses AI models, or has users in the EU.
- How: A model can remember the people in its training data. Someone who queries it may learn that a person was in the set, such as a patient in a medical model. They may even pull out the records. The EDPB says a model trained on personal data is anonymous only when both risks are insignificant. It names tests against membership inference, extraction, regurgitation, model inversion and reconstruction. Run an attack that tells training rows from held-out rows by the model's confidence, and record its AUC. The default margin is an AUC of 0.55, where 0.5 is a coin toss. Set a lower margin for sensitive data, such as health records. A model that fails keeps every personal data duty, such as a legal basis, access and erasure. Lower the risk with regularisation, fewer rare features or differential privacy, then test again. Record the result in the [model card](https://agentwranglers.se/playbook/machine-learning/training-runs-and-the-registry/model-cards.md).

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### Python example

```python
import numpy as np
from sklearn.metrics import roc_auc_score

MAX_ATTACK_AUC = 0.55

def membership_attack_auc(model, train_x, train_y, holdout_x, holdout_y) -> float:
    def confidence(x, y):
        return model.predict_proba(x)[np.arange(len(y)), y]
    scores = np.concatenate([confidence(train_x, train_y), confidence(holdout_x, holdout_y)])
    is_member = np.concatenate([np.ones(len(train_y)), np.zeros(len(holdout_y))])
    return roc_auc_score(is_member, scores)
```

### TypeScript example

```ts
export const MAX_ATTACK_AUC = 0.55;

export function attackAuc(memberScores: number[], holdoutScores: number[]): number {
  let wins = 0;
  for (const member of memberScores) {
    for (const holdout of holdoutScores) wins += member > holdout ? 1 : member === holdout ? 0.5 : 0;
  }
  return wins / (memberScores.length * holdoutScores.length);
}
```

## Sources

- [EDPB Opinion 28/2024 on AI models and personal data](https://www.edpb.europa.eu/system/files/2024-12/edpb_opinion_202428_ai-models_en.pdf)
- [OWASP Secure AI Model Ops Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Secure_AI_Model_Ops_Cheat_Sheet.html)

## Related rules

- MLO-40 [Limit what the prediction endpoint gives away](https://agentwranglers.se/playbook/machine-learning/serving-and-release/limit-prediction-output.md) in Machine learning
- MLO-22 [Write a model card for every model in production](https://agentwranglers.se/playbook/machine-learning/training-runs-and-the-registry/model-cards.md) in Machine learning
- MLO-44 [Remove erased people from every model trained on their data](https://agentwranglers.se/playbook/machine-learning/people-affected-by-the-model/erasure-from-models.md) in Machine learning

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/machine-learning/people-affected-by-the-model/training-data-extraction-test">Agent Wranglers machine learning playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Keep a record of every decision about a person](https://agentwranglers.se/playbook/machine-learning/people-affected-by-the-model/decision-records.md)
- Next: [Remove erased people from every model trained on their data](https://agentwranglers.se/playbook/machine-learning/people-affected-by-the-model/erasure-from-models.md)
- [People affected by the model](https://agentwranglers.se/playbook/machine-learning/people-affected-by-the-model.md)