---
title: "Test that answers do not change with who · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/evals-and-quality/counterfactual-evals
updated: 2026-10-21
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Test that answers do not change with who asks

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers). Updated 2026-10-21.

For AI agents: [llms.txt](https://agentwranglers.se/llms.txt) is a short guide that sends you to the rules your problem needs, and [rules.json](https://agentwranglers.se/playbook/ai-features/rules.json) holds every rule of this playbook as JSON.

- Level: Recommended
- Chapter: [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)
- Rule: For features that rank, score or decide about people, add eval pairs that differ only in one personal detail, such as a name or age. Fail the run when outcomes differ.
- Applies when: The project it uses AI models.
- How: Copy cases from the eval set and change one detail, such as a name common in another country or another pronoun. Keep everything else the same. Compare the outcome field in code, such as the score, the rank or the yes or no, not the wording. Run each pair as many times as [run each case several times](https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs) sets, because one run can differ by chance. Use made-up people, not customer records, so the test needs no sensitive data. NIST AI 600-1 lists harmful bias among the risks generative AI makes worse. In the EU, AI Act Article 4a allows special category data for bias checks only when synthetic or other data cannot do the job. Such a feature may be high-risk, so run [Find out if a feature is high-risk](https://agentwranglers.se/playbook/compliance/ai-act/ai-high-risk-check) first. Record the feature and its test in the [AI register](https://agentwranglers.se/playbook/compliance/ai-act/ai-register).
- Check: Each feature that ranks, scores or decides about people has at least 20 counterfactual pairs in its eval set. CI fails when more than 1 pair in 20 gives two different outcomes.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### TypeScript example

```ts
const pairs = readJsonLines('ai/evals/applicant-ranking-pairs.jsonl');
const differing = (await Promise.all(pairs.map(async p => (await rank(p.a)).score !== (await rank(p.b)).score))).filter(Boolean).length;
expect(pairs.length).toBeGreaterThanOrEqual(20);
expect(differing / pairs.length).toBeLessThanOrEqual(1 / 20);
```

### Python example

```python
pairs = [json.loads(line) for line in open("ai/evals/applicant-ranking-pairs.jsonl")]
differing = sum(rank(p["a"]).score != rank(p["b"]).score for p in pairs)
assert len(pairs) >= 20
assert differing / len(pairs) <= 1 / 20
```

## Sources

- [NIST AI 600-1, Generative AI Profile, section 2.6 on harmful bias](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf)
- [Regulation (EU) 2026/1744 adding AI Act Article 4a](https://eur-lex.europa.eu/eli/reg/2026/1744/oj)

## Related rules

- [Run each case several times](https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs) in AI features
- [Find out if a feature is high-risk](https://agentwranglers.se/playbook/compliance/ai-act/ai-high-risk-check) in Compliance
- [Register every AI feature with role and risk](https://agentwranglers.se/playbook/compliance/ai-act/ai-register) in Compliance

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/ai-features/evals-and-quality/counterfactual-evals">Agent Wranglers AI features playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Have a person read a sample every week](https://agentwranglers.se/playbook/ai-features/evals-and-quality/weekly-output-review)
- [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)