For AI agents: llms.txt is a short guide that sends you to the rules your problem needs, and rules.json holds every rule of this playbook as JSON.

Applies when

The project it uses AI models.

How to do it

Copy cases from the eval set and change one detail, such as a name common in another country or another pronoun. Keep everything else the same. Compare the outcome field in code, such as the score, the rank or the yes or no, not the wording. Run each pair as many times as run each case several times sets, because one run can differ by chance. Use made-up people, not customer records, so the test needs no sensitive data. NIST AI 600-1 lists harmful bias among the risks generative AI makes worse. In the EU, AI Act Article 4a allows special category data for bias checks only when synthetic or other data cannot do the job. Such a feature may be high-risk, so run Find out if a feature is high-risk first. Record the feature and its test in the AI register.

Examples

Using these examples: they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

TypeScript example

const pairs = readJsonLines('ai/evals/applicant-ranking-pairs.jsonl');
const differing = (await Promise.all(pairs.map(async p => (await rank(p.a)).score !== (await rank(p.b)).score))).filter(Boolean).length;
expect(pairs.length).toBeGreaterThanOrEqual(20);
expect(differing / pairs.length).toBeLessThanOrEqual(1 / 20);

Python example

pairs = [json.loads(line) for line in open("ai/evals/applicant-ranking-pairs.jsonl")]
differing = sum(rank(p["a"]).score != rank(p["b"]).score for p in pairs)
assert len(pairs) >= 20
assert differing / len(pairs) <= 1 / 20

How to check

Each feature that ranks, scores or decides about people has at least 20 counterfactual pairs in its eval set. CI fails when more than 1 pair in 20 gives two different outcomes.

Sources

• NIST AI 600-1, Generative AI Profile, section 2.6 on harmful bias
• Regulation (EU) 2026/1744 adding AI Act Article 4a

Related rules

• Run each case several times in AI features
• Find out if a feature is high-risk in Compliance
• Register every AI feature with role and risk in Compliance