---
title: "Version the eval set and keep a held-out · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/evals-and-quality/eval-set-versions
id: AIF-76
level: Recommended
applies: ai-features
updated: 2026-10-27
build: a2a6bdfed830
tokens: 1076
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Version the eval set and keep a held-out part

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: AIF-76
- Level: Recommended
- Chapter: [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality.md)
- Rule: Give each eval set a version that rises on every change, and compare scores only within one version. Keep at least 20% of cases held out, never shown in prompts, examples, fine-tune data or feedback stores.
- Check: Every eval report shows the set version and hash, and CI refuses to compare two scores with different versions. A script finds no held-out case, by hash of its input, in any prompt file, fine-tune file or feedback store. The held-out share is at least 20%.
- Applies when: The project uses AI models.
- How: A score from a changed set says nothing about a changed prompt. Store the set in version control with a version number and a hash of its cases, and print both in every eval report. When the set grows, run the current release on the new version too, so the pull request compares like with like. Mark held-out cases in the file. People tuning a prompt look only at the other cases, and the held-out score is read once per release. Cases leak into training and prompts in quiet ways, through copied examples, approved feedback and fine-tune files. Check for overlap by hash, as [show the model worked examples](https://agentwranglers.se/playbook/ai-features/prompts-as-code/worked-examples.md) does for prompts. Approved feedback follows [approve user feedback before it trains or teaches the model](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/approve-feedback-before-training.md).

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### TypeScript example

```ts
const MIN_HELD_OUT_SHARE = 0.2;

export function assertEvalSetClean(cases: EvalCase[], leakSources: Set<string>) {
  const heldOut = cases.filter(item => item.heldOut);
  if (heldOut.length / cases.length < MIN_HELD_OUT_SHARE) throw new Error('Too few held-out cases');
  const leaked = heldOut.filter(item => leakSources.has(sha256(item.input)));
  if (leaked.length) throw new Error(`Held-out cases leaked: ${leaked.map(item => item.id).join(', ')}`);
}
```

### Python example

```python
MIN_HELD_OUT_SHARE = 0.2

def assert_eval_set_clean(cases: list[EvalCase], leak_sources: set[str]) -> None:
    held_out = [c for c in cases if c.held_out]
    if len(held_out) / len(cases) < MIN_HELD_OUT_SHARE:
        raise ValueError("Too few held-out cases")
    leaked = [c.id for c in held_out if sha256(c.input) in leak_sources]
    if leaked:
        raise ValueError(f"Held-out cases leaked: {leaked}")
```

## Sources

- [NIST AI 100-1, AI Risk Management Framework](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf)

## Related rules

- AIF-10 [Show the model worked examples](https://agentwranglers.se/playbook/ai-features/prompts-as-code/worked-examples.md) in AI features
- AIF-69 [Approve user feedback before it trains or teaches the model](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/approve-feedback-before-training.md) in AI features

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/ai-features/evals-and-quality/eval-set-versions">Agent Wranglers AI features playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production.md)
- Next: [Put attack cases in every eval run](https://agentwranglers.se/playbook/ai-features/evals-and-quality/attack-cases-in-evals.md)
- [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality.md)