---
title: "Rebuild fine-tuned models from stored data · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/evals-and-quality/reproducible-fine-tunes
id: AIF-62
level: Recommended
applies: ai-features
updated: 2026-10-22
tokens: 1039
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Rebuild fine-tuned models from stored data and compare before switching

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: AIF-62
- Level: Recommended
- Chapter: [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality.md)
- Rule: Store each fine-tuned model's training file hash, base model version and settings under one version ID. Before switching to a new fine-tune, run the full eval set on the old and the new model, and switch only when the score holds.
- Check: Every fine-tuned model ID in the gateway settings maps to a record with the data hash, base model and settings. A script finds 0 eval cases inside the training file. The last switch has an eval run with both scores, such as from 88% to 90%.
- Applies when: The project uses AI models.
- How: A fine-tune depends on its base model. When the provider retires that base, you train again from the stored data, as [Pin the AI model version you depend on](https://agentwranglers.se/playbook/ai-features/prompts-as-code/pin-model-versions.md) expects. Rebuild the training file before each run, so people who opted out or asked for erasure drop out. That is what [Erase a person's data from every AI store](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/erase-from-ai-stores.md) requires. Keep every eval case out of the training file. A model that trained on its own test scores well and proves nothing. Compare the runs as [Run each case several times](https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs.md) describes.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### TypeScript example

```ts
const evalInputs = new Set(evalCases.map(item => item.input.trim()));
const leaked = trainingRows.filter(row => evalInputs.has(row.input.trim()));
if (leaked.length > 0) throw new Error(`${leaked.length} eval cases are in the training file`);
```

### Python example

```python
eval_inputs = {case["input"].strip() for case in eval_cases}
leaked = [row for row in training_rows if row["input"].strip() in eval_inputs]
if leaked:
    raise SystemExit(f"{len(leaked)} eval cases are in the training file")
```

## Sources

- [Microsoft Foundry, fine-tuning overview](https://learn.microsoft.com/en-us/azure/foundry/fine-tuning/overview)

## Related rules

- AIF-59 [Pin the AI model version you depend on](https://agentwranglers.se/playbook/ai-features/prompts-as-code/pin-model-versions.md) in AI features
- AIF-51 [Erase a person's data from every AI store](https://agentwranglers.se/playbook/ai-features/logs-privacy-and-feedback/erase-from-ai-stores.md) in AI features
- AIF-28 [Run each case several times](https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs.md) in AI features
- MLO-19 [Gate every release on an offline evaluation](https://agentwranglers.se/playbook/machine-learning/evaluation/offline-eval-gate.md) in Machine learning
- MLO-20 [Record how every model was trained](https://agentwranglers.se/playbook/machine-learning/training-runs-and-the-registry/training-run-record.md) in Machine learning

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/ai-features/evals-and-quality/reproducible-fine-tunes">Agent Wranglers AI features playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Test that answers do not change with who asks](https://agentwranglers.se/playbook/ai-features/evals-and-quality/counterfactual-evals.md)
- [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality.md)