Rebuild fine-tuned models from stored data and compare before switching

How to do it

A fine-tune depends on its base model. When the provider retires that base, you train again from the stored data, as Pin the AI model version you depend on expects. Rebuild the training file before each run, so people who opted out or asked for erasure drop out. That is what Erase a person's data from every AI store requires. Keep every eval case out of the training file. A model that trained on its own test scores well and proves nothing. Compare the runs as Run each case several times describes.

Examples

TypeScript example

const evalInputs = new Set(evalCases.map(item => item.input.trim()));
const leaked = trainingRows.filter(row => evalInputs.has(row.input.trim()));
if (leaked.length > 0) throw new Error(`${leaked.length} eval cases are in the training file`);

Python example

eval_inputs = {case["input"].strip() for case in eval_cases}
leaked = [row for row in training_rows if row["input"].strip() in eval_inputs]
if leaked:
    raise SystemExit(f"{len(leaked)} eval cases are in the training file")

How to check

Every fine-tuned model ID in the gateway settings maps to a record with the data hash, base model and settings. A script finds 0 eval cases inside the training file. The last switch has an eval run with both scores, such as from 88% to 90%.

Sources

• Microsoft Foundry, fine-tuning overview

Related rules

• AIF-59 Pin the AI model version you depend on in AI features
• AIF-51 Erase a person's data from every AI store in AI features
• AIF-28 Run each case several times in AI features