Version the eval set and keep a held-out part

How to do it

A score from a changed set says nothing about a changed prompt. Store the set in version control with a version number and a hash of its cases, and print both in every eval report. When the set grows, run the current release on the new version too, so the pull request compares like with like. Mark held-out cases in the file. People tuning a prompt look only at the other cases, and the held-out score is read once per release. Cases leak into training and prompts in quiet ways, through copied examples, approved feedback and fine-tune files. Check for overlap by hash, as show the model worked examples does for prompts. Approved feedback follows approve user feedback before it trains or teaches the model.

Examples

TypeScript example

const MIN_HELD_OUT_SHARE = 0.2;

export function assertEvalSetClean(cases: EvalCase[], leakSources: Set<string>) {
  const heldOut = cases.filter(item => item.heldOut);
  if (heldOut.length / cases.length < MIN_HELD_OUT_SHARE) throw new Error('Too few held-out cases');
  const leaked = heldOut.filter(item => leakSources.has(sha256(item.input)));
  if (leaked.length) throw new Error(`Held-out cases leaked: ${leaked.map(item => item.id).join(', ')}`);
}

Python example

MIN_HELD_OUT_SHARE = 0.2

def assert_eval_set_clean(cases: list[EvalCase], leak_sources: set[str]) -> None:
    held_out = [c for c in cases if c.held_out]
    if len(held_out) / len(cases) < MIN_HELD_OUT_SHARE:
        raise ValueError("Too few held-out cases")
    leaked = [c.id for c in held_out if sha256(c.input) in leak_sources]
    if leaked:
        raise ValueError(f"Held-out cases leaked: {leaked}")

How to check

Every eval report shows the set version and hash, and CI refuses to compare two scores with different versions. A script finds no held-out case, by hash of its input, in any prompt file, fine-tune file or feedback store. The held-out share is at least 20%.

Sources

• NIST AI 100-1, AI Risk Management Framework

Related rules

• AIF-10 Show the model worked examples in AI features
• AIF-69 Approve user feedback before it trains or teaches the model in AI features