---
title: "Check every new snapshot against the last · Agent Wranglers"
url: https://agentwranglers.se/playbook/machine-learning/training-data/validate-training-snapshot
id: MLO-39
level: Recommended
applies: ai-features
updated: 2026-10-22
build: a2a6bdfed830
tokens: 1242
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Check every new snapshot against the last one before training

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Training data](https://agentwranglers.se/playbook/machine-learning/training-data.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: MLO-39
- Level: Recommended
- Chapter: [Training data](https://agentwranglers.se/playbook/machine-learning/training-data.md)
- Rule: Before a training run, compare the new snapshot with the schema and statistics of the snapshot behind the production model. Stop on a missing or new column, a row count change over the set limit or a label share that moved over its limit.
- Check: A test runs the pipeline on a snapshot with one column dropped, and on one with a label share moved by 10 points. Both runs stop before training and name the reason.
- Applies when: The project uses AI models.
- How: A broken export, a changed upstream field or a batch of poisoned rows trains a model that still passes many tests. Google's MLOps guide puts this check before training and stops the pipeline on schema problems. Save the schema, row count and label shares with every snapshot. Set the limits per model, such as a 20% change in rows or 5 points in any label's share. Also flag a source or account that supplies far more rows than before, because injected data often shows up that way. A stopped run leaves the current model serving, as [Retrain on written triggers](https://agentwranglers.se/playbook/machine-learning/monitoring-and-retraining/retraining-triggers.md) says. Serving input gets the same kind of check in [Check every input against what the model was trained on](https://agentwranglers.se/playbook/machine-learning/serving-and-release/serving-input-schema.md).

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### Python example

```python
MAX_ROW_CHANGE = 0.20
MAX_LABEL_SHIFT = 0.05

def snapshot_problems(new: SnapshotStats, last: SnapshotStats) -> list[str]:
    problems = [f"column {c} missing" for c in last.columns - new.columns]
    problems += [f"column {c} is new" for c in new.columns - last.columns]
    if abs(new.rows - last.rows) / last.rows > MAX_ROW_CHANGE:
        problems.append(f"rows changed from {last.rows} to {new.rows}")
    for label, share in last.label_shares.items():
        if abs(new.label_shares.get(label, 0.0) - share) > MAX_LABEL_SHIFT:
            problems.append(f"label {label} share moved from {share:.2f}")
    return problems
```

### TypeScript example

```ts
const MAX_ROW_CHANGE = 0.2;
const MAX_LABEL_SHIFT = 0.05;

export function snapshotProblems(next: SnapshotStats, last: SnapshotStats): string[] {
  const problems = [
    ...last.columns.filter(c => !next.columns.includes(c)).map(c => `column ${c} missing`),
    ...next.columns.filter(c => !last.columns.includes(c)).map(c => `column ${c} is new`),
  ];
  if (Math.abs(next.rows - last.rows) / last.rows > MAX_ROW_CHANGE) problems.push(`rows changed from ${last.rows} to ${next.rows}`);
  for (const [label, share] of Object.entries(last.labelShares)) {
    if (Math.abs((next.labelShares[label] ?? 0) - share) > MAX_LABEL_SHIFT) problems.push(`label ${label} share moved from ${share.toFixed(2)}`);
  }
  return problems;
}
```

## Sources

- [Google Cloud, MLOps continuous delivery and automation pipelines](https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning)
- [OWASP Secure AI Model Ops Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Secure_AI_Model_Ops_Cheat_Sheet.html)

## Related rules

- MLO-32 [Retrain on written triggers](https://agentwranglers.se/playbook/machine-learning/monitoring-and-retraining/retraining-triggers.md) in Machine learning
- MLO-24 [Check every input against what the model was trained on](https://agentwranglers.se/playbook/machine-learning/serving-and-release/serving-input-schema.md) in Machine learning

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/machine-learning/training-data/validate-training-snapshot">Agent Wranglers machine learning playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Check that every group you serve is in the training data](https://agentwranglers.se/playbook/machine-learning/training-data/group-coverage.md)
- Next: [Limit what any one source can put into training data](https://agentwranglers.se/playbook/machine-learning/training-data/poisoning-checks.md)
- [Training data](https://agentwranglers.se/playbook/machine-learning/training-data.md)