---
title: "Test a new model on live outcomes with a · Agent Wranglers"
url: https://agentwranglers.se/playbook/machine-learning/serving-and-release/model-ab-test
id: MLO-46
level: Recommended
applies: ai-features
updated: 2026-10-27
build: a2a6bdfed830
tokens: 996
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Test a new model on live outcomes with a randomised split

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Serving and release](https://agentwranglers.se/playbook/machine-learning/serving-and-release.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: MLO-46
- Level: Recommended
- Chapter: [Serving and release](https://agentwranglers.se/playbook/machine-learning/serving-and-release.md)
- Rule: Before a new model replaces one that drives a business outcome, split users at random between the two. Write the success metric, the sample size and the stop date first, and ship only when the result clears the plan.
- Check: Each model release that changed a business metric links an experiment plan dated before its start. The prediction log shows each user under one model version for the whole test.
- Applies when: The project uses AI models.
- How: Offline scores and a canary show that a model works. Only a randomised split shows that it helps, such as more sales or fewer missed fraud cases. Assign by user, not by request, so one person sees one model. Plan the experiment the way [write the experiment plan before it starts](https://agentwranglers.se/playbook/analytics/experiments/experiment-plan.md) says, with guardrail metrics such as complaints. Read the result once, at the stop date, with the interval from [report each score with its uncertainty](https://agentwranglers.se/playbook/machine-learning/evaluation/score-intervals.md). Run it after the [canary](https://agentwranglers.se/playbook/machine-learning/serving-and-release/model-canary.md) passes.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### Python example

```python
def model_for(user_id: str, experiment: Experiment) -> str:
    bucket = int(hashlib.sha256(f"{experiment.id}:{user_id}".encode()).hexdigest(), 16) % 100
    return experiment.candidate if bucket < experiment.candidate_percent else experiment.control
```

### TypeScript example

```ts
export function modelFor(userId: string, experiment: Experiment): string {
  const bucket = Number(BigInt(`0x${sha256(`${experiment.id}:${userId}`)}`) % 100n);
  return bucket < experiment.candidatePercent ? experiment.candidate : experiment.control;
}
```

## Sources

- [Google Cloud, MLOps continuous delivery and automation pipelines](https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning)

## Related rules

- ANA-11 [Write the experiment plan before it starts](https://agentwranglers.se/playbook/analytics/experiments/experiment-plan.md) in Product analytics and experiments
- MLO-17 [Report each score with its uncertainty](https://agentwranglers.se/playbook/machine-learning/evaluation/score-intervals.md) in Machine learning
- MLO-27 [Release models through a canary that compares predictions](https://agentwranglers.se/playbook/machine-learning/serving-and-release/model-canary.md) in Machine learning

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/machine-learning/serving-and-release/model-ab-test">Agent Wranglers machine learning playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Limit what the prediction endpoint gives away](https://agentwranglers.se/playbook/machine-learning/serving-and-release/limit-prediction-output.md)
- Next: [Retire models on purpose](https://agentwranglers.se/playbook/machine-learning/serving-and-release/model-retirement.md)
- [Serving and release](https://agentwranglers.se/playbook/machine-learning/serving-and-release.md)