---
title: "Stop the analysis on a sample ratio · Agent Wranglers"
url: https://agentwranglers.se/playbook/analytics/experiments/sample-ratio-mismatch
id: ANA-13
level: Recommended
updated: 2026-10-27
build: a2a6bdfed830
tokens: 849
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Stop the analysis on a sample ratio mismatch

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Experiments](https://agentwranglers.se/playbook/analytics/experiments.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: ANA-13
- Level: Recommended
- Chapter: [Experiments](https://agentwranglers.se/playbook/analytics/experiments.md)
- Rule: Before reading any result, compare the number of exposed users in each variant with the planned split. When a chi-square test gives p below 0.001, treat the results as void and find the cause.
- Check: A unit test feeds the check 50,000 and 48,500 for a planned 50/50 split, and the analysis stops with a sample ratio mismatch. With 50,000 and 49,800, it goes on.
- How: A 50/50 test that ends at 50,000 and 48,500 users has lost people in one variant. A redirect that drops tracking, a bot filter, or a crash in the new page all do this. Whatever caused it also changes who is left, so the measured effect cannot be trusted. Run the check in the analysis script before any metric, per day and for the whole test, so a break shows the day it began. Common causes are exposure logged in only one branch, caching that serves one variant to many people, and bots that one variant blocks. Fix the cause, then restart the test with fresh assignment.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### Python example

```python
from scipy.stats import chisquare

SRM_P_VALUE = 0.001

def has_sample_ratio_mismatch(counts: list[int], planned_shares: list[float]) -> bool:
    total = sum(counts)
    return chisquare(counts, [share * total for share in planned_shares]).pvalue < SRM_P_VALUE
```

### JavaScript example

```js
// Two variants: chi-square with one degree of freedom; 10.83 is its value at p = 0.001
const CHI_SQUARE_AT_P_0_001 = 10.83;

export function hasSampleRatioMismatch([control, treatment], controlShare = 0.5) {
  const total = control + treatment;
  const expected = [total * controlShare, total * (1 - controlShare)];
  const chiSquare = [control, treatment].reduce((sum, seen, i) => sum + (seen - expected[i]) ** 2 / expected[i], 0);
  return chiSquare > CHI_SQUARE_AT_P_0_001;
}
```

## Related rules

- ANA-12 [Assign each user to one variant for good](https://agentwranglers.se/playbook/analytics/experiments/experiment-assignment.md) in Product analytics and experiments

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/analytics/experiments/sample-ratio-mismatch">Agent Wranglers product analytics and experiments playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Assign each user to one variant for good](https://agentwranglers.se/playbook/analytics/experiments/experiment-assignment.md)
- Next: [Decide only at the planned sample size or with a sequential test](https://agentwranglers.se/playbook/analytics/experiments/no-peeking.md)
- [Experiments](https://agentwranglers.se/playbook/analytics/experiments.md)