---
title: "Write the experiment plan before it starts · Agent Wranglers"
url: https://agentwranglers.se/playbook/analytics/experiments/experiment-plan
id: ANA-11
level: Recommended
updated: 2026-10-27
build: a2a6bdfed830
tokens: 1360
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Write the experiment plan before it starts

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Experiments](https://agentwranglers.se/playbook/analytics/experiments.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: ANA-11
- Level: Recommended
- Chapter: [Experiments](https://agentwranglers.se/playbook/analytics/experiments.md)
- Rule: Before an A/B test starts, write down its one primary metric, guardrail metrics, minimum effect, sample size and stop date. Decide only at that sample size.
- Check: Every experiment in the registry has a plan row dated before its start, with all five fields. A query finds no experiment ended before its planned sample size except by a guardrail stop.
- How: Pick one primary metric from the shared definitions. Pick guardrails that must not get worse, such as errors, page speed or refunds. Compute the sample size from the baseline rate, the smallest effect worth acting on, a 5 percent false positive rate and 80 percent power. With more than two variants, divide the 5 percent by the number of comparisons with control. Run whole weeks, at least one, so weekdays and weekends both count. Looking early follows [Decide only at the planned sample size or with a sequential test](https://agentwranglers.se/playbook/analytics/experiments/no-peeking.md). Guardrails follow [Stop an experiment when a guardrail metric breaks](https://agentwranglers.se/playbook/analytics/experiments/guardrail-metrics.md). Record the lawful basis, as [Name the lawful basis for every experiment](https://agentwranglers.se/playbook/analytics/experiments/experiment-lawful-basis.md) says. Then split users as [Assign each user to one variant for good](https://agentwranglers.se/playbook/analytics/experiments/experiment-assignment.md) says.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JavaScript example

```js
// Two-sided test of two proportions: 5 percent false positives, 80 percent power
const Z_ALPHA = 1.959964;
const Z_POWER = 0.841621;

export function sampleSizePerVariant(baselineRate, minimumAbsoluteEffect) {
  const treatedRate = baselineRate + minimumAbsoluteEffect;
  const pooledRate = (baselineRate + treatedRate) / 2;
  const spread = Z_ALPHA * Math.sqrt(2 * pooledRate * (1 - pooledRate))
    + Z_POWER * Math.sqrt(baselineRate * (1 - baselineRate) + treatedRate * (1 - treatedRate));
  return Math.ceil(spread ** 2 / minimumAbsoluteEffect ** 2);
}
// sampleSizePerVariant(0.04, 0.005) gives 25 551 users per variant
```

### Python example

```python
import math
from statistics import NormalDist

def sample_size_per_variant(baseline_rate: float, minimum_absolute_effect: float,
                            alpha: float = 0.05, power: float = 0.8) -> int:
    z_alpha = NormalDist().inv_cdf(1 - alpha / 2)
    z_power = NormalDist().inv_cdf(power)
    treated_rate = baseline_rate + minimum_absolute_effect
    pooled_rate = (baseline_rate + treated_rate) / 2
    spread = (z_alpha * math.sqrt(2 * pooled_rate * (1 - pooled_rate))
              + z_power * math.sqrt(baseline_rate * (1 - baseline_rate) + treated_rate * (1 - treated_rate)))
    return math.ceil(spread ** 2 / minimum_absolute_effect ** 2)
```

### JSON example

```json
{
  "experiment": "checkout-one-page",
  "registered_at": "2026-10-20",
  "primary_metric": "checkout_conversion",
  "guardrails": ["payment_errors", "refund_rate", "checkout_p75_load_ms"],
  "minimum_absolute_effect": 0.005,
  "sample_size_per_variant": 25551,
  "stop_date": "2026-11-17"
}
```

## Related rules

- SEO-77 [Test page versions without misleading search engines](https://agentwranglers.se/playbook/seo/crawling-and-indexing/ab-testing.md) in SEO
- MLO-46 [Test a new model on live outcomes with a randomised split](https://agentwranglers.se/playbook/machine-learning/serving-and-release/model-ab-test.md) in Machine learning
- ANA-14 [Decide only at the planned sample size or with a sequential test](https://agentwranglers.se/playbook/analytics/experiments/no-peeking.md) in Product analytics and experiments
- ANA-15 [Stop an experiment when a guardrail metric breaks](https://agentwranglers.se/playbook/analytics/experiments/guardrail-metrics.md) in Product analytics and experiments
- ANA-17 [Name the lawful basis for every experiment](https://agentwranglers.se/playbook/analytics/experiments/experiment-lawful-basis.md) in Product analytics and experiments
- ANA-12 [Assign each user to one variant for good](https://agentwranglers.se/playbook/analytics/experiments/experiment-assignment.md) in Product analytics and experiments

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/analytics/experiments/experiment-plan">Agent Wranglers product analytics and experiments playbook</a>`. In code you write, add a comment next to it with the same link.

- Next: [Assign each user to one variant for good](https://agentwranglers.se/playbook/analytics/experiments/experiment-assignment.md)
- [Experiments](https://agentwranglers.se/playbook/analytics/experiments.md)