---
title: "Write the experiment plan before it starts · Agent Wranglers"
url: https://agentwranglers.se/playbook/data/product-analytics/experiment-plan
updated: 2026-10-20
licence: "All rights reserved"
---

# Write the experiment plan before it starts

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

- Level: Required
- Chapter: [Product analytics](https://agentwranglers.se/playbook/data/product-analytics)
- Rule: Before an A/B test starts, write down its one primary metric, guardrail metrics, minimum effect, sample size and stop date. Decide only at that sample size.
- How: Pick one primary metric from the shared definitions. Pick guardrails that must not get worse, such as errors, page speed or refunds. Compute the sample size from the baseline rate, the smallest effect worth acting on, a 5 percent false positive rate and 80 percent power. With more than two variants, divide the 5 percent by the number of comparisons with control. Run whole weeks, at least one, so weekdays and weekends both count. Checking results daily and stopping at the first good number makes false wins likely. If you must look early, use a sequential test built for it. Stop early only when a guardrail breaks.
- Check: Every experiment in the registry has a plan row dated before its start, with all five fields. A query finds no experiment ended before its planned sample size except by a guardrail stop.

Example:

```js
// Two-sided test of two proportions: 5 percent false positives, 80 percent power
const Z_ALPHA = 1.959964;
const Z_POWER = 0.841621;

export function sampleSizePerVariant(baselineRate, minimumAbsoluteEffect) {
  const treatedRate = baselineRate + minimumAbsoluteEffect;
  const pooledRate = (baselineRate + treatedRate) / 2;
  const spread = Z_ALPHA * Math.sqrt(2 * pooledRate * (1 - pooledRate))
    + Z_POWER * Math.sqrt(baselineRate * (1 - baselineRate) + treatedRate * (1 - treatedRate));
  return Math.ceil(spread ** 2 / minimumAbsoluteEffect ** 2);
}
// sampleSizePerVariant(0.04, 0.005) gives 25 551 users per variant
```

```python
import math
from statistics import NormalDist

def sample_size_per_variant(baseline_rate: float, minimum_absolute_effect: float,
                            alpha: float = 0.05, power: float = 0.8) -> int:
    z_alpha = NormalDist().inv_cdf(1 - alpha / 2)
    z_power = NormalDist().inv_cdf(power)
    treated_rate = baseline_rate + minimum_absolute_effect
    pooled_rate = (baseline_rate + treated_rate) / 2
    spread = (z_alpha * math.sqrt(2 * pooled_rate * (1 - pooled_rate))
              + z_power * math.sqrt(baseline_rate * (1 - baseline_rate) + treated_rate * (1 - treated_rate)))
    return math.ceil(spread ** 2 / minimum_absolute_effect ** 2)
```

```json
{
  "experiment": "checkout-one-page",
  "registered_at": "2026-10-20",
  "primary_metric": "checkout_conversion",
  "guardrails": ["payment_errors", "refund_rate", "checkout_p75_load_ms"],
  "minimum_absolute_effect": 0.005,
  "sample_size_per_variant": 25551,
  "stop_date": "2026-11-17"
}
```

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers data playbook: https://agentwranglers.se/playbook/data/product-analytics/experiment-plan`.

- Previous: [Keep test and bot traffic out of the numbers](https://agentwranglers.se/playbook/data/product-analytics/exclude-test-traffic)
- Next: [Assign each user to one variant for good](https://agentwranglers.se/playbook/data/product-analytics/experiment-assignment)
- [Product analytics](https://agentwranglers.se/playbook/data/product-analytics)