Write the experiment plan before it starts

How to do it

Pick one primary metric from the shared definitions. Pick guardrails that must not get worse, such as errors, page speed or refunds. Compute the sample size from the baseline rate, the smallest effect worth acting on, a 5 percent false positive rate and 80 percent power. With more than two variants, divide the 5 percent by the number of comparisons with control. Run whole weeks, at least one, so weekdays and weekends both count. Looking early follows Decide only at the planned sample size or with a sequential test. Guardrails follow Stop an experiment when a guardrail metric breaks. Record the lawful basis, as Name the lawful basis for every experiment says. Then split users as Assign each user to one variant for good says.

Examples

JavaScript example

// Two-sided test of two proportions: 5 percent false positives, 80 percent power
const Z_ALPHA = 1.959964;
const Z_POWER = 0.841621;

export function sampleSizePerVariant(baselineRate, minimumAbsoluteEffect) {
  const treatedRate = baselineRate + minimumAbsoluteEffect;
  const pooledRate = (baselineRate + treatedRate) / 2;
  const spread = Z_ALPHA * Math.sqrt(2 * pooledRate * (1 - pooledRate))
    + Z_POWER * Math.sqrt(baselineRate * (1 - baselineRate) + treatedRate * (1 - treatedRate));
  return Math.ceil(spread ** 2 / minimumAbsoluteEffect ** 2);
}
// sampleSizePerVariant(0.04, 0.005) gives 25 551 users per variant

Python example

import math
from statistics import NormalDist

def sample_size_per_variant(baseline_rate: float, minimum_absolute_effect: float,
                            alpha: float = 0.05, power: float = 0.8) -> int:
    z_alpha = NormalDist().inv_cdf(1 - alpha / 2)
    z_power = NormalDist().inv_cdf(power)
    treated_rate = baseline_rate + minimum_absolute_effect
    pooled_rate = (baseline_rate + treated_rate) / 2
    spread = (z_alpha * math.sqrt(2 * pooled_rate * (1 - pooled_rate))
              + z_power * math.sqrt(baseline_rate * (1 - baseline_rate) + treated_rate * (1 - treated_rate)))
    return math.ceil(spread ** 2 / minimum_absolute_effect ** 2)

JSON example

{
  "experiment": "checkout-one-page",
  "registered_at": "2026-10-20",
  "primary_metric": "checkout_conversion",
  "guardrails": ["payment_errors", "refund_rate", "checkout_p75_load_ms"],
  "minimum_absolute_effect": 0.005,
  "sample_size_per_variant": 25551,
  "stop_date": "2026-11-17"
}

How to check

Every experiment in the registry has a plan row dated before its start, with all five fields. A query finds no experiment ended before its planned sample size except by a guardrail stop.

Related rules

• SEO-77 Test page versions without misleading search engines in SEO
• MLO-46 Test a new model on live outcomes with a randomised split in Machine learning
• ANA-14 Decide only at the planned sample size or with a sequential test in Product analytics and experiments
• ANA-15 Stop an experiment when a guardrail metric breaks in Product analytics and experiments
• ANA-17 Name the lawful basis for every experiment in Product analytics and experiments
• ANA-12 Assign each user to one variant for good in Product analytics and experiments