Write the experiment plan before it starts

How to do it

Pick one primary metric from the shared definitions. Pick guardrails that must not get worse, such as errors, page speed or refunds. Compute the sample size from the baseline rate, the smallest effect worth acting on, a 5 percent false positive rate and 80 percent power. With more than two variants, divide the 5 percent by the number of comparisons with control. Run whole weeks, at least one, so weekdays and weekends both count. Checking results daily and stopping at the first good number makes false wins likely. If you must look early, use a sequential test built for it. Stop early only when a guardrail breaks.

Example

// Two-sided test of two proportions: 5 percent false positives, 80 percent power
const Z_ALPHA = 1.959964;
const Z_POWER = 0.841621;

export function sampleSizePerVariant(baselineRate, minimumAbsoluteEffect) {
  const treatedRate = baselineRate + minimumAbsoluteEffect;
  const pooledRate = (baselineRate + treatedRate) / 2;
  const spread = Z_ALPHA * Math.sqrt(2 * pooledRate * (1 - pooledRate))
    + Z_POWER * Math.sqrt(baselineRate * (1 - baselineRate) + treatedRate * (1 - treatedRate));
  return Math.ceil(spread ** 2 / minimumAbsoluteEffect ** 2);
}
// sampleSizePerVariant(0.04, 0.005) gives 25 551 users per variant

import math
from statistics import NormalDist

def sample_size_per_variant(baseline_rate: float, minimum_absolute_effect: float,
                            alpha: float = 0.05, power: float = 0.8) -> int:
    z_alpha = NormalDist().inv_cdf(1 - alpha / 2)
    z_power = NormalDist().inv_cdf(power)
    treated_rate = baseline_rate + minimum_absolute_effect
    pooled_rate = (baseline_rate + treated_rate) / 2
    spread = (z_alpha * math.sqrt(2 * pooled_rate * (1 - pooled_rate))
              + z_power * math.sqrt(baseline_rate * (1 - baseline_rate) + treated_rate * (1 - treated_rate)))
    return math.ceil(spread ** 2 / minimum_absolute_effect ** 2)

{
  "experiment": "checkout-one-page",
  "registered_at": "2026-10-20",
  "primary_metric": "checkout_conversion",
  "guardrails": ["payment_errors", "refund_rate", "checkout_p75_load_ms"],
  "minimum_absolute_effect": 0.005,
  "sample_size_per_variant": 25551,
  "stop_date": "2026-11-17"
}

How to check

Every experiment in the registry has a plan row dated before its start, with all five fields. A query finds no experiment ended before its planned sample size except by a guardrail stop.