Decide only at the planned sample size or with a sequential test

How to do it

A test checked every day and stopped at the first p below 0.05 calls far more false wins than 5 percent. Each look is another chance for noise to cross the line. Dashboards that show a live p-value invite exactly that. Either hide the result until the stop date, or use a method made for looks at any time. Examples are a sequential probability ratio test and always-valid confidence intervals. Many experiment tools offer one of these, so turn it on and record which. A sequential test needs more users for the same effect, so plan the size with that method. Stopping early for harm is separate, as Stop an experiment when a guardrail metric breaks says.

Examples

JSON example

{
  "experiment": "checkout-one-page",
  "method": "fixed-size",
  "result_visible_from": "2026-11-17",
  "sample_size_per_variant": 25551
}

Python example

def may_read_result(experiment, today, exposed_per_variant: list[int]) -> bool:
    if experiment.method == "sequential":
        return True
    return today >= experiment.stop_date and min(exposed_per_variant) >= experiment.sample_size_per_variant

JavaScript example

export function mayReadResult(experiment, today, exposedPerVariant) {
  if (experiment.method === 'sequential') return true;
  return today >= experiment.stopDate && Math.min(...exposedPerVariant) >= experiment.sampleSizePerVariant;
}

How to check

The registry shows, for every finished experiment, either a decision dated on or after the planned stop date or a named sequential method. A query finds no experiment called a win before its planned size with a fixed-size method.

Related rules

• ANA-11 Write the experiment plan before it starts in Product analytics and experiments
• ANA-15 Stop an experiment when a guardrail metric breaks in Product analytics and experiments