---
title: "Decide only at the planned sample size or · Agent Wranglers"
url: https://agentwranglers.se/playbook/analytics/experiments/no-peeking
id: ANA-14
level: Recommended
updated: 2026-10-27
build: a2a6bdfed830
tokens: 929
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Decide only at the planned sample size or with a sequential test

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Experiments](https://agentwranglers.se/playbook/analytics/experiments.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: ANA-14
- Level: Recommended
- Chapter: [Experiments](https://agentwranglers.se/playbook/analytics/experiments.md)
- Rule: Read the result of a fixed-size test only when it reaches its planned sample size. When the team must look earlier, use a sequential test built for repeated looks, and name it in the plan.
- Check: The registry shows, for every finished experiment, either a decision dated on or after the planned stop date or a named sequential method. A query finds no experiment called a win before its planned size with a fixed-size method.
- How: A test checked every day and stopped at the first p below 0.05 calls far more false wins than 5 percent. Each look is another chance for noise to cross the line. Dashboards that show a live p-value invite exactly that. Either hide the result until the stop date, or use a method made for looks at any time. Examples are a sequential probability ratio test and always-valid confidence intervals. Many experiment tools offer one of these, so turn it on and record which. A sequential test needs more users for the same effect, so plan the size with that method. Stopping early for harm is separate, as [Stop an experiment when a guardrail metric breaks](https://agentwranglers.se/playbook/analytics/experiments/guardrail-metrics.md) says.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JSON example

```json
{
  "experiment": "checkout-one-page",
  "method": "fixed-size",
  "result_visible_from": "2026-11-17",
  "sample_size_per_variant": 25551
}
```

### Python example

```python
def may_read_result(experiment, today, exposed_per_variant: list[int]) -> bool:
    if experiment.method == "sequential":
        return True
    return today >= experiment.stop_date and min(exposed_per_variant) >= experiment.sample_size_per_variant
```

### JavaScript example

```js
export function mayReadResult(experiment, today, exposedPerVariant) {
  if (experiment.method === 'sequential') return true;
  return today >= experiment.stopDate && Math.min(...exposedPerVariant) >= experiment.sampleSizePerVariant;
}
```

## Related rules

- ANA-11 [Write the experiment plan before it starts](https://agentwranglers.se/playbook/analytics/experiments/experiment-plan.md) in Product analytics and experiments
- ANA-15 [Stop an experiment when a guardrail metric breaks](https://agentwranglers.se/playbook/analytics/experiments/guardrail-metrics.md) in Product analytics and experiments

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/analytics/experiments/no-peeking">Agent Wranglers product analytics and experiments playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Stop the analysis on a sample ratio mismatch](https://agentwranglers.se/playbook/analytics/experiments/sample-ratio-mismatch.md)
- Next: [Stop an experiment when a guardrail metric breaks](https://agentwranglers.se/playbook/analytics/experiments/guardrail-metrics.md)
- [Experiments](https://agentwranglers.se/playbook/analytics/experiments.md)