---
title: "Experiments · Agent Wranglers"
url: https://agentwranglers.se/playbook/analytics/experiments
parts: 8
tokens_total: 7609
updated: 2026-10-27
build: a2a6bdfed830
tokens: 996
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Experiments

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Product analytics and experiments](https://agentwranglers.se/playbook/analytics.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

> Plan each test before it starts, assign people for good and check the split. Decide at the planned size, watch guardrails, and leave prices out.

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/analytics/experiments">Agent Wranglers product analytics and experiments playbook</a>`. In code you write, add a comment next to it with the same link.

## Required

- ANA-17 [Name the lawful basis for every experiment](https://agentwranglers.se/playbook/analytics/experiments/experiment-lawful-basis.md) · 1,089 tokens · updated 2026-10-27
  Record the GDPR lawful basis for each experiment before it starts, and ask for consent before storing an experiment ID on the device. Screen tests that change prices, credit, health or content for children for a data protection impact assessment.

## Recommended

- ANA-11 [Write the experiment plan before it starts](https://agentwranglers.se/playbook/analytics/experiments/experiment-plan.md) · 1,296 tokens · updated 2026-10-27
  Before an A/B test starts, write down its one primary metric, guardrail metrics, minimum effect, sample size and stop date. Decide only at that sample size.
- ANA-12 [Assign each user to one variant for good](https://agentwranglers.se/playbook/analytics/experiments/experiment-assignment.md) · 897 tokens · updated 2026-10-27
  Assign variants by hashing a stable user ID with the experiment name, so a person sees the same variant on every visit and device. Log an exposure when the variant is shown.
- ANA-13 [Stop the analysis on a sample ratio mismatch](https://agentwranglers.se/playbook/analytics/experiments/sample-ratio-mismatch.md) · 815 tokens · updated 2026-10-27
  Before reading any result, compare the number of exposed users in each variant with the planned split. When a chi-square test gives p below 0.001, treat the results as void and find the cause.
- ANA-14 [Decide only at the planned sample size or with a sequential test](https://agentwranglers.se/playbook/analytics/experiments/no-peeking.md) · 887 tokens · updated 2026-10-27
  Read the result of a fixed-size test only when it reaches its planned sample size. When the team must look earlier, use a sequential test built for repeated looks, and name it in the plan.
- ANA-15 [Stop an experiment when a guardrail metric breaks](https://agentwranglers.se/playbook/analytics/experiments/guardrail-metrics.md) · 941 tokens · updated 2026-10-27
  Give every experiment at least two guardrail metrics, such as errors, page speed, refunds or unsubscribes, each with a threshold. Check them daily and stop the variant that crosses one.

## Optional

- ANA-16 [Keep a holdout group to measure the sum of changes](https://agentwranglers.se/playbook/analytics/experiments/holdout-groups.md) · 697 tokens · updated 2026-10-27
  Keep a small share of users, such as 5 percent, on the product as it was at the start of the quarter. At the quarter's end, compare them with everyone else on the primary business metric.

## Avoid

- ANA-18 [Testing prices or targeting vulnerable people in experiments](https://agentwranglers.se/playbook/analytics/experiments/no-price-or-vulnerable-tests.md) · 987 tokens · updated 2026-10-27
  Never show two people different prices for the same thing in an A/B test. Leave minors and customers flagged as vulnerable out of every experiment.

- Previous: [Privacy and consent](https://agentwranglers.se/playbook/analytics/privacy-and-consent.md)
- [All chapters](https://agentwranglers.se/playbook/analytics.md)