Agent WranglersExperiments
Plan each test before it starts, assign people for good and check the split. Decide at the planned size, watch guardrails, and leave prices out.
- Required 1
- Recommended 5
- Optional 1
- Avoid 1
Required
Name the lawful basis for every experiment
ANA-17 EU lawRecord the GDPR lawful basis for each experiment before it starts, and ask for consent before storing an experiment ID on the device. Screen tests that change prices, credit, health or content for children for a data protection impact assessment.
Recommended
Write the experiment plan before it starts
ANA-11Before an A/B test starts, write down its one primary metric, guardrail metrics, minimum effect, sample size and stop date. Decide only at that sample size.
Assign each user to one variant for good
ANA-12Assign variants by hashing a stable user ID with the experiment name, so a person sees the same variant on every visit and device. Log an exposure when the variant is shown.
Stop the analysis on a sample ratio mismatch
ANA-13Before reading any result, compare the number of exposed users in each variant with the planned split. When a chi-square test gives p below 0.001, treat the results as void and find the cause.
Decide only at the planned sample size or with a sequential test
ANA-14Read the result of a fixed-size test only when it reaches its planned sample size. When the team must look earlier, use a sequential test built for repeated looks, and name it in the plan.
Stop an experiment when a guardrail metric breaks
ANA-15Give every experiment at least two guardrail metrics, such as errors, page speed, refunds or unsubscribes, each with a threshold. Check them daily and stop the variant that crosses one.
Optional
Keep a holdout group to measure the sum of changes
ANA-16Keep a small share of users, such as 5 percent, on the product as it was at the start of the quarter. At the quarter's end, compare them with everyone else on the primary business metric.
Avoid
Testing prices or targeting vulnerable people in experiments
ANA-18Never show two people different prices for the same thing in an A/B test. Leave minors and customers flagged as vulnerable out of every experiment.
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/analytics/experiments">Agent Wranglers product analytics and experiments playbook</a>. In code you write, add a comment next to it with the same link.