---
title: "Product analytics and experiments rules ยท Agent Wranglers"
url: https://agentwranglers.se/playbook/analytics/checklist
updated: 2026-10-27
build: a2a6bdfed830
tokens: 2128
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---
# Product analytics and experiments checklist
For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Product analytics and experiments](https://agentwranglers.se/playbook/analytics.md).
By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.
Every rule in the product analytics and experiments playbook, by chapter, required rules first. Each links to its own page with how we built it and how to check it. This is a lot to check. [An Expert can check it for you](https://agentwranglers.se/contact.md).
**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `Agent Wranglers product analytics and experiments playbook`. In code you write, add a comment next to it with the same link.
## Events and metrics
- [ ] ANA-1 **Recommended** [Keep a tracking plan in the code](https://agentwranglers.se/playbook/analytics/events-and-metrics/tracking-plan.md): List every analytics event in one file in the repository, with its name, purpose, owner and properties. Name events as an object and a past-tense action.
- [ ] ANA-2 **Recommended** [Validate every event before you store it](https://agentwranglers.se/playbook/analytics/events-and-metrics/event-validation.md): Check each incoming event against its schema at the collector. Store valid events, and set invalid ones aside with the reason.
- [ ] ANA-3 **Recommended** [Record business facts on the server](https://agentwranglers.se/playbook/analytics/events-and-metrics/server-side-business-events.md): Record events that count money, sign-ups or other business facts on the server, at the moment the fact is committed. Use browser events only for behaviour in the page.
- [ ] ANA-4 **Recommended** [Keep test and bot traffic out of the numbers](https://agentwranglers.se/playbook/analytics/events-and-metrics/exclude-test-traffic.md): Mark every event from staff, test accounts, automated tests and known bots at the collector, and leave marked events out of every metric.
- [ ] ANA-5 **Recommended** [Define each metric once](https://agentwranglers.se/playbook/analytics/events-and-metrics/metric-definitions.md): Define every business metric once, in version control, and build every dashboard and report from that definition.
- [ ] ANA-6 **Recommended** [Measure how many new accounts reach their first result](https://agentwranglers.se/playbook/analytics/events-and-metrics/activation-rate.md): Define one activation event, such as the first invoice sent, and record it on the server. Report the share of each week's new accounts that reach it within 7 days.
## Privacy and consent
- [ ] ANA-7 **Required** (EU law) [Track only after consent](https://agentwranglers.se/playbook/analytics/privacy-and-consent/analytics-consent.md): Send no analytics event that uses device storage or identifies a person until that person has agreed. Read the stored choice in the tracking module itself. Without a yes, count visits only, with no cookie, no device storage and no ID.
- [ ] ANA-8 **Required** [Keep personal data out of events](https://agentwranglers.se/playbook/analytics/privacy-and-consent/events-without-personal-data.md): Never put names, email addresses, phone numbers, free text or full URLs with query strings in event properties. Use an internal ID that analytics cannot turn back into a person.
- [ ] ANA-9 **Required** [Hide personal data from session recordings](https://agentwranglers.se/playbook/analytics/privacy-and-consent/session-recording-privacy.md): Session recording and heatmap tools mask every typed value and all text from customer records before anything leaves the browser. They record nothing on payment, sign-in or account settings screens.
- [ ] ANA-10 **Recommended** [Delete raw events on a schedule](https://agentwranglers.se/playbook/analytics/privacy-and-consent/analytics-retention.md): Keep raw events with an analytics ID for a fixed time only, such as 14 months. Then delete them or roll them up into counts without IDs.
## Experiments
- [ ] ANA-17 **Required** (EU law) [Name the lawful basis for every experiment](https://agentwranglers.se/playbook/analytics/experiments/experiment-lawful-basis.md): Record the GDPR lawful basis for each experiment before it starts, and ask for consent before storing an experiment ID on the device. Screen tests that change prices, credit, health or content for children for a data protection impact assessment.
- [ ] ANA-11 **Recommended** [Write the experiment plan before it starts](https://agentwranglers.se/playbook/analytics/experiments/experiment-plan.md): Before an A/B test starts, write down its one primary metric, guardrail metrics, minimum effect, sample size and stop date. Decide only at that sample size.
- [ ] ANA-12 **Recommended** [Assign each user to one variant for good](https://agentwranglers.se/playbook/analytics/experiments/experiment-assignment.md): Assign variants by hashing a stable user ID with the experiment name, so a person sees the same variant on every visit and device. Log an exposure when the variant is shown.
- [ ] ANA-13 **Recommended** [Stop the analysis on a sample ratio mismatch](https://agentwranglers.se/playbook/analytics/experiments/sample-ratio-mismatch.md): Before reading any result, compare the number of exposed users in each variant with the planned split. When a chi-square test gives p below 0.001, treat the results as void and find the cause.
- [ ] ANA-14 **Recommended** [Decide only at the planned sample size or with a sequential test](https://agentwranglers.se/playbook/analytics/experiments/no-peeking.md): Read the result of a fixed-size test only when it reaches its planned sample size. When the team must look earlier, use a sequential test built for repeated looks, and name it in the plan.
- [ ] ANA-15 **Recommended** [Stop an experiment when a guardrail metric breaks](https://agentwranglers.se/playbook/analytics/experiments/guardrail-metrics.md): Give every experiment at least two guardrail metrics, such as errors, page speed, refunds or unsubscribes, each with a threshold. Check them daily and stop the variant that crosses one.
- [ ] ANA-16 **Optional** [Keep a holdout group to measure the sum of changes](https://agentwranglers.se/playbook/analytics/experiments/holdout-groups.md): Keep a small share of users, such as 5 percent, on the product as it was at the start of the quarter. At the quarter's end, compare them with everyone else on the primary business metric.
- [ ] ANA-18 **Avoid** [Testing prices or targeting vulnerable people in experiments](https://agentwranglers.se/playbook/analytics/experiments/no-price-or-vulnerable-tests.md): Never show two people different prices for the same thing in an A/B test. Leave minors and customers flagged as vulnerable out of every experiment.
## Related playbooks
- [Records and data map](https://agentwranglers.se/playbook/compliance/records-and-data-map.md) in the Compliance and governance playbook. Analytics and experiments need a lawful basis, a retention period and a place in the data map.
- [Privacy](https://agentwranglers.se/playbook/website/privacy.md) in the Website playbook. The consent banner and the privacy signal decide what analytics may record.
- [Warehouse and models](https://agentwranglers.se/playbook/data-engineering/warehouse-and-models.md) in the Data engineering playbook. Events copied into a warehouse become tested models with one owner each.
- [Catalog and prices](https://agentwranglers.se/playbook/ecommerce/catalog-and-prices.md) in the E-commerce and fulfilment playbook. A price set for one person must be disclosed, so price tests follow these rules.
## Show the badge on your project
When your project passes every Required rule, you can show this badge on your site or in your README. It links to this checklist, so anyone can run the same checks.
```html
```
```markdown
[](https://agentwranglers.se/playbook/analytics/checklist.md)
```