Agent Wranglers

ANA-16Optional

Keep a holdout group to measure the sum of changes

Keep a small share of users, such as 5 percent, on the product as it was at the start of the quarter. At the quarter's end, compare them with everyone else on the primary business metric.

By Agent Wranglers. Updated .

How to do it

Each test can win on its own while their sum does nothing, because some wins were noise and some cancel out. A holdout measures what the shipped changes added together. Assign it with the same stable hash as Assign each user to one variant for good, under its own name. Leave it out of every test that quarter. Keep it small and short, since those users miss improvements. Never hold back security fixes, legal changes or accessibility fixes.

Examples

JavaScript example

const HOLDOUT = { name: 'holdout-2026-q4', share: 0.05 };

export function inHoldout(userId) {
  return bucketFor(HOLDOUT.name, userId) < HOLDOUT.share * BUCKETS;
}

Python example

HOLDOUT = {"name": "holdout-2026-q4", "share": 0.05}

def in_holdout(user_id: str) -> bool:
    return bucket_for(HOLDOUT["name"], user_id) < HOLDOUT["share"] * BUCKETS

How to check

The assignment code puts the holdout's users in no other experiment. A test assigns 100,000 fake IDs and finds the holdout within 0.5 percentage points of its planned share.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/analytics/experiments/holdout-groups">Agent Wranglers product analytics and experiments playbook</a>. In code you write, add a comment next to it with the same link.