Agent Wranglers

ANA-13Recommended

Stop the analysis on a sample ratio mismatch

Before reading any result, compare the number of exposed users in each variant with the planned split. When a chi-square test gives p below 0.001, treat the results as void and find the cause.

By Agent Wranglers. Updated .

How to do it

A 50/50 test that ends at 50,000 and 48,500 users has lost people in one variant. A redirect that drops tracking, a bot filter, or a crash in the new page all do this. Whatever caused it also changes who is left, so the measured effect cannot be trusted. Run the check in the analysis script before any metric, per day and for the whole test, so a break shows the day it began. Common causes are exposure logged in only one branch, caching that serves one variant to many people, and bots that one variant blocks. Fix the cause, then restart the test with fresh assignment.

Examples

Python example

from scipy.stats import chisquare

SRM_P_VALUE = 0.001

def has_sample_ratio_mismatch(counts: list[int], planned_shares: list[float]) -> bool:
    total = sum(counts)
    return chisquare(counts, [share * total for share in planned_shares]).pvalue < SRM_P_VALUE

JavaScript example

// Two variants: chi-square with one degree of freedom; 10.83 is its value at p = 0.001
const CHI_SQUARE_AT_P_0_001 = 10.83;

export function hasSampleRatioMismatch([control, treatment], controlShare = 0.5) {
  const total = control + treatment;
  const expected = [total * controlShare, total * (1 - controlShare)];
  const chiSquare = [control, treatment].reduce((sum, seen, i) => sum + (seen - expected[i]) ** 2 / expected[i], 0);
  return chiSquare > CHI_SQUARE_AT_P_0_001;
}

How to check

A unit test feeds the check 50,000 and 48,500 for a planned 50/50 split, and the analysis stops with a sample ratio mismatch. With 50,000 and 49,800, it goes on.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/analytics/experiments/sample-ratio-mismatch">Agent Wranglers product analytics and experiments playbook</a>. In code you write, add a comment next to it with the same link.