Assign each user to one variant for good

How to do it

Hash experiment:user_id with a stable hash, such as SHA-256, and map it to a bucket from 0 to 9999. Never use Math.random() per page view. Signed-out visitors get a random ID that moves to the account at sign-in. Log the exposure once, when the person sees the change, and analyse only exposed users. Before reading results, test for sample ratio mismatch. A chi-square test with p below 0.001 means assignment or logging is broken, so the results are void.

Example

import { createHash } from 'node:crypto';

const BUCKETS = 10_000;
export function variantFor(experiment, userId, variants = ['control', 'treatment']) {
  const bucket = createHash('sha256').update(`${experiment}:${userId}`).digest().readUInt32BE(0) % BUCKETS;
  return variants[Math.floor((bucket * variants.length) / BUCKETS)];
}

import hashlib
from scipy.stats import chisquare

BUCKETS = 10_000

def variant_for(experiment: str, user_id: str, variants=("control", "treatment")) -> str:
    digest = hashlib.sha256(f"{experiment}:{user_id}".encode()).digest()
    bucket = int.from_bytes(digest[:4], "big") % BUCKETS
    return variants[bucket * len(variants) // BUCKETS]

def has_sample_ratio_mismatch(counts: list[int], expected_shares: list[float]) -> bool:
    total = sum(counts)
    return chisquare(counts, [share * total for share in expected_shares]).pvalue < 0.001

How to check

A unit test assigns the same user 1000 times and gets one variant. A test over 100 000 fake IDs gets a 50/50 split within 1 percent. The analysis script stops on a sample ratio mismatch.