Test a new model on live outcomes with a randomised split

How to do it

Offline scores and a canary show that a model works. Only a randomised split shows that it helps, such as more sales or fewer missed fraud cases. Assign by user, not by request, so one person sees one model. Plan the experiment the way write the experiment plan before it starts says, with guardrail metrics such as complaints. Read the result once, at the stop date, with the interval from report each score with its uncertainty. Run it after the canary passes.

Examples

Python example

def model_for(user_id: str, experiment: Experiment) -> str:
    bucket = int(hashlib.sha256(f"{experiment.id}:{user_id}".encode()).hexdigest(), 16) % 100
    return experiment.candidate if bucket < experiment.candidate_percent else experiment.control

TypeScript example

export function modelFor(userId: string, experiment: Experiment): string {
  const bucket = Number(BigInt(`0x${sha256(`${experiment.id}:${userId}`)}`) % 100n);
  return bucket < experiment.candidatePercent ? experiment.candidate : experiment.control;
}

How to check

Each model release that changed a business metric links an experiment plan dated before its start. The prediction log shows each user under one model version for the whole test.

Sources

• Google Cloud, MLOps continuous delivery and automation pipelines

Related rules

• ANA-11 Write the experiment plan before it starts in Product analytics and experiments
• MLO-17 Report each score with its uncertainty in Machine learning
• MLO-27 Release models through a canary that compares predictions in Machine learning