Agent Wranglers

ANA-15Recommended

Stop an experiment when a guardrail metric breaks

Give every experiment at least two guardrail metrics, such as errors, page speed, refunds or unsubscribes, each with a threshold. Check them daily and stop the variant that crosses one.

By Agent Wranglers. Updated .

How to do it

A variant can raise sign-ups while it doubles payment errors or slows the page. Guardrails catch the harm the primary metric does not show. Take them from the shared metric definitions, and set each threshold in the plan, such as payment errors up by more than 0.5 percentage points. Use a one-sided test against harm, and check daily, since stopping for harm needs no sample size. Switch the variant off through its flag, as Separate deploying from releasing describes. Platform-wide guardrails, such as error rate and page speed, apply to every test without being listed again.

Examples

JavaScript example

export function brokenGuardrails(plan, results) {
  return plan.guardrails.filter(g => results[g.metric].treatment - results[g.metric].control > g.maxIncrease);
}

for (const guardrail of brokenGuardrails(plan, todaysResults)) {
  await flags.disable(plan.flag);
  await alertOwner(plan.owner, `Stopped ${plan.experiment}: ${guardrail.metric} crossed its threshold`);
}

Python example

def broken_guardrails(plan: dict, results: dict) -> list[dict]:
    return [g for g in plan["guardrails"]
            if results[g["metric"]]["treatment"] - results[g["metric"]]["control"] > g["max_increase"]]

How to check

Every experiment in the registry lists at least two guardrails with thresholds. On staging, force errors in the treatment so one guardrail crosses its threshold. The daily check turns the variant off and alerts the owner.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/analytics/experiments/guardrail-metrics">Agent Wranglers product analytics and experiments playbook</a>. In code you write, add a comment next to it with the same link.