---
title: "Stop an experiment when a guardrail metric · Agent Wranglers"
url: https://agentwranglers.se/playbook/analytics/experiments/guardrail-metrics
id: ANA-15
level: Recommended
updated: 2026-10-27
build: a2a6bdfed830
tokens: 994
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Stop an experiment when a guardrail metric breaks

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Experiments](https://agentwranglers.se/playbook/analytics/experiments.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: ANA-15
- Level: Recommended
- Chapter: [Experiments](https://agentwranglers.se/playbook/analytics/experiments.md)
- Rule: Give every experiment at least two guardrail metrics, such as errors, page speed, refunds or unsubscribes, each with a threshold. Check them daily and stop the variant that crosses one.
- Check: Every experiment in the registry lists at least two guardrails with thresholds. On staging, force errors in the treatment so one guardrail crosses its threshold. The daily check turns the variant off and alerts the owner.
- How: A variant can raise sign-ups while it doubles payment errors or slows the page. Guardrails catch the harm the primary metric does not show. Take them from the [shared metric definitions](https://agentwranglers.se/playbook/analytics/events-and-metrics/metric-definitions.md), and set each threshold in the plan, such as payment errors up by more than 0.5 percentage points. Use a one-sided test against harm, and check daily, since stopping for harm needs no sample size. Switch the variant off through its flag, as [Separate deploying from releasing](https://agentwranglers.se/playbook/engineering/running-in-production/deploy-is-not-release.md) describes. Platform-wide guardrails, such as error rate and page speed, apply to every test without being listed again.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JavaScript example

```js
export function brokenGuardrails(plan, results) {
  return plan.guardrails.filter(g => results[g.metric].treatment - results[g.metric].control > g.maxIncrease);
}

for (const guardrail of brokenGuardrails(plan, todaysResults)) {
  await flags.disable(plan.flag);
  await alertOwner(plan.owner, `Stopped ${plan.experiment}: ${guardrail.metric} crossed its threshold`);
}
```

### Python example

```python
def broken_guardrails(plan: dict, results: dict) -> list[dict]:
    return [g for g in plan["guardrails"]
            if results[g["metric"]]["treatment"] - results[g["metric"]]["control"] > g["max_increase"]]
```

## Related rules

- ANA-11 [Write the experiment plan before it starts](https://agentwranglers.se/playbook/analytics/experiments/experiment-plan.md) in Product analytics and experiments
- ANA-14 [Decide only at the planned sample size or with a sequential test](https://agentwranglers.se/playbook/analytics/experiments/no-peeking.md) in Product analytics and experiments
- ANA-5 [Define each metric once](https://agentwranglers.se/playbook/analytics/events-and-metrics/metric-definitions.md) in Product analytics and experiments
- ENG-25 [Separate deploying from releasing](https://agentwranglers.se/playbook/engineering/running-in-production/deploy-is-not-release.md) in Engineering

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/analytics/experiments/guardrail-metrics">Agent Wranglers product analytics and experiments playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Decide only at the planned sample size or with a sequential test](https://agentwranglers.se/playbook/analytics/experiments/no-peeking.md)
- Next: [Keep a holdout group to measure the sum of changes](https://agentwranglers.se/playbook/analytics/experiments/holdout-groups.md)
- [Experiments](https://agentwranglers.se/playbook/analytics/experiments.md)