Agent Wranglers

Alerts and on-call Required

Make every alert actionable

Every alert needs a person to act now, names the service, and links to a runbook and a dashboard.

By Erik Bergstedt, Agent Wranglers. Updated .

How to do it

Delete or downgrade alerts that people acknowledge and ignore. Write the runbook as numbered steps that someone half awake can follow. Review the alerts after every on-call week.

Example

- alert: CheckoutFastBurn
  annotations:
    summary: Checkout errors are burning the monthly budget fast
    runbook_url: https://wiki.example.com/runbooks/checkout-errors
    dashboard: https://grafana.example.com/d/checkout

How to check

Every alert rule has a runbook link and a dashboard link. In the last month, no alert fired more than three times without anyone taking an action.

For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers operations playbook: https://agentwranglers.se/playbook/operations/alerts-and-on-call/actionable-alerts.