Agent WranglersAlerts and on-call
Wake a person only for problems users feel and a person can fix. Give every alert a runbook, a clear owner and a path to escalate.
- Required 5
- Recommended 1
Required
Alert on symptoms
RequiredPage people for what users feel, such as errors, slow answers and a service that is down. Watch causes such as CPU on dashboards instead.
Make every alert actionable
RequiredEvery alert needs a person to act now, names the service, and links to a runbook and a dashboard.
Page only for what cannot wait
RequiredSplit alerts into pages, which need someone now, and tickets, which can wait for working hours.
Escalate when nobody answers
RequiredWhen a page is not acknowledged within a set time, send it to the next person automatically.
Alert when a scheduled job stops running
RequiredEvery scheduled job reports when it finishes. Alert when a report is missing, not only when a job fails loudly.
Recommended
Share on-call fairly
RecommendedPut named people on a rota with a backup, hand over between shifts, and give time back after busy nights.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers operations playbook: https://agentwranglers.se/playbook/operations/alerts-and-on-call.