Agent WranglersAlerts and on-call Required
Alert on symptoms
Page people for what users feel, such as errors, slow answers and a service that is down. Watch causes such as CPU on dashboards instead.
How to do it
Base paging alerts on the service level indicators and on outside checks. A full disk that will stop writes soon counts as a symptom too. High CPU alone does not, because users may notice nothing.
How to check
List every paging alert. Each one is tied to an indicator, an outside check or a resource that will run out within hours.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers operations playbook: https://agentwranglers.se/playbook/operations/alerts-and-on-call/alert-on-symptoms.