Agent WranglersAlerts and on-call Required
Prove that alerts still reach you
Keep one alert that always fires and goes to an outside service, which pages when it stops arriving. Send a test page every month.
How to do it
A stopped alert manager, an expired paging key or a broken mail relay fails silently, and then every other alert is silent too. Prometheus setups call this always-firing alert Watchdog. Route it to a heartbeat monitor outside your own systems that expects it every few minutes. Once a month, send a test page and confirm the on-call phone rings.
Example
# Prometheus rule: always firing
- alert: Watchdog
expr: vector(1)
labels:
severity: heartbeat
# Alertmanager: repeat it to the outside monitor every minute
route:
routes:
- matchers: ['severity="heartbeat"']
receiver: outside-heartbeat
group_wait: 0s
group_interval: 1m
repeat_interval: 1m
receivers:
- name: outside-heartbeat
webhook_configs:
- url: https://heartbeat.example.com/ping/alertingHow to check
Stop the alert manager on staging. The outside monitor pages within 15 minutes. The paging tool shows a test page from the last 31 days.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers operations playbook: https://agentwranglers.se/playbook/operations/alerts-and-on-call/alert-pipeline-heartbeat.