Alert on how fast the budget burns

How to do it

Use two windows per alert, a long one and one twelve times shorter, so alerts start fast and stop soon after recovery. The Google SRE workbook pages at 14.4 times the normal burn over 1 hour, and at 6 times over 6 hours. It opens a ticket at the normal burn over 3 days. These factors spend 2%, 5% and 10% of a 30-day budget. For another window, scale the factor by the window length in days divided by 30.

Example

# Prometheus: page when about 2% of a 30-day budget burns within 1 hour (99.9% objective)
- alert: CheckoutFastBurn
  expr: |
    (sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h])) / sum(rate(http_requests_total{job="checkout"}[1h]))) > (14.4 * 0.001)
    and
    (sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="checkout"}[5m]))) > (14.4 * 0.001)
  labels:
    severity: page

How to check

Inject 100% errors on staging. The page fires within 5 minutes. Twice the budgeted error rate for an hour opens no page. A steady 1.5 times the budgeted rate opens a ticket within 3 days.