Agent WranglersOPS-42Required
Warn before resources run out
Alert on how soon a disk, memory pool or quota will be full, not only on how full it is.
How to do it
Predict from the recent trend, for example with Prometheus predict_linear. Page when the disk will fill within 4 hours, and open a ticket when it will fill within 7 days. Alert on TLS certificates as certificate renewal sets, and on domains 30 days before renewal.
Examples
Plain text example
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 3600) < 0
(probe_ssl_earliest_cert_expiry - time()) / 86400 < 14How to check
Fill a test disk on staging at a steady rate. The page fires at least 1 hour before it is full.
Sources
Related rules
- DAT-36 Pool database connections in Data and databases
- APS-102 Renew certificates automatically in Application security
- OPS-65 Cap the size of container logs in Operations and observability
- SCL-14 Cap the cache server's memory and choose its eviction in Scaling
- SCL-36 Watch size and growth in Scaling
- MED-29 Watch data sent out per file and per owner in Files and media
- EVT-8 Cap and watch replication slots in Integration and events
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/operations/capacity-and-resilience/exhaustion-alerts">Agent Wranglers operations and observability playbook</a>. In code you write, add a comment next to it with the same link.