Agent Wranglers

Capacity and cost Required

Warn before resources run out

Alert on how soon a disk, memory pool or quota will be full, not only on how full it is.

By Erik Bergstedt, Agent Wranglers. Updated .

How to do it

Predict from the recent trend, for example with Prometheus predict_linear. Page when the disk will fill within 4 hours, and open a ticket when it will fill within 7 days. Alert on TLS certificates as certificate renewal sets, and on domains 30 days before renewal.

Example

predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 3600) < 0
(probe_ssl_earliest_cert_expiry - time()) / 86400 < 14

How to check

Fill a test disk on staging at a steady rate. The page fires at least 1 hour before it is full.

For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers operations playbook: https://agentwranglers.se/playbook/operations/capacity-and-cost/exhaustion-alerts.