Agent WranglersOPS-84Recommended
Watch the status of the services you depend on
List every outside service the product needs, such as payments, email and the cloud. Measure each one's error rate and speed from your own calls, and subscribe the incident channel to its status feed.
How to do it
When a provider fails, your own alerts fire on symptoms, and the on-call person loses time hunting a bug that is not yours. Record each outside call's result and time per provider, as Measure rate, errors and duration does for your own routes. Subscribe to each provider's status page by RSS, email or webhook, and to the cloud's account health events, such as AWS Health. Put the provider name on the service dashboard, so a failing provider shows at a glance. Calls to a failing provider stop through Stop calling a failing service with a circuit breaker.
How to check
The dependency list matches the outgoing hosts in the last 30 days of traffic logs. Every listed provider has an error rate and speed graph on the dashboard and a status subscription that posts to the incident channel.
Sources
Related rules
- OPS-12 Measure rate, errors and duration in Operations and observability
- OPS-74 Stop calling a failing service with a circuit breaker in Operations and observability
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/operations/alerts-and-on-call/dependency-status">Agent Wranglers operations and observability playbook</a>. In code you write, add a comment next to it with the same link.