Metrics and traces Recommended
Give each service one dashboard
Give every service one dashboard that shows its objectives, its rate, errors and duration, and its resources, top to bottom.
How to do it
Build the dashboard from code or a template, such as Grafana JSON in version control. Link it from the runbook and from every alert for that service.
How to check
Every service has exactly one dashboard, and every alert links to it. Every dashboard exists in version control.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers operations playbook: https://agentwranglers.se/playbook/operations/metrics-and-traces/service-dashboards.