Measure rate, errors and duration

For every service, record the request rate, the error rate and the time each request takes, split by route.

By Erik Bergstedt, Agent Wranglers. Updated 20 October 2026.

How to do it

Use a metrics library or an agent, such as OpenTelemetry or a Prometheus client. Record duration as a histogram, so you can read percentiles such as p95 and p99. Averages hide the slow requests users notice. For a single small service, derive the three numbers from the access log, such as status codes and request times per line.

Example

# Prometheus queries
# OpenTelemetry metric http.server.request.duration, as exported to Prometheus
sum by (http_route) (rate(http_server_request_duration_seconds_count[5m]))
sum by (http_route) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m]))
histogram_quantile(0.95, sum by (le, http_route) (rate(http_server_request_duration_seconds_bucket[5m])))

How to check

For each service, a query returns requests per second, errors per second and p95 duration for the last hour, per route. For a single service, a daily script over the access log may produce the same three numbers.

For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers operations playbook: https://agentwranglers.se/playbook/operations/metrics-and-traces/red-metrics.