---
title: "Measure cost and speed in every eval run · Agent Wranglers"
url: https://agentwranglers.se/playbook/ai-features/evals-and-quality/eval-cost-and-latency
updated: 2026-10-20
licence: "All rights reserved"
---

# Measure cost and speed in every eval run

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

- Level: Recommended
- Chapter: [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)
- Rule: Record tokens, cost and latency for each case in every eval run, and fail the run when they pass the limits in the feature's spec.
- How: Read token counts from the provider's usage fields and price them from one price table in the gateway. Report the median and 95th percentile latency and the average cost per case. A prompt change that adds a long context can keep the score and double the bill. This check catches it before release.
- Check: The eval report shows cost and latency per case. On a branch, set the spec's cost limit to half the last measured average cost, and the eval job fails on cost.

Example:

```ts
const p95 = percentile(results.map(r => r.latencyMs), 95);
const avgCost = results.reduce((sum, r) => sum + r.costUsd, 0) / results.length;
if (p95 > spec.p95LatencyMs || avgCost > spec.maxCostPerRequestUsd) process.exitCode = 1;
```

```python
p95 = statistics.quantiles([r.latency_ms for r in results], n=100)[94]
avg_cost = sum(r.cost_usd for r in results) / len(results)
if p95 > spec["p95_latency_ms"] or avg_cost > spec["max_cost_per_request_usd"]:
    sys.exit(1)
```

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers ai features playbook: https://agentwranglers.se/playbook/ai-features/evals-and-quality/eval-cost-and-latency`.

- Previous: [Run each case several times](https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs)
- Next: [Watch quality in production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/production-quality-metrics)
- [Evals and quality](https://agentwranglers.se/playbook/ai-features/evals-and-quality)