Measure cost and speed in every eval run

How to do it

Read token counts from the provider's usage fields and price them from one price table in the gateway. Report the median and 95th percentile latency and the average cost per case. A prompt change that adds a long context can keep the score and double the bill. This check catches it before release.

Example

const p95 = percentile(results.map(r => r.latencyMs), 95);
const avgCost = results.reduce((sum, r) => sum + r.costUsd, 0) / results.length;
if (p95 > spec.p95LatencyMs || avgCost > spec.maxCostPerRequestUsd) process.exitCode = 1;

p95 = statistics.quantiles([r.latency_ms for r in results], n=100)[94]
avg_cost = sum(r.cost_usd for r in results) / len(results)
if p95 > spec["p95_latency_ms"] or avg_cost > spec["max_cost_per_request_usd"]:
    sys.exit(1)

How to check

The eval report shows cost and latency per case. On a branch, set the spec's cost limit to half the last measured average cost, and the eval job fails on cost.