Agent WranglersDeciding to use a model Required
Define success before you build
Before building an AI feature, write down its task, its target score, its worst possible mistake, its cost limit per use and its latency limit.
How to do it
Write a one-page spec. Name the task, the users and the inputs. Set a target score on an eval set, such as "90 of 100 answers correct". Name the failure that hurts most, such as "promises a refund we do not give". Set its limit, such as zero in the eval set. Set a cost ceiling per request and a latency limit, such as 3 seconds at the 95th percentile. A streamed answer also gets a limit on time to the first token. The AI evals rule turns the score into a test.
Example
# ai/specs/return-assistant.yaml
task: Answer return and refund questions for Fernwood Bikes customers
target_score: 0.90
worst_failure: Promises a refund outside the 30-day policy
worst_failure_limit: 0
max_cost_per_request_usd: 0.01
p95_latency_ms: 3000
p95_first_token_ms: 1500How to check
The spec file exists next to the feature's code and has a target score, a worst-failure limit, a cost per request and a latency limit. CI reads the target score from it.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers ai features playbook: https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/define-ai-success.