Agent WranglersMeasuring and improving Recommended
Test changes to the agent setup on fixed tasks
Before you change the model, the agent tool, the instructions or the tools, rerun a fixed set of past tasks and compare.
How to do it
Pick ten to thirty finished tasks from your history. Store each one as a starting commit, the prompt and the check command that proves it done. Run the set on the old and the new setup in a sandbox. Record how many pass and what each run cost. Keep the new setup only if the pass rate is the same or higher. See AI evals for the same method on product features.
How to check
The last change to the agent setup has a run attached. It states the pass rate, such as from 22 of 25 to 23 of 25.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers ai development playbook: https://agentwranglers.se/playbook/ai-assisted-development/measuring-and-improving/evaluate-agent-setup.