Agent WranglersMeasuring and improving
Track what agents get wrong, add a check for every repeated mistake, measure delivery, and keep a person's sign-off before every release.
- Required 2
- Recommended 4
Required
Add a check for every repeated mistake
RequiredWhen the same mistake appears twice, add a check that catches it, instead of another reminder in the instructions.
Keep a person's sign-off before release
RequiredA person approves each release, or approves the automated gate itself in writing and reviews releases afterwards.
Recommended
Track what agents get wrong
RecommendedRecord each mistake found in review, with its kind, so you can see which ones repeat.
Mark which changes an agent wrote
RecommendedMark every commit or pull request an agent wrote, so you can compare its failure and rework rates with other changes.
Test changes to the agent setup on fixed tasks
RecommendedBefore you change the model, the agent tool, the instructions or the tools, rerun a fixed set of past tasks and compare.
Measure delivery instead of lines of code
RecommendedJudge agent use by how often you ship, how often changes fail and how fast you recover, not by how much code agents write.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers ai development playbook: https://agentwranglers.se/playbook/ai-assisted-development/measuring-and-improving.