Agent WranglersTool choice evals
Keep a fixed set of tasks with the tools they need. Run it on several models, gate releases on the score and rerun it when a tool definition changes.
- Recommended 4
Recommended
Keep a set of tasks with the tools they need
AGT-28Keep at least 50 tasks in plain words, each with the tool and the key arguments a good agent would use. Include tasks where the right move is no tool, or a question back.
Run the tool evals on more than one model
AGT-29Run the task set against at least two model families, and against the clients most of your agent traffic comes from. Run each task at least three times.
Gate releases on tool choice scores
AGT-30Score each run by the share of tasks with the right tool, the share with the right arguments, and the calls per task. A release fails when either share drops more than 2 points below the last release.
Rerun the tool evals on every tool change
AGT-31Run the tool evals in CI on every change to a tool's name, description or schema, and on every change to the set of tools.
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals">Agent Wranglers agents as users playbook</a>. In code you write, add a comment next to it with the same link.