---
title: "Tool choice evals · Agent Wranglers"
url: https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals
parts: 4
tokens_total: 3212
updated: 2026-10-22
tokens: 604
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Tool choice evals

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Agents as users](https://agentwranglers.se/playbook/agents-as-users.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

> Keep a fixed set of tasks with the tools they need. Run it on several models, gate releases on the score and rerun it when a tool definition changes.

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals">Agent Wranglers agents as users playbook</a>`. In code you write, add a comment next to it with the same link.

## Recommended

- AGT-28 [Keep a set of tasks with the tools they need](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-choice-cases.md) · 906 tokens · updated 2026-10-22
  Keep at least 50 tasks in plain words, each with the tool and the key arguments a good agent would use. Include tasks where the right move is no tool, or a question back.
- AGT-29 [Run the tool evals on more than one model](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-evals-several-models.md) · 715 tokens · updated 2026-10-22
  Run the task set against at least two model families, and against the clients most of your agent traffic comes from. Run each task at least three times.
- AGT-30 [Gate releases on tool choice scores](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-choice-gate.md) · 933 tokens · updated 2026-10-22
  Score each run by the share of tasks with the right tool, the share with the right arguments, and the calls per task. A release fails when either share drops more than 2 points below the last release.
- AGT-31 [Rerun the tool evals on every tool change](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/evals-on-tool-change.md) · 658 tokens · updated 2026-10-22
  Run the tool evals in CI on every change to a tool's name, description or schema, and on every change to the set of tools.

- Previous: [Quotas and pricing](https://agentwranglers.se/playbook/agents-as-users/quotas-and-pricing.md)
- [All chapters](https://agentwranglers.se/playbook/agents-as-users.md)