---
title: "Keep a set of tasks with the tools they · Agent Wranglers"
url: https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-choice-cases
id: AGT-28
level: Recommended
updated: 2026-10-22
tokens: 951
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Keep a set of tasks with the tools they need

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Tool choice evals](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: AGT-28
- Level: Recommended
- Chapter: [Tool choice evals](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals.md)
- Rule: Keep at least 50 tasks in plain words, each with the tool and the key arguments a good agent would use. Include tasks where the right move is no tool, or a question back.
- Check: The case file has at least 50 tasks, at least 10 of them with no expected tool. A test confirms every expected tool name exists in the current `tools/list`.
- How: You do not control the agents that call your tools, but you can measure whether a model picks the right one from your descriptions. Write tasks the way people phrase them, such as "move my Friday table to 8 pm", not with your tool names. Give each one the expected tool, the arguments that must match and the tools that must not be called. Add at least 10 tasks the tools cannot do, where the right answer is to say so. Grow the set from failed calls in production, as [Grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production.md) says. Grade with code, as [Grade with code before a grading model](https://agentwranglers.se/playbook/ai-features/evals-and-quality/code-graders-first.md) says.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JSON example

```json
[
  { "task": "Move my Friday table to 8 pm", "expect": { "tool": "change_booking", "args": { "time": "20:00" } }, "never": ["cancel_booking"] },
  { "task": "Do you have vegan options?", "expect": { "tool": "search_menu", "args": { "diet": "vegan" } } },
  { "task": "Book me a flight to Rome", "expect": { "tool": null } }
]
```

### JavaScript example

```js
test('every expected tool exists', async () => {
  const names = new Set((await client.listTools()).tools.map(tool => tool.name));
  cases.filter(c => c.expect.tool).forEach(c => expect(names.has(c.expect.tool)).toBe(true));
});
```

### Python example

```python
def test_every_expected_tool_exists(client, cases):
    names = {tool["name"] for tool in client.list_tools()["tools"]}
    assert all(case["expect"]["tool"] in names for case in cases if case["expect"]["tool"])
```

## Related rules

- AGT-15 [Keep the tool list within a token budget](https://agentwranglers.se/playbook/agents-as-users/mcp-servers/tool-list-budget.md) in Agents as users
- AIF-26 [Grow the eval set from production](https://agentwranglers.se/playbook/ai-features/evals-and-quality/evals-from-production.md) in AI features
- AIF-27 [Grade with code before a grading model](https://agentwranglers.se/playbook/ai-features/evals-and-quality/code-graders-first.md) in AI features

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-choice-cases">Agent Wranglers agents as users playbook</a>`. In code you write, add a comment next to it with the same link.

- Next: [Run the tool evals on more than one model](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-evals-several-models.md)
- [Tool choice evals](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals.md)