---
title: "Run the tool evals on more than one model · Agent Wranglers"
url: https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-evals-several-models
id: AGT-29
level: Recommended
updated: 2026-10-22
tokens: 753
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Run the tool evals on more than one model

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Tool choice evals](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: AGT-29
- Level: Recommended
- Chapter: [Tool choice evals](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals.md)
- Rule: Run the task set against at least two model families, and against the clients most of your agent traffic comes from. Run each task at least three times.
- Check: The eval report lists at least two model families, with every task run at least three times on each. The three clients that sent the most MCP requests in the last 30 days appear in the report.
- How: Descriptions that one model reads well can confuse another, and your callers choose their own models. Read the client names from `io.modelcontextprotocol/clientInfo` in your logs to see which clients matter. Give each model only your tool list and the task, with no hints. Models answer differently each run, so repeat each task as [Run each case several times](https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs.md) says. Report each model's result on its own line, never only the average.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JavaScript example

```js
const MODELS = ['provider-a/large', 'provider-b/large'];
const RUNS_PER_TASK = 3;
for (const model of MODELS) for (const c of cases) for (let run = 0; run < RUNS_PER_TASK; run++) results.push(await runTask(model, tools, c));
```

### Python example

```python
MODELS = ["provider-a/large", "provider-b/large"]
RUNS_PER_TASK = 3
results = [run_task(model, tools, case) for model in MODELS for case in cases for _ in range(RUNS_PER_TASK)]
```

## Related rules

- AIF-28 [Run each case several times](https://agentwranglers.se/playbook/ai-features/evals-and-quality/repeat-eval-runs.md) in AI features

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-evals-several-models">Agent Wranglers agents as users playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Keep a set of tasks with the tools they need](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-choice-cases.md)
- Next: [Gate releases on tool choice scores](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-choice-gate.md)
- [Tool choice evals](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals.md)