---
title: "Rerun the tool evals on every tool change · Agent Wranglers"
url: https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/evals-on-tool-change
id: AGT-31
level: Recommended
updated: 2026-10-22
tokens: 700
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Rerun the tool evals on every tool change

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Tool choice evals](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: AGT-31
- Level: Recommended
- Chapter: [Tool choice evals](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals.md)
- Rule: Run the tool evals in CI on every change to a tool's name, description or schema, and on every change to the set of tools.
- Check: The CI configuration starts the eval job on changes under the tool definition paths, and branch protection requires that job. A test change to one description starts it.
- How: A one-word change to a description can move traffic from one tool to another. Keep tool definitions in their own files, so CI can tell when they change. Trigger the eval job on those paths, and add a check that the job ran before merge. A changed tool list also changes its size, which [Keep the tool list within a token budget](https://agentwranglers.se/playbook/agents-as-users/mcp-servers/tool-list-budget.md) limits. The gate itself is [Gate releases on tool choice scores](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-choice-gate.md).

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### YAML example

```yaml
on:
  pull_request:
    paths: ["src/tools/**", "evals/tool-choice/**"]
jobs:
  tool-choice-evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run evals:tools
```

## Related rules

- AGT-15 [Keep the tool list within a token budget](https://agentwranglers.se/playbook/agents-as-users/mcp-servers/tool-list-budget.md) in Agents as users
- AGT-30 [Gate releases on tool choice scores](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-choice-gate.md) in Agents as users

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/evals-on-tool-change">Agent Wranglers agents as users playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Gate releases on tool choice scores](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-choice-gate.md)
- [Tool choice evals](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals.md)