---
title: "Gate releases on tool choice scores · Agent Wranglers"
url: https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-choice-gate
id: AGT-30
level: Recommended
updated: 2026-10-22
tokens: 974
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Gate releases on tool choice scores

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Tool choice evals](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: AGT-30
- Level: Recommended
- Chapter: [Tool choice evals](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals.md)
- Rule: Score each run by the share of tasks with the right tool, the share with the right arguments, and the calls per task. A release fails when either share drops more than 2 points below the last release.
- Check: The CI job writes the three scores per model to a file kept with the release. It fails when the right-tool or right-argument share falls more than 2 points below the stored scores of the last release.
- How: The right tool counts the tasks where the first call was the expected tool, or no call where none was expected. The right arguments counts those where the key arguments matched too. Calls per task shows agents that wander through several tools before the right one. Store the scores per release, as [Measure cost and speed in every eval run](https://agentwranglers.se/playbook/ai-features/evals-and-quality/eval-cost-and-latency.md) does for cost. A drop sends the change back for better descriptions or fewer, clearer tools.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JavaScript example

```js
const MAX_DROP_POINTS = 2;
export const score = results => ({
  rightTool: 100 * results.filter(r => r.firstTool === r.expect.tool).length / results.length,
  rightArgs: 100 * results.filter(r => r.firstTool === r.expect.tool && r.argsMatch).length / results.length,
  callsPerTask: results.reduce((sum, r) => sum + r.calls, 0) / results.length,
});
export const regressed = (now, last) => now.rightTool < last.rightTool - MAX_DROP_POINTS || now.rightArgs < last.rightArgs - MAX_DROP_POINTS;
```

### Python example

```python
MAX_DROP_POINTS = 2

def score(results: list[dict]) -> dict:
    total = len(results)
    right_tool = [r for r in results if r["first_tool"] == r["expect"]["tool"]]
    return {"right_tool": 100 * len(right_tool) / total,
            "right_args": 100 * sum(r["args_match"] for r in right_tool) / total,
            "calls_per_task": sum(r["calls"] for r in results) / total}

def regressed(now: dict, last: dict) -> bool:
    return now["right_tool"] < last["right_tool"] - MAX_DROP_POINTS or now["right_args"] < last["right_args"] - MAX_DROP_POINTS
```

## Related rules

- AIF-29 [Measure cost and speed in every eval run](https://agentwranglers.se/playbook/ai-features/evals-and-quality/eval-cost-and-latency.md) in AI features
- AGT-31 [Rerun the tool evals on every tool change](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/evals-on-tool-change.md) in Agents as users

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-choice-gate">Agent Wranglers agents as users playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Run the tool evals on more than one model](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-evals-several-models.md)
- Next: [Rerun the tool evals on every tool change](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/evals-on-tool-change.md)
- [Tool choice evals](https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals.md)