Agent Wranglers

AGT-30Recommended

Gate releases on tool choice scores

Score each run by the share of tasks with the right tool, the share with the right arguments, and the calls per task. A release fails when either share drops more than 2 points below the last release.

By Agent Wranglers. Updated .

How to do it

The right tool counts the tasks where the first call was the expected tool, or no call where none was expected. The right arguments counts those where the key arguments matched too. Calls per task shows agents that wander through several tools before the right one. Store the scores per release, as Measure cost and speed in every eval run does for cost. A drop sends the change back for better descriptions or fewer, clearer tools.

Examples

JavaScript example

const MAX_DROP_POINTS = 2;
export const score = results => ({
  rightTool: 100 * results.filter(r => r.firstTool === r.expect.tool).length / results.length,
  rightArgs: 100 * results.filter(r => r.firstTool === r.expect.tool && r.argsMatch).length / results.length,
  callsPerTask: results.reduce((sum, r) => sum + r.calls, 0) / results.length,
});
export const regressed = (now, last) => now.rightTool < last.rightTool - MAX_DROP_POINTS || now.rightArgs < last.rightArgs - MAX_DROP_POINTS;

Python example

MAX_DROP_POINTS = 2

def score(results: list[dict]) -> dict:
    total = len(results)
    right_tool = [r for r in results if r["first_tool"] == r["expect"]["tool"]]
    return {"right_tool": 100 * len(right_tool) / total,
            "right_args": 100 * sum(r["args_match"] for r in right_tool) / total,
            "calls_per_task": sum(r["calls"] for r in results) / total}

def regressed(now: dict, last: dict) -> bool:
    return now["right_tool"] < last["right_tool"] - MAX_DROP_POINTS or now["right_args"] < last["right_args"] - MAX_DROP_POINTS

How to check

The CI job writes the three scores per model to a file kept with the release. It fails when the right-tool or right-argument share falls more than 2 points below the stored scores of the last release.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/tool-choice-gate">Agent Wranglers agents as users playbook</a>. In code you write, add a comment next to it with the same link.