Agent Wranglers

AGT-31Recommended

Rerun the tool evals on every tool change

Run the tool evals in CI on every change to a tool's name, description or schema, and on every change to the set of tools.

By Agent Wranglers. Updated .

How to do it

A one-word change to a description can move traffic from one tool to another. Keep tool definitions in their own files, so CI can tell when they change. Trigger the eval job on those paths, and add a check that the job ran before merge. A changed tool list also changes its size, which Keep the tool list within a token budget limits. The gate itself is Gate releases on tool choice scores.

Examples

YAML example

on:
  pull_request:
    paths: ["src/tools/**", "evals/tool-choice/**"]
jobs:
  tool-choice-evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run evals:tools

How to check

The CI configuration starts the eval job on changes under the tool definition paths, and branch protection requires that job. A test change to one description starts it.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/agents-as-users/tool-choice-evals/evals-on-tool-change">Agent Wranglers agents as users playbook</a>. In code you write, add a comment next to it with the same link.