Keep a set of tasks with the tools they need

How to do it

You do not control the agents that call your tools, but you can measure whether a model picks the right one from your descriptions. Write tasks the way people phrase them, such as "move my Friday table to 8 pm", not with your tool names. Give each one the expected tool, the arguments that must match and the tools that must not be called. Add at least 10 tasks the tools cannot do, where the right answer is to say so. Grow the set from failed calls in production, as Grow the eval set from production says. Grade with code, as Grade with code before a grading model says.

Examples

JSON example

[
  { "task": "Move my Friday table to 8 pm", "expect": { "tool": "change_booking", "args": { "time": "20:00" } }, "never": ["cancel_booking"] },
  { "task": "Do you have vegan options?", "expect": { "tool": "search_menu", "args": { "diet": "vegan" } } },
  { "task": "Book me a flight to Rome", "expect": { "tool": null } }
]

JavaScript example

test('every expected tool exists', async () => {
  const names = new Set((await client.listTools()).tools.map(tool => tool.name));
  cases.filter(c => c.expect.tool).forEach(c => expect(names.has(c.expect.tool)).toBe(true));
});

Python example

def test_every_expected_tool_exists(client, cases):
    names = {tool["name"] for tool in client.list_tools()["tools"]}
    assert all(case["expect"]["tool"] in names for case in cases if case["expect"]["tool"])

How to check

The case file has at least 50 tasks, at least 10 of them with no expected tool. A test confirms every expected tool name exists in the current tools/list.

Related rules

• AGT-15 Keep the tool list within a token budget in Agents as users
• AIF-26 Grow the eval set from production in AI features
• AIF-27 Grade with code before a grading model in AI features