Measure how much an agent reads to find an answer

How to do it

Write about 30 questions the way people ask them, taken from search logs and support mail. Run them in CI with a scripted agent that starts at llms.txt and follows links. Record tokens read and fetches made per question, and keep the results per release. A rise shows where the layers or the links lost their way. Pair it with Match how people ask as well as title words.

Examples

JavaScript example

const MAX_TOKENS = 5000, MAX_FETCHES = 3;
for (const { question, expected } of questions) {
  const run = await scriptedAgent.find(question, { start: 'https://example.com/llms.txt' });
  assert(run.reached === expected && run.tokens < MAX_TOKENS && run.fetches <= MAX_FETCHES, question);
}

Python example

MAX_TOKENS, MAX_FETCHES = 5000, 3
for case in questions:
    run = scripted_agent.find(case["question"], start="https://example.com/llms.txt")
    assert run.reached == case["expected"] and run.tokens < MAX_TOKENS and run.fetches <= MAX_FETCHES, case["question"]

How to check

Each question reaches its expected page in under 5,000 tokens read and three fetches or fewer. CI keeps the numbers for each release.

Related rules

• WEB-247 Match how people ask as well as title words in Website