Agent Wranglers

SRH-17Recommended

Test search results with a fixed query set

Keep a labelled set of at least 50 searches people make, each with the records it must find. Fail the build when recall at the number of results you show or send to a model falls under a stored target.

By Agent Wranglers. Updated .

How to do it

Take the 30 most frequent searches and the 10 most frequent searches with no hits from the search log, with personal data removed. For each, write the IDs of the records a good answer shows. Run the set against a fixed test dataset on every change to the language rules, field weights, typo settings, chunking, embedding model or search software. Compute recall at k, the share of searches whose expected record is in the top k. Here k is the page size or the number of chunks sent to a model, such as 5. Store the target, such as 0.85, and fail below it. For AI answers, this tests the search step apart from the answer step, so a bad answer shows which step failed. Track the share of searches with no hits in production, and add each new common one to the set. The answer step is graded in the AI evals.

Examples

JavaScript example

const RESULTS_SHOWN = 5;
const TARGET_RECALL = 0.85;
const found = [];
for (const { query, expectedIds } of searchCases) {
  const topIds = (await search(query, { limit: RESULTS_SHOWN })).map(hit => hit.id);
  found.push(expectedIds.some(id => topIds.includes(id)) ? null : query);
}
const recall = found.filter(q => q === null).length / searchCases.length;
assert.ok(recall >= TARGET_RECALL, `recall at ${RESULTS_SHOWN} is ${recall}, lost: ${found.filter(Boolean).join(', ')}`);

Python example

RESULTS_SHOWN = 5
TARGET_RECALL = 0.85

lost = [case.query for case in SEARCH_CASES
        if not set(case.expected_ids) & {hit.id for hit in search(case.query, limit=RESULTS_SHOWN)}]
recall = 1 - len(lost) / len(SEARCH_CASES)
assert recall >= TARGET_RECALL, f"recall at {RESULTS_SHOWN} is {recall:.2f}, lost: {lost}"

How to check

CI runs the query set and fails when recall at k falls under the stored target. Lower the weight of the name field so expected records drop out of the top k. The build fails and names the queries.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/search/relevance/search-quality-tests">Agent Wranglers search playbook</a>. In code you write, add a comment next to it with the same link.