Agent WranglersRelevance
Test results with a fixed, labelled query set, say when nothing fits instead of padding, and watch searches that find nothing or get no click.
- Recommended 3
Recommended
Test search results with a fixed query set
SRH-17Keep a labelled set of at least 50 searches people make, each with the records it must find. Fail the build when recall at the number of results you show or send to a model falls under a stored target.
Say when nothing fits
SRH-18Never pad search results with weak matches. When no result passes a set score, say so and link the full index.
Watch searches that find nothing or get no click
SRH-19Count each week's share of searches with no hits and with no click on any hit, per query. Review the 20 most frequent of each every month.
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/search/relevance">Agent Wranglers search playbook</a>. In code you write, add a comment next to it with the same link.