---
title: "Test search results with a fixed query set · Agent Wranglers"
url: https://agentwranglers.se/playbook/search/relevance/search-quality-tests
id: SRH-17
level: Recommended
updated: 2026-10-27
build: a2a6bdfed830
tokens: 1147
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Test search results with a fixed query set

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Relevance](https://agentwranglers.se/playbook/search/relevance.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: SRH-17
- Level: Recommended
- Chapter: [Relevance](https://agentwranglers.se/playbook/search/relevance.md)
- Rule: Keep a labelled set of at least 50 searches people make, each with the records it must find. Fail the build when recall at the number of results you show or send to a model falls under a stored target.
- Check: CI runs the query set and fails when recall at k falls under the stored target. Lower the weight of the name field so expected records drop out of the top k. The build fails and names the queries.
- How: Take the 30 most frequent searches and the 10 most frequent searches with no hits from the search log, with personal data removed. For each, write the IDs of the records a good answer shows. Run the set against a fixed test dataset on every change to the language rules, field weights, typo settings, chunking, embedding model or search software. Compute recall at k, the share of searches whose expected record is in the top k. Here k is the page size or the number of chunks sent to a model, such as 5. Store the target, such as 0.85, and fail below it. For AI answers, this tests the search step apart from the answer step, so a bad answer shows which step failed. Track the share of searches with no hits in production, and add each new common one to the set. The answer step is graded in the [AI evals](https://agentwranglers.se/playbook/ai-features/evals-and-quality/llm-evals.md).

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JavaScript example

```js
const RESULTS_SHOWN = 5;
const TARGET_RECALL = 0.85;
const found = [];
for (const { query, expectedIds } of searchCases) {
  const topIds = (await search(query, { limit: RESULTS_SHOWN })).map(hit => hit.id);
  found.push(expectedIds.some(id => topIds.includes(id)) ? null : query);
}
const recall = found.filter(q => q === null).length / searchCases.length;
assert.ok(recall >= TARGET_RECALL, `recall at ${RESULTS_SHOWN} is ${recall}, lost: ${found.filter(Boolean).join(', ')}`);
```

### Python example

```python
RESULTS_SHOWN = 5
TARGET_RECALL = 0.85

lost = [case.query for case in SEARCH_CASES
        if not set(case.expected_ids) & {hit.id for hit in search(case.query, limit=RESULTS_SHOWN)}]
recall = 1 - len(lost) / len(SEARCH_CASES)
assert recall >= TARGET_RECALL, f"recall at {RESULTS_SHOWN} is {recall:.2f}, lost: {lost}"
```

## Related rules

- SRH-2 [Search by meaning and by words](https://agentwranglers.se/playbook/search/engines-and-indexes/hybrid-search.md) in Search
- SRH-3 [Search by meaning when the collection is small](https://agentwranglers.se/playbook/search/engines-and-indexes/small-semantic-search.md) in Search
- SRH-5 [Split documents along their structure](https://agentwranglers.se/playbook/search/engines-and-indexes/structured-chunks.md) in Search
- SRH-9 [Rebuild an index under a new name and switch an alias](https://agentwranglers.se/playbook/search/index-contents-and-sync/reindex-with-alias.md) in Search
- SRH-12 [Keep personal data out of search logs](https://agentwranglers.se/playbook/search/access-and-limits/search-log-privacy.md) in Search
- SRH-15 [Match how people ask as well as title words](https://agentwranglers.se/playbook/search/language-and-matching/match-plain-questions.md) in Search
- AIF-86 [Grade AI features against fixed examples](https://agentwranglers.se/playbook/ai-features/evals-and-quality/llm-evals.md) in AI features
- SRH-19 [Watch searches that find nothing or get no click](https://agentwranglers.se/playbook/search/relevance/zero-result-searches.md) in Search

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/search/relevance/search-quality-tests">Agent Wranglers search playbook</a>`. In code you write, add a comment next to it with the same link.

- Next: [Say when nothing fits](https://agentwranglers.se/playbook/search/relevance/say-when-nothing-fits.md)
- [Relevance](https://agentwranglers.se/playbook/search/relevance.md)