---
title: "Test search results with a fixed query set · Agent Wranglers"
url: https://agentwranglers.se/playbook/data/search/search-quality-tests
updated: 2026-10-21
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Test search results with a fixed query set

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Search](https://agentwranglers.se/playbook/data/search.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers). Updated 2026-10-21.

- ID: DAT-65
- Level: Recommended
- Chapter: [Search](https://agentwranglers.se/playbook/data/search)
- Rule: Keep a set of the searches people make, each with the records it must find. Fail the build when a change pushes an expected record out of the top 10.
- How: Take the 30 most frequent searches and the 10 most frequent searches with no hits from the search log, with personal data removed. For each, write the IDs of the records a good answer shows. Run the set against a fixed test dataset on every change to the language rules, field weights, typo settings or search software. Track the share of searches with no hits in production, and add each new common one to the set. Search for AI answers has its own set in [Measure retrieval on its own](https://agentwranglers.se/playbook/ai-features/retrieval/retrieval-evals).
- Check: CI runs the query set. Lower the weight of the name field so one expected record drops out of the top 10. The build fails and names the query.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JavaScript example

```js
for (const { query, expectedIds } of searchCases) {
  const topIds = (await search(query, { limit: 10 })).map(hit => hit.id);
  for (const id of expectedIds) assert.ok(topIds.includes(id), `"${query}" lost record ${id}`);
}
```

### Python example

```python
for case in SEARCH_CASES:
    top_ids = [hit.id for hit in search(case.query, limit=10)]
    for expected_id in case.expected_ids:
        assert expected_id in top_ids, f'"{case.query}" lost record {expected_id}'
```

## Related rules

- [Measure retrieval on its own](https://agentwranglers.se/playbook/ai-features/retrieval/retrieval-evals) in AI features

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/data/search/search-quality-tests">Agent Wranglers data playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Cap the size of every search](https://agentwranglers.se/playbook/data/search/search-limits)
- [Search](https://agentwranglers.se/playbook/data/search)