Test search results with a fixed query set

How to do it

Take the 30 most frequent searches and the 10 most frequent searches with no hits from the search log, with personal data removed. For each, write the IDs of the records a good answer shows. Run the set against a fixed test dataset on every change to the language rules, field weights, typo settings or search software. Track the share of searches with no hits in production, and add each new common one to the set. Search for AI answers has its own set in Measure retrieval on its own.

Examples

JavaScript example

for (const { query, expectedIds } of searchCases) {
  const topIds = (await search(query, { limit: 10 })).map(hit => hit.id);
  for (const id of expectedIds) assert.ok(topIds.includes(id), `"${query}" lost record ${id}`);
}

Python example

for case in SEARCH_CASES:
    top_ids = [hit.id for hit in search(case.query, limit=10)]
    for expected_id in case.expected_ids:
        assert expected_id in top_ids, f'"{case.query}" lost record {expected_id}'

How to check

CI runs the query set. Lower the weight of the name field so one expected record drops out of the top 10. The build fails and names the query.

Related rules

• Measure retrieval on its own in AI features