Measure retrieval on its own

How to do it

Write at least 50 questions, each with the IDs of the documents that answer it. Run only the search and compute recall at the number of chunks you send to the model, such as recall at 5. Bad answers often come from bad search, and this test shows which step failed. Run it on every change to chunking, embedding model or search settings. Keep the answer step in the AI evals.

Example

const recallAt = (k: number) => cases.filter(c => search(c.question, k).some(hit => c.expectedDocumentIds.includes(hit.documentId))).length / cases.length;
if (recallAt(5) < TARGET_RECALL_AT_5) throw new Error(`Recall at 5 is under ${TARGET_RECALL_AT_5}`);

def recall_at(k: int) -> float:
    found = sum(any(hit.document_id in case.expected_document_ids for hit in search(case.question, k)) for case in cases)
    return found / len(cases)

assert recall_at(5) >= TARGET_RECALL_AT_5, f"recall at 5 is under {TARGET_RECALL_AT_5}"

How to check

CI runs the retrieval set and fails when recall at 5 falls under the stored target, such as 0.85.