Search by meaning and by words

How to do it

Vector search finds text with the same meaning in other words. Keyword search, such as BM25, finds exact product names, codes and error messages that vectors often miss. Run both, take about 20 results from each, and merge them by rank. Reciprocal rank fusion scores each result as the sum of 1 divided by 60 plus its rank in each list. Optionally rerank the top results with a reranking model. Keep the merge only if it scores better in the retrieval evals.

Example

const RRF_K = 60;

export function reciprocalRankFusion(...rankings: string[][]): string[] {
  const scores = new Map<string, number>();
  for (const ranking of rankings) ranking.forEach((id, rank) => scores.set(id, (scores.get(id) ?? 0) + 1 / (RRF_K + rank + 1)));
  return [...scores.entries()].sort((a, b) => b[1] - a[1]).map(([id]) => id);
}

RRF_K = 60

def reciprocal_rank_fusion(*rankings: list[str]) -> list[str]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, chunk_id in enumerate(ranking, start=1):
            scores[chunk_id] = scores.get(chunk_id, 0) + 1 / (RRF_K + rank)
    return sorted(scores, key=scores.get, reverse=True)

How to check

The retrieval eval set includes questions with exact product codes. Hybrid search scores at least as well as vector search alone on the whole set.