Retrieval

Required

• Record the embedding model with every vector

  Required

  Store the embedding model name and version with every vector. Never compare vectors from two different embedding models.

• Measure retrieval on its own

  Required

  Test the search step apart from the answer step, with labelled questions and the documents that should be found. Fail the build when recall falls under a stored target.

• Cite a source for every answer

  Required

  Make the model return the IDs of the chunks it used, and check each ID was in the retrieved set. Show the sources as links next to the answer.

• Keep the index as fresh as its sources

  Required

  Update or remove a document's chunks within a set time after the source changes, such as 15 minutes. Show the source date with every answer.

• Copy access rights into the index and check them again

  Required

  Store each document's access rights with its chunks, filter every search by them, and re-check the source's rights before a chunk reaches the model.

Recommended

• Split documents along their structure

  Recommended

  Split documents into chunks along headings and paragraphs, of 200 to 800 tokens each. Store each chunk with its document ID, title, heading path and address.

• Search by meaning and by words

  Recommended

  Combine a keyword search and a vector search, and merge the two result lists, for example with reciprocal rank fusion.

• Say so when nothing relevant was found

  Recommended

  When search finds nothing above a set relevance threshold, reply that the documents do not cover the question. Do not let the model answer from its general knowledge.