Keep the index as fresh as its sources

How to do it

Index on change events from the source, not only in a nightly job. Delete chunks by document ID when a document is deleted or unpublished. Store the source's last-modified time with each chunk and show it with the citation. Run a full rebuild on a schedule to catch missed events. The general method is in keep search indexes in step.

Example

SELECT c.document_id FROM chunks c LEFT JOIN documents d ON d.id = c.document_id
WHERE d.id IS NULL OR (c.source_updated_at < d.updated_at AND d.updated_at < now() - interval '15 minutes');
-- PASS when no rows are returned

documents.on('deleted', async ({ id }) => vectorIndex.deleteWhere({ documentId: id }));

@documents.on("deleted")
def remove_from_index(event: DocumentDeleted) -> None:
    vector_index.delete_where(document_id=event.id)

How to check

Delete a test document at the source. Within the set time, a search for its unique phrase returns nothing. A nightly job compares document IDs in the source and the index and alerts on any difference.