Agent Wranglers

SRH-5Recommended

Split documents along their structure

Split documents into chunks along headings and paragraphs, of 200 to 800 tokens each. Store each chunk with its document ID, title, heading path and address.

By Agent Wranglers. Updated .

How to do it

Split at headings first, then at paragraphs, and never in the middle of a sentence, table row or code block. Add a small overlap, such as one sentence, only where a section must be cut. Prefix each chunk's text with its document title and heading path before embedding, so a chunk about "Returns" still says which product it covers. Keep the address, so answers can link to the source. Measure each change to the chunking as Measure retrieval on its own says.

Examples

TypeScript example

type Chunk = { documentId: string; url: string; headingPath: string[]; text: string };

export function chunkByHeadings(doc: { id: string; url: string; title: string; sections: Section[] }): Chunk[] {
  return doc.sections.flatMap(section =>
    splitParagraphs(section.paragraphs, { maxTokens: 800 }).map(text => ({
      documentId: doc.id, url: `${doc.url}#${section.anchor}`, headingPath: [doc.title, ...section.headings],
      text: `${[doc.title, ...section.headings].join(' > ')}\n\n${text}`,
    })));
}

Python example

def chunk_by_headings(doc: Document) -> list[Chunk]:
    chunks = []
    for section in doc.sections:
        path = [doc.title, *section.headings]
        for text in split_paragraphs(section.paragraphs, max_tokens=800):
            chunks.append(Chunk(document_id=doc.id, url=f"{doc.url}#{section.anchor}",
                                heading_path=path, text=" > ".join(path) + "\n\n" + text))
    return chunks

How to check

A test chunks the sample documents. Every chunk has 200 to 800 tokens or is a whole short section. Each ends at a sentence end and has a document ID, heading path and address.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/search/engines-and-indexes/structured-chunks">Agent Wranglers search playbook</a>. In code you write, add a comment next to it with the same link.