Offer one file only while it fits an agent's context

How to do it

Join each page's Markdown, with a heading and its URL above each one. Build it at build time from the same page list as llms.txt. Count its tokens at build time with a common tokenizer. Past the limit, publish one file per section instead, as Split large agent content into named parts behind an index says. A single file is a common convention from documentation tools, not part of the llms.txt proposal. Write the short guide first, as Give AI tools a site guide says.

Examples

Plain text example

# Services
URL: https://example.com/services

We fix leaky roofs, gutters and skylights...

# Contact
URL: https://example.com/contact

Call us on...

JavaScript example

import { encode } from 'gpt-tokenizer';
const SINGLE_FILE_TOKEN_LIMIT = 100_000;
const full = pages.map(p => `# ${p.title}\nURL: ${p.url}\n\n${p.markdown}`).join('\n\n');
if (encode(full).length > SINGLE_FILE_TOKEN_LIMIT) throw new Error('llms-full.txt is too large: publish one file per section');

Python example

import tiktoken

SINGLE_FILE_TOKEN_LIMIT = 100_000
full = "\n\n".join(f"# {p['title']}\nURL: {p['url']}\n\n{p['markdown']}" for p in pages)
if len(tiktoken.get_encoding("o200k_base").encode(full)) > SINGLE_FILE_TOKEN_LIMIT:
    raise SystemExit("llms-full.txt is too large: publish one file per section")

How to check

A test counts the pages in the file, and every public page appears exactly once. The build fails when the file passes 100,000 tokens. Every file linked from llms.txt states its size in tokens.

Related rules

• AGT-48 Split large agent content into named parts behind an index in Agents as users
• AGT-43 Give AI tools a site guide in Agents as users
• AGT-45 Serving one file too large for any agent to read in Agents as users