State the size of every agent file in tokens

How to do it

Write the count next to each link in llms.txt, in a tokens field in JSON indexes, and in each Markdown copy's front matter. Send it in a response header such as X-Markdown-Tokens too, so a HEAD request is enough. Count at build time with a common tokenizer. Counts differ by a few percent between tokenizers, which is close enough to plan a budget. The size in bytes from Give AI tools a site guide stays useful for people.

Examples

Plain text example

- [Payments](https://example.com/guide/payments.md): Taking money online (6,200 tokens)
- [Refunds](https://example.com/guide/refunds.md): Partial and full refunds (2,100 tokens)

JavaScript example

const tokens = encode(markdown).length;
res.set({ 'Content-Type': 'text/markdown; charset=utf-8', 'X-Markdown-Tokens': String(tokens) });

Python example

tokens = len(encoding.encode(markdown))
return Response(markdown, media_type="text/markdown", headers={"X-Markdown-Tokens": str(tokens)})

How to check

A script fetches every agent file. The stated count is within 10% of a count made with a common tokenizer.

Related rules

• WEB-184 Give AI tools a site guide in Website
• WEB-259 Layer agent content with a budget per layer in Website
• WEB-261 Put the answer first in every Markdown copy in Website