---
title: "State the size of every agent file in · Agent Wranglers"
url: https://agentwranglers.se/playbook/website/ready-for-ai-agents/token-counts
id: WEB-258
level: Recommended
updated: 2026-10-22
tokens: 802
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# State the size of every agent file in tokens

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Ready for AI agents](https://agentwranglers.se/playbook/website/ready-for-ai-agents.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: WEB-258
- Level: Recommended
- Chapter: [Ready for AI agents](https://agentwranglers.se/playbook/website/ready-for-ai-agents.md)
- Rule: Every file for agents states its size in tokens, so an agent knows the cost before it fetches the file.
- Check: A script fetches every agent file. The stated count is within 10% of a count made with a common tokenizer.
- How: Write the count next to each link in llms.txt, in a `tokens` field in JSON indexes, and in each Markdown copy's front matter. Send it in a response header such as `X-Markdown-Tokens` too, so a `HEAD` request is enough. Count at build time with a common tokenizer. Counts differ by a few percent between tokenizers, which is close enough to plan a budget. The size in bytes from [Give AI tools a site guide](https://agentwranglers.se/playbook/website/ready-for-ai-agents/llms-txt.md) stays useful for people.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### Plain text example

```text
- [Payments](https://example.com/guide/payments.md): Taking money online (6,200 tokens)
- [Refunds](https://example.com/guide/refunds.md): Partial and full refunds (2,100 tokens)
```

### JavaScript example

```js
const tokens = encode(markdown).length;
res.set({ 'Content-Type': 'text/markdown; charset=utf-8', 'X-Markdown-Tokens': String(tokens) });
```

### Python example

```python
tokens = len(encoding.encode(markdown))
return Response(markdown, media_type="text/markdown", headers={"X-Markdown-Tokens": str(tokens)})
```

## Related rules

- WEB-184 [Give AI tools a site guide](https://agentwranglers.se/playbook/website/ready-for-ai-agents/llms-txt.md) in Website
- WEB-259 [Layer agent content with a budget per layer](https://agentwranglers.se/playbook/website/ready-for-ai-agents/layer-budgets.md) in Website
- WEB-261 [Put the answer first in every Markdown copy](https://agentwranglers.se/playbook/website/ready-for-ai-agents/answer-first-markdown.md) in Website

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/website/ready-for-ai-agents/token-counts">Agent Wranglers website playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Serving one file too large for any agent to read](https://agentwranglers.se/playbook/website/ready-for-ai-agents/oversized-agent-file.md)
- Next: [Layer agent content with a budget per layer](https://agentwranglers.se/playbook/website/ready-for-ai-agents/layer-budgets.md)
- [Ready for AI agents](https://agentwranglers.se/playbook/website/ready-for-ai-agents.md)