---
title: "Tell crawlers where they may go · Agent Wranglers"
url: https://agentwranglers.se/playbook/website/seo/robots-txt
updated: 2026-10-20
licence: "All rights reserved"
---

# Tell crawlers where they may go

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

- Level: Recommended
- Chapter: [SEO](https://agentwranglers.se/playbook/website/seo)
- Rule: Serve a plain-text `/robots.txt` at the root of every host. It allows public content, blocks only private paths and names the sitemap.
- How: Follow RFC 9309, the robots.txt standard. Send it as `text/plain` with status 200, under 500 KiB, which is all Google reads. Start with `User-agent: *` and `Allow: /`. Add `Disallow:` lines for private paths only, and a `Sitemap:` line with the full sitemap URL. Never block the CSS, scripts or images crawlers need to draw your pages. `Disallow` stops crawling, not indexing, so use `noindex` to keep a page out of results. Never let robots.txt answer 5xx. Google then stops crawling the whole site until it gets a good answer.
- Check: `curl -sI https://example.com/robots.txt` shows status 200 and `content-type: text/plain`. The body has no `Disallow: /` line for `User-agent: *`, and its `Sitemap:` URL answers 200. Search Console's robots.txt report shows no errors.
- Measured: On a reference site, removing robots.txt entirely took Lighthouse SEO from 92 to 100. The 92 came only from its `Content-Signal` lines, so the score does not reward having a robots.txt.

Example:

```text
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml
```

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers website playbook: https://agentwranglers.se/playbook/website/seo/robots-txt`.

- Next: [List every public page in a sitemap](https://agentwranglers.se/playbook/website/seo/xml-sitemaps)
- [SEO](https://agentwranglers.se/playbook/website/seo)