---
title: "Tell crawlers where they may go · Agent Wranglers"
url: https://agentwranglers.se/playbook/seo/crawling-and-indexing/robots-txt
id: SEO-1
level: Recommended
updated: 2026-10-22
tokens: 848
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Tell crawlers where they may go

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Crawling and indexing](https://agentwranglers.se/playbook/seo/crawling-and-indexing.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: SEO-1
- Level: Recommended
- Chapter: [Crawling and indexing](https://agentwranglers.se/playbook/seo/crawling-and-indexing.md)
- Rule: Serve a plain-text `/robots.txt` at the root of every host. It allows public content, blocks only private paths and names the sitemap.
- Check: `curl -sI https://example.com/robots.txt` shows status 200 and `content-type: text/plain`. A test reads every `User-agent` group, not only `*`, because a crawler named in a group ignores the `*` group. Each private path is disallowed for `*` and every named agent, no group blocks `/` unless you mean it, and the `Sitemap:` URL answers 200. Search Console's robots.txt report shows no errors.
- How: Follow RFC 9309, the robots.txt standard. Send it as `text/plain` with status 200, under 500 KiB, which is all Google reads. Start with `User-agent: *` and `Allow: /`. Add `Disallow:` lines for private paths only, and a `Sitemap:` line with the full sitemap URL. Never block the CSS, scripts or images crawlers need to draw your pages. `Disallow` stops crawling, not indexing, so use `noindex` to keep a page out of results. Never let robots.txt answer 5xx. Google then stops crawling the site for 12 hours. For the next 30 days it uses the last good copy it fetched. Keep pages out of results as [Keep private pages out of search](https://agentwranglers.se/playbook/seo/crawling-and-indexing/meta-robots.md) says.
- Measured: On a reference site, removing robots.txt entirely took Lighthouse SEO from 92 to 100. The 92 came only from its `Content-Signal` lines, so the score does not reward having a robots.txt.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### Plain text example

```text
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml
```

## Sources

- [RFC 9309 – Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)
- [Google Search Central – How Google interprets robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)

## Related rules

- SEO-9 [Keep private pages out of search](https://agentwranglers.se/playbook/seo/crawling-and-indexing/meta-robots.md) in SEO
- SEO-15 [Keep filter combinations out of the crawl](https://agentwranglers.se/playbook/seo/site-structure/faceted-navigation.md) in SEO

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/seo/crawling-and-indexing/robots-txt">Agent Wranglers SEO playbook</a>`. In code you write, add a comment next to it with the same link.

- Next: [List every public page in a sitemap](https://agentwranglers.se/playbook/seo/crawling-and-indexing/xml-sitemaps.md)
- [Crawling and indexing](https://agentwranglers.se/playbook/seo/crawling-and-indexing.md)