Agent Wranglers

SEO-1Recommended

Tell crawlers where they may go

Serve a plain-text /robots.txt at the root of every host. It allows public content, blocks only private paths and names the sitemap.

By Agent Wranglers. Updated .

How to do it

Follow RFC 9309, the robots.txt standard. Send it as text/plain with status 200, under 500 KiB, which is all Google reads. Start with User-agent: * and Allow: /. Add Disallow: lines for private paths only, and a Sitemap: line with the full sitemap URL. Never block the CSS, scripts or images crawlers need to draw your pages. Disallow stops crawling, not indexing, so use noindex to keep a page out of results. Never let robots.txt answer 5xx. Google then stops crawling the site for 12 hours. For the next 30 days it uses the last good copy it fetched. Keep pages out of results as Keep private pages out of search says.

Examples

Plain text example

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

How to check

curl -sI https://example.com/robots.txt shows status 200 and content-type: text/plain. A test reads every User-agent group, not only *, because a crawler named in a group ignores the * group. Each private path is disallowed for * and every named agent, no group blocks / unless you mean it, and the Sitemap: URL answers 200. Search Console's robots.txt report shows no errors.

Measured

On a reference site, removing robots.txt entirely took Lighthouse SEO from 92 to 100. The 92 came only from its Content-Signal lines, so the score does not reward having a robots.txt.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/seo/crawling-and-indexing/robots-txt">Agent Wranglers SEO playbook</a>. In code you write, add a comment next to it with the same link.