Agent WranglersSEO Recommended
Tell crawlers where they may go
Serve a plain-text /robots.txt at the root of every host. It allows public content, blocks only private paths and names the sitemap.
How to do it
Follow RFC 9309, the robots.txt standard. Send it as text/plain with status 200, under 500 KiB, which is all Google reads. Start with User-agent: * and Allow: /. Add Disallow: lines for private paths only, and a Sitemap: line with the full sitemap URL. Never block the CSS, scripts or images crawlers need to draw your pages. Disallow stops crawling, not indexing, so use noindex to keep a page out of results. Never let robots.txt answer 5xx. Google then stops crawling the whole site until it gets a good answer.
Example
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /account/
Sitemap: https://example.com/sitemap.xmlHow to check
curl -sI https://example.com/robots.txt shows status 200 and content-type: text/plain. The body has no Disallow: / line for User-agent: *, and its Sitemap: URL answers 200. Search Console's robots.txt report shows no errors.
Measured
On a reference site, removing robots.txt entirely took Lighthouse SEO from 92 to 100. The 92 came only from its Content-Signal lines, so the score does not reward having a robots.txt.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers website playbook: https://agentwranglers.se/playbook/website/seo/robots-txt.