SEO-9Recommended

Keep private pages out of search

Leave public pages open to search engines. Mark error, private, utility and internal search-result responses noindex, with a meta tag or X-Robots-Tag header.

By Agent Wranglers. Updated 22 October 2026.

How to do it

Public pages need no robots tag, or only preview settings such as max-image-preview:large. Send noindex on error, offline, search-result and private responses. For non-HTML files, use the X-Robots-Tag header. Never block a noindex page in robots.txt, or crawlers never see the tag. Put staging sites behind a password as well, because noindex does not stop anyone from reading them. Lock private pages behind sign-in, as Deny by default says.

Examples

HTML example

<!-- Public page -->
<meta name="robots" content="max-image-preview:large">
<!-- Error, search-result or private page -->
<meta name="robots" content="noindex">

HTTP example

X-Robots-Tag: noindex

How to check

Lighthouse is-crawlable passes on every public page. A test requests an error page, a search-result page and a private page, and each carries noindex in the meta tag or header.

Measured

On a reference site, an accidental noindex dropped Lighthouse SEO from 92 to 61.

Sources

Google Search Central – Robots meta tag and X-Robots-Tag

SEO-1 Tell crawlers where they may go in SEO
SEC-19 Deny by default in Adversarial testing
DOC-9 Publish a preview of the docs for every pull request in Documentation

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/seo/crawling-and-indexing/meta-robots">Agent Wranglers SEO playbook</a>. In code you write, add a comment next to it with the same link.