SEO Required

Keep private pages out of search

Leave public pages open to search engines. Mark error, private, utility and internal search-result responses noindex, with a meta tag or X-Robots-Tag header.

By Erik Bergstedt, Agent Wranglers. Updated 20 October 2026.

How to do it

Public pages need no robots tag, or only preview settings such as max-image-preview:large. Send noindex on error, offline, search-result and private responses. For non-HTML files, use the X-Robots-Tag header. Never block a noindex page in robots.txt, or crawlers never see the tag. Put staging sites behind a password as well, because noindex does not stop anyone from reading them.

Example

<!-- Public page -->
<meta name="robots" content="max-image-preview:large">
<!-- Error, search-result or private page -->
<meta name="robots" content="noindex">

X-Robots-Tag: noindex

How to check

Lighthouse is-crawlable passes on every public page. A test requests an error page, a search-result page and a private page, and each carries noindex in the meta tag or header.

Measured

On a reference site, an accidental noindex dropped Lighthouse SEO from 92 to 61.

For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers website playbook: https://agentwranglers.se/playbook/website/seo/meta-robots.