---
title: "Keep private pages out of search · Agent Wranglers"
url: https://agentwranglers.se/playbook/website/seo/meta-robots
updated: 2026-10-20
licence: "All rights reserved"
---

# Keep private pages out of search

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

- Level: Required
- Chapter: [SEO](https://agentwranglers.se/playbook/website/seo)
- Rule: Leave public pages open to search engines. Mark error, private, utility and internal search-result responses `noindex`, with a meta tag or `X-Robots-Tag` header.
- How: Public pages need no robots tag, or only preview settings such as `max-image-preview:large`. Send `noindex` on error, offline, search-result and private responses. For non-HTML files, use the `X-Robots-Tag` header. Never block a `noindex` page in robots.txt, or crawlers never see the tag. Put staging sites behind a password as well, because `noindex` does not stop anyone from reading them.
- Check: Lighthouse `is-crawlable` passes on every public page. A test requests an error page, a search-result page and a private page, and each carries `noindex` in the meta tag or header.
- Measured: On a reference site, an accidental `noindex` dropped Lighthouse SEO from 92 to 61.

Example:

```html
<!-- Public page -->
<meta name="robots" content="max-image-preview:large">
<!-- Error, search-result or private page -->
<meta name="robots" content="noindex">
```

```http
X-Robots-Tag: noindex
```

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers website playbook: https://agentwranglers.se/playbook/website/seo/meta-robots`.

- Previous: [Serve phones the same content](https://agentwranglers.se/playbook/website/seo/mobile-first-indexing)
- Next: [Use one h1 and keep headings in order](https://agentwranglers.se/playbook/website/seo/heading-hierarchy)
- [SEO](https://agentwranglers.se/playbook/website/seo)