Agent WranglersSEO-15Recommended
Keep filter combinations out of the crawl
Stop crawlers from following every combination of filters and sorting. Let them crawl only the filtered pages you want found.
How to do it
Choose the few filter pages worth finding, such as one per category, and give them clean addresses. Block other filter and sort parameters in robots.txt, or make those links plain buttons. Keep parameters in one fixed order. Answer 404 when a filter combination has no results. Write the blocking lines as Tell crawlers where they may go says.
Examples
Plain text example
User-agent: *
Disallow: /*?*sort=
Disallow: /*?*&How to check
A crawl from the home page finds no URL with more than one filter parameter. An empty filter combination answers 404.
Sources
Related rules
- SEO-1 Tell crawlers where they may go in SEO
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/seo/site-structure/faceted-navigation">Agent Wranglers SEO playbook</a>. In code you write, add a comment next to it with the same link.