SEO Recommended

Keep filter combinations out of the crawl

Stop crawlers from following every combination of filters and sorting. Let them crawl only the filtered pages you want found.

By Erik Bergstedt, Agent Wranglers. Updated 20 October 2026.

How to do it

Choose the few filter pages worth finding, such as one per category, and give them clean addresses. Block other filter and sort parameters in robots.txt, or make those links plain buttons. Keep parameters in one fixed order. Answer 404 when a filter combination has no results.

Example

User-agent: *
Disallow: /*?*sort=
Disallow: /*?*&

How to check

A crawl from the home page finds no URL with more than one filter parameter. An empty filter combination answers 404.

For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers website playbook: https://agentwranglers.se/playbook/website/seo/faceted-navigation.