Agent Wranglers

SEO-53Avoid

Showing crawlers a different page

Serving search engine crawlers different content from what visitors get, or redirecting visitors to a page the crawler never saw. Both are cloaking and sneaky redirects.

By Agent Wranglers. Updated .

How to do it

Google defines cloaking as presenting different content to users and search engines to manipulate rankings and mislead users. Sneaky redirects send people somewhere other than what the crawler was shown, such as phone visitors to a spam domain. Bing treats both as manipulation. A hacked site often does this without the owner knowing, so test it even if you never would. Normal redirects are fine, such as a site move, a merged page, or a signed-in visitor sent to their account. Serve agents the same page too, as Show agents the same page as people says.

Examples

Shell example

for ua in 'Mozilla/5.0' 'Mozilla/5.0 (Linux; Android 14; Pixel 8) Mobile' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'; do
  curl -sL -A "$ua" -o /dev/null -w '%{url_effective} ' https://example.com/pricing
  curl -sL -A "$ua" https://example.com/pricing | sed -n 's:.*<h1[^>]*>\(.*\)</h1>.*:\1:p' | sha256sum
done

How to check

A test fetches each page type three times, with a browser, a phone and a Googlebot user agent. All three end at the same final address, with the same h1 and the same main text.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/seo/search-spam-and-penalties/cloaking">Agent Wranglers SEO playbook</a>. In code you write, add a comment next to it with the same link.