Agent Wranglers

AGT-51Avoid

Showing agents a different page than people

Never serve AI crawlers or agents different content from what people see, chosen by user agent, IP address or bot signature.

By Agent Wranglers. Updated .

How to do it

Build the HTML, Markdown copy, JSON-LD and llms.txt from the same source, so they state the same facts. Never add text that only bots see, such as hidden claims or instructions to AI models. Search engines treat it as cloaking, as Showing crawlers a different page explains. Agents treat hidden instructions as prompt injection. Only access rules, such as rate limits or a blocked path, may differ by agent. To keep an agent out, block it as Decide what each AI crawler may do says.

Examples

Shell example

text() { curl -s -A "$1" https://example.com/services | sed 's/<[^>]*>//g'; }
diff <(text "Mozilla/5.0") <(text "GPTBot/1.2") && echo same

How to check

Fetch each page with a browser user agent and with an AI crawler's user agent, such as GPTBot. The visible text of the two answers is identical.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents/same-content-for-agents">Agent Wranglers agents as users playbook</a>. In code you write, add a comment next to it with the same link.