Agent Wranglers

Ready for AI agents Recommended

Decide what each AI crawler may do

Decide what each AI crawler may do. Write it in robots.txt as its own user-agent group.

By Erik Bergstedt, Agent Wranglers. Updated .

How to do it

Add a group for each AI user agent you care about. Training crawlers include GPTBot, ClaudeBot and CCBot. Search crawlers include OAI-SearchBot, Claude-SearchBot and PerplexityBot. Google-Extended is a token, not a crawler, and controls Gemini training and grounding. Give each group an explicit Allow or Disallow. Fetchers that act for one user, such as ChatGPT-User, may not follow robots.txt, and robots.txt never stops a crawler that ignores it.

Example

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

How to check

A robots.txt tester, such as the one in Google Search Console, shows each agent allowed or blocked as you meant. Your server logs show the named crawlers obeying.

For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers website playbook: https://agentwranglers.se/playbook/website/ready-for-ai-agents/robots-for-ai-crawlers.