Agent Wranglers

AGT-52Recommended

Decide what each AI crawler may do

Decide what each AI crawler may do. Write it in robots.txt as its own user-agent group.

By Agent Wranglers. Updated .

How to do it

Add a group for each AI user agent you care about, using the names each vendor documents. Training crawlers include GPTBot, ClaudeBot, Meta-ExternalAgent and CCBot. Search crawlers include OAI-SearchBot, Claude-SearchBot and PerplexityBot. Google-Extended and Applebot-Extended are tokens, not crawlers. They control whether Google and Apple may train on pages their normal crawlers fetch. Give each group an explicit Allow or Disallow. A crawler that finds a group with its own name ignores the * group, so repeat shared rules in each group. Fetchers that act for one person, such as ChatGPT-User and Perplexity-User, may not follow robots.txt. Robots.txt never stops a crawler that ignores it. Enforce limits on the server, as Limit bulk reads and scraping says.

Examples

Plain text example

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Python example

from urllib.robotparser import RobotFileParser
robots = RobotFileParser("https://example.com/robots.txt")
robots.read()
assert robots.can_fetch("OAI-SearchBot", "https://example.com/services")
assert not robots.can_fetch("ClaudeBot", "https://example.com/services")

JavaScript example

import robotsParser from 'robots-parser';
const url = 'https://example.com/robots.txt';
const robots = robotsParser(url, await (await fetch(url)).text());
console.assert(robots.isAllowed('https://example.com/services', 'OAI-SearchBot'));
console.assert(robots.isDisallowed('https://example.com/services', 'ClaudeBot'));

How to check

A unit test parses the served robots.txt with a robots parser and asserts allowed or blocked for each named agent. Every named group carries every Disallow line of the * group. Server logs show no blocked agent fetching blocked paths.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents/robots-for-ai-crawlers">Agent Wranglers agents as users playbook</a>. In code you write, add a comment next to it with the same link.