---
title: "Decide what each AI crawler may do · Agent Wranglers"
url: https://agentwranglers.se/playbook/website/ready-for-ai-agents/robots-for-ai-crawlers
updated: 2026-10-20
licence: "All rights reserved"
---

# Decide what each AI crawler may do

By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20.

- Level: Recommended
- Chapter: [Ready for AI agents](https://agentwranglers.se/playbook/website/ready-for-ai-agents)
- Rule: Decide what each AI crawler may do. Write it in robots.txt as its own user-agent group.
- How: Add a group for each AI user agent you care about, using the names each vendor documents. Training crawlers include GPTBot, ClaudeBot, Meta-ExternalAgent and CCBot. Search crawlers include OAI-SearchBot, Claude-SearchBot and PerplexityBot. `Google-Extended` and `Applebot-Extended` are tokens, not crawlers. They control whether Google and Apple may train on pages their normal crawlers fetch. Give each group an explicit `Allow` or `Disallow`. A crawler that finds a group with its own name ignores the `*` group, so repeat shared rules in each group. Fetchers that act for one person, such as ChatGPT-User and Perplexity-User, may not follow robots.txt. Robots.txt never stops a crawler that ignores it.
- Check: A unit test parses the served robots.txt with a robots parser and asserts allowed or blocked for each named agent. Server logs show no blocked agent fetching blocked paths.

Example:

```text
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml
```

```python
from urllib.robotparser import RobotFileParser
robots = RobotFileParser("https://example.com/robots.txt")
robots.read()
assert robots.can_fetch("OAI-SearchBot", "https://example.com/services")
assert not robots.can_fetch("ClaudeBot", "https://example.com/services")
```

```js
import robotsParser from 'robots-parser';
const url = 'https://example.com/robots.txt';
const robots = robotsParser(url, await (await fetch(url)).text());
console.assert(robots.isAllowed('https://example.com/services', 'OAI-SearchBot'));
console.assert(robots.isDisallowed('https://example.com/services', 'ClaudeBot'));
```

**For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers website playbook: https://agentwranglers.se/playbook/website/ready-for-ai-agents/robots-for-ai-crawlers`.

- Previous: [Show agents the same page as people](https://agentwranglers.se/playbook/website/ready-for-ai-agents/same-content-for-agents)
- Next: [Say how your content may be used](https://agentwranglers.se/playbook/website/ready-for-ai-agents/content-signals)
- [Ready for AI agents](https://agentwranglers.se/playbook/website/ready-for-ai-agents)