---
title: "Decide what each AI crawler may do · Agent Wranglers"
url: https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents/robots-for-ai-crawlers
id: AGT-52
level: Recommended
updated: 2026-10-27
build: a2a6bdfed830
tokens: 1203
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Decide what each AI crawler may do

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Ready for AI agents](https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: AGT-52
- Level: Recommended
- Chapter: [Ready for AI agents](https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents.md)
- Rule: Decide what each AI crawler may do. Write it in robots.txt as its own user-agent group.
- Check: A unit test parses the served robots.txt with a robots parser and asserts allowed or blocked for each named agent. Every named group carries every `Disallow` line of the `*` group. Server logs show no blocked agent fetching blocked paths.
- How: Add a group for each AI user agent you care about, using the names each vendor documents. Training crawlers include GPTBot, ClaudeBot, Meta-ExternalAgent and CCBot. Search crawlers include OAI-SearchBot, Claude-SearchBot and PerplexityBot. `Google-Extended` and `Applebot-Extended` are tokens, not crawlers. They control whether Google and Apple may train on pages their normal crawlers fetch. Give each group an explicit `Allow` or `Disallow`. A crawler that finds a group with its own name ignores the `*` group, so repeat shared rules in each group. Fetchers that act for one person, such as ChatGPT-User and Perplexity-User, may not follow robots.txt. Robots.txt never stops a crawler that ignores it. Enforce limits on the server, as [Limit bulk reads and scraping](https://agentwranglers.se/playbook/agents-as-users/quotas-and-pricing/limit-bulk-reads.md) says.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### Plain text example

```text
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml
```

### Python example

```python
from urllib.robotparser import RobotFileParser
robots = RobotFileParser("https://example.com/robots.txt")
robots.read()
assert robots.can_fetch("OAI-SearchBot", "https://example.com/services")
assert not robots.can_fetch("ClaudeBot", "https://example.com/services")
```

### JavaScript example

```js
import robotsParser from 'robots-parser';
const url = 'https://example.com/robots.txt';
const robots = robotsParser(url, await (await fetch(url)).text());
console.assert(robots.isAllowed('https://example.com/services', 'OAI-SearchBot'));
console.assert(robots.isDisallowed('https://example.com/services', 'ClaudeBot'));
```

## Sources

- [RFC 9309 (Robots Exclusion Protocol)](https://www.rfc-editor.org/rfc/rfc9309)
- [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

## Related rules

- AGT-51 [Showing agents a different page than people](https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents/same-content-for-agents.md) in Agents as users
- AGT-78 [Limit bulk reads and scraping](https://agentwranglers.se/playbook/agents-as-users/quotas-and-pricing/limit-bulk-reads.md) in Agents as users
- AGT-53 [Let invited agents reach what you invite them to](https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents/robots-allow-invited-endpoints.md) in Agents as users
- AGT-54 [Say how your content may be used](https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents/content-signals.md) in Agents as users
- SEO-22 [Choose what search and AI answers may quote](https://agentwranglers.se/playbook/seo/search-result-appearance/snippet-controls.md) in SEO
- SEO-72 [Treat AI answers as search results](https://agentwranglers.se/playbook/seo/ai-search-answers/ai-features-same-seo.md) in SEO

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents/robots-for-ai-crawlers">Agent Wranglers agents as users playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Mark user content that agents read](https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents/mark-user-content.md)
- Next: [Let invited agents reach what you invite them to](https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents/robots-allow-invited-endpoints.md)
- [Ready for AI agents](https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents.md)