---
title: "Let invited agents reach what you invite · Agent Wranglers"
url: https://agentwranglers.se/playbook/website/ready-for-ai-agents/robots-allow-invited-endpoints
id: WEB-253
level: Recommended
updated: 2026-10-22
tokens: 813
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Let invited agents reach what you invite them to

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Ready for AI agents](https://agentwranglers.se/playbook/website/ready-for-ai-agents.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: WEB-253
- Level: Recommended
- Chapter: [Ready for AI agents](https://agentwranglers.se/playbook/website/ready-for-ai-agents.md)
- Rule: robots.txt never blocks an endpoint that llms.txt, a skill or an agent card tells agents to use.
- Check: A script collects every endpoint named in llms.txt, the skill files and the agent cards. A robots parser confirms that robots.txt allows each one for ChatGPT-User, Claude-User and Perplexity-User.
- How: Fetchers that act for one person, such as ChatGPT-User and Claude-User, read robots.txt in some setups. If robots.txt blocks `/mcp` while your skill tells agents to call it, those agents give up. Allow user-triggered agents to reach every endpoint you advertise. Keep search crawlers from indexing them with `X-Robots-Tag: noindex` on those answers instead. Set the crawler groups as [Decide what each AI crawler may do](https://agentwranglers.se/playbook/website/ready-for-ai-agents/robots-for-ai-crawlers.md) says.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### Plain text example

```text
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /mcp
Allow: /a2a
Allow: /ask
```

### Python example

```python
from urllib.robotparser import RobotFileParser
robots = RobotFileParser("https://example.com/robots.txt")
robots.read()
for agent in ["ChatGPT-User", "Claude-User", "Perplexity-User"]:
    assert all(robots.can_fetch(agent, url) for url in advertised_endpoints), agent
```

### JavaScript example

```js
import robotsParser from 'robots-parser';
const robots = robotsParser('https://example.com/robots.txt', robotsText);
for (const agent of ['ChatGPT-User', 'Claude-User', 'Perplexity-User'])
  for (const url of advertisedEndpoints) console.assert(robots.isAllowed(url, agent), `${agent} blocked from ${url}`);
```

## Related rules

- WEB-188 [Decide what each AI crawler may do](https://agentwranglers.se/playbook/website/ready-for-ai-agents/robots-for-ai-crawlers.md) in Website

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/website/ready-for-ai-agents/robots-allow-invited-endpoints">Agent Wranglers website playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Decide what each AI crawler may do](https://agentwranglers.se/playbook/website/ready-for-ai-agents/robots-for-ai-crawlers.md)
- Next: [Say how your content may be used](https://agentwranglers.se/playbook/website/ready-for-ai-agents/content-signals.md)
- [Ready for AI agents](https://agentwranglers.se/playbook/website/ready-for-ai-agents.md)