Let invited agents reach what you invite them to

How to do it

Fetchers that act for one person, such as ChatGPT-User and Claude-User, read robots.txt in some setups. If robots.txt blocks /mcp while your skill tells agents to call it, those agents give up. Allow user-triggered agents to reach every endpoint you advertise. Keep search crawlers from indexing them with X-Robots-Tag: noindex on those answers instead. Set the crawler groups as Decide what each AI crawler may do says.

Examples

Plain text example

User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /mcp
Allow: /a2a
Allow: /ask

Python example

from urllib.robotparser import RobotFileParser
robots = RobotFileParser("https://example.com/robots.txt")
robots.read()
for agent in ["ChatGPT-User", "Claude-User", "Perplexity-User"]:
    assert all(robots.can_fetch(agent, url) for url in advertised_endpoints), agent

JavaScript example

import robotsParser from 'robots-parser';
const robots = robotsParser('https://example.com/robots.txt', robotsText);
for (const agent of ['ChatGPT-User', 'Claude-User', 'Perplexity-User'])
  for (const url of advertisedEndpoints) console.assert(robots.isAllowed(url, agent), `${agent} blocked from ${url}`);

How to check

A script collects every endpoint named in llms.txt, the skill files and the agent cards. A robots parser confirms that robots.txt allows each one for ChatGPT-User, Claude-User and Perplexity-User.

Related rules

• WEB-188 Decide what each AI crawler may do in Website