Agent WranglersAGT-54Optional
Say how your content may be used
Use Content-Signal lines in robots.txt to say whether your content may be used for search, AI answers or AI training.
How to do it
Add Content-Signal: search=yes, ai-input=yes, ai-train=no (your choice) to each group. It comes from the Content Signals Policy, a proposal published by Cloudflare. It states your wishes but does not enforce them. Lighthouse marks it as an unknown directive, which is expected. The IETF AI Preferences group drafts a similar Content-Usage line. It is still an Internet-Draft, and its names have changed between versions, so add it only once it is final. Set up the groups first, as Decide what each AI crawler may do says.
Examples
Plain text example
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /How to check
Fetch robots.txt. Every user-agent group has one Content-Signal line with your three choices.
Measured
On a reference site, removing these lines raises Lighthouse SEO from 92 to 100, because of the "unknown directive" warning. RFC 9309 has crawlers use only the lines they can parse, so other crawlers skip them. Keep them if you want the signals.
Sources
Related rules
- AGT-52 Decide what each AI crawler may do in Agents as users
- AGT-55 State your text-mining choice in Agents as users
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents/content-signals">Agent Wranglers agents as users playbook</a>. In code you write, add a comment next to it with the same link.