Say how your content may be used

Use Content-Signal lines in robots.txt to say whether your content may be used for search, AI answers or AI training.

By Erik Bergstedt, Agent Wranglers. Updated 20 October 2026.

How to do it

Add Content-Signal: search=yes, ai-input=yes, ai-train=no (your choice) to each group. It comes from the Content Signals Policy, a proposal published by Cloudflare. It states your wishes but does not enforce them. Lighthouse marks it as an unknown directive, which is expected. The IETF AI Preferences group drafts a similar Content-Usage line. It is still an Internet-Draft, and its names have changed between versions, so add it only once it is final.

Example

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

How to check

Fetch robots.txt. Every user-agent group has one Content-Signal line with your three choices.

Measured

Removing these lines raises Lighthouse SEO from 92 to 100, because of the "unknown directive" warning. RFC 9309 has crawlers use only the lines they can parse, so other crawlers skip them. Keep them if you want the signals.

For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers website playbook: https://agentwranglers.se/playbook/website/ready-for-ai-agents/content-signals.