Ready for AI agents

Recommended

• Make the site easy for AI agents

  AGT-42

  Make the site easy for AI agents to read and use. Put content in the HTML, and add structured data, Markdown copies, clear crawler rules and discovery files.

• Offer every page as Markdown

  AGT-49

  Serve every page as Markdown too. Use its address plus .md, and answer requests with Accept: text/markdown. Point back to the HTML as the canonical copy.

• Link Markdown copies to Markdown copies

  AGT-50

  Every internal link in a Markdown copy points to the Markdown copy of its target.

• Mark user content that agents read

  AGT-79

  Keep text written by your users apart from your own content, so other people's agents can tell it is not yours.

• Decide what each AI crawler may do

  AGT-52

  Decide what each AI crawler may do. Write it in robots.txt as its own user-agent group.

• Let invited agents reach what you invite them to

  AGT-53

  robots.txt never blocks an endpoint that llms.txt, a skill or an agent card tells agents to use.

• Offer content machines can read

  AGT-59

  Offer the site's content as a sitemap, feeds, Markdown and JSON-LD.

• Answer bursts with 429 instead of a stall

  AGT-72

  When a client asks too often, answer 429 with Retry-After at once. Serve large agent files as files built ahead of time, never built on request.

• Resolve retired IDs everywhere

  AGT-73

  Every place that accepts an ID follows the map of moved IDs. A lookup of several IDs returns what it found, what moved and what it did not find.

• Use one name for each thing in every agent file

  AGT-74

  Give each section, product or document one name. Use it in the page title, the h1, the front matter, llms.txt, the JSON files and the agent cards.

• Keep feeds to recent changes

  AGT-75

  A site feed carries recent updates only, with a summary and a link per entry, not the full text of every page.

Optional

• Give AI tools a site guide

  AGT-43

  Publish /llms.txt, a short Markdown guide to your site for AI tools. Give the site's name, a one-line summary, and links to the Markdown version of each important page.

• Offer one file only while it fits an agent's context

  AGT-44

  Offer all content in one Markdown file, such as /llms-full.txt, only while it stays under 100,000 tokens. Above that, split it by section behind an index.

• State the size of every agent file in tokens

  AGT-46

  Every file for agents states its size in tokens, so an agent knows the cost before it fetches the file.

• Layer agent content with a budget per layer

  AGT-47

  Arrange agent content in layers, each with a token budget, and link each layer to the next as Markdown.

• Split large agent content into named parts behind an index

  AGT-48

  Split content too large for one read into parts by meaning. An index lists every part with a summary, a size in tokens, an address and a date.

• Say how your content may be used

  AGT-54

  Use Content-Signal lines in robots.txt to say whether your content may be used for search, AI answers or AI training.

• State your text-mining choice

  AGT-55

  State whether you reserve text-and-data-mining rights. Use a tdm-reservation header and /.well-known/tdmrep.json, matching robots.txt.

• Sign your bot's requests

  AGT-56

  If you run a crawler or agent, sign its requests with Web Bot Auth. As a site, don't block unsigned bots by default.

• Explain how agents sign up

  AGT-57

  If agents must register before using your service, explain how in an auth.md file at the root of your service.

• Publish each page's data as a file

  AGT-58

  Publish each page's JSON-LD (structured facts) in the page and as its own file. Then agents can read the facts without parsing HTML.

• Advertise agent files in headers

  AGT-60

  List your discovery files in HTTP Link headers on every response. These include llms.txt, the sitemap, feeds, the API catalog and agent cards.

• Publish a server card for your MCP server

  AGT-61

  If you run an MCP server, a way for AI assistants to use your tools, publish a server card that lists it.

• Publish an agent card

  AGT-62

  If you run an A2A agent, one that talks to other agents, publish its card at /.well-known/agent-card.json. Keep a working endpoint behind it.

• List skills agents can learn

  AGT-63

  Publish agent skills as SKILL.md files, listed in /.well-known/agent-skills/index.json. Give each one a digest, a hash of its exact bytes.

• List agent endpoints in DNS

  AGT-64

  List your agent endpoints in DNS with SVCB records, as the DNS-AID draft describes. Sign the zone with DNSSEC so answers can be trusted.

• Keep one catalogue of agent tools

  AGT-65

  List your MCP servers, agents and skills in /.well-known/ai-catalog.json. Link it from every page.

• Answer questions from your own pages

  AGT-66

  Offer an endpoint that answers plain-language questions from your own content, with sources.

• Publish a topic index for agents

  AGT-67

  Offer one small file that maps common topics to the pages that answer them.

• Give in-browser agents your tools

  AGT-68

  Register the site's tools with the browser through WebMCP. Mark the ones with effects, and report expected failures in the tool's answer.

• Package your knowledge for agents

  AGT-69

  Publish the site's knowledge as an Open Knowledge Format bundle. It has an index, one typed Markdown file per concept, and a change log.

• Index every page's data file

  AGT-70

  List every JSON-LD file with its types in /schemamap.xml. Link it from the page head.

• Show one credit line to every reader

  AGT-71

  If others may reuse your content, show one visible credit line on every page, with the same words for people and agents. Repeat it in the Markdown copy.

• Put the answer first in every Markdown copy

  AGT-76

  A Markdown copy opens with front matter and then the main point. It repeats no text that appears on every page.

• Measure how much an agent reads to find an answer

  AGT-77

  Keep a fixed set of questions with their expected pages. Measure the tokens and fetches a fresh agent needs to reach each one.

Avoid

• Serving one file too large for any agent to read

  AGT-45

  Never publish a file for agents to read that is larger than an agent's context window. A full-site dump of millions of tokens is one example.

• Showing agents a different page than people

  AGT-51

  Never serve AI crawlers or agents different content from what people see, chosen by user agent, IP address or bot signature.

Read this with

• Website playbook

  AI tools read sites too. Give them Markdown copies, a site guide and clear rules for each AI crawler.

• SEO playbook

  AI tools read sites too. Give them Markdown copies, a site guide and clear rules for each AI crawler.