Agent Wranglers

AGT-42Recommended

Make the site easy for AI agents

Make the site easy for AI agents to read and use. Put content in the HTML, and add structured data, Markdown copies, clear crawler rules and discovery files.

By Agent Wranglers. Updated .

How to do it

Start with the basics of server-rendered HTML, structured data, robots rules and Markdown copies. Add llms.txt when coding agents read your pages. Then add tool endpoints such as MCP if agents can act on your site. Many of the newer formats in this chapter are drafts, so each rule names its status. For an API, see Make the API easy for agents to find.

Examples

Shell example

# Quick check that each file answers
for p in / /index.md /llms.txt /robots.txt /sitemap.xml; do
  curl -s -o /dev/null -w "%{http_code} %{content_type} $p\n" "https://example.com$p"
done

How to check

Every discovery file answers 200 with the right content type. The HTML of every page, fetched without running scripts, contains its h1 and main text.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents/agent-readiness-overview">Agent Wranglers agents as users playbook</a>. In code you write, add a comment next to it with the same link.