Agent Wranglers

AGT-49Recommended

Offer every page as Markdown

Serve every page as Markdown too. Use its address plus .md, and answer requests with Accept: text/markdown. Point back to the HTML as the canonical copy.

By Agent Wranglers. Updated .

How to do it

Make a Markdown copy of each page at build time. Serve it at /page.md. On the page URL, serve it when Accept ranks text/markdown above text/html, and serve HTML otherwise. Send Vary: Accept on both answers. Give the copy front matter (title, URL, updated) and a Link: <page>; rel="canonical" header. Add X-Robots-Tag: noindex to the Markdown copy, so search engines index only the HTML. Answering Accept: text/markdown is a convention some CDNs and agents follow, not a standard.

Examples

HTTP example

GET /services HTTP/1.1
Host: example.com
Accept: text/markdown

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
Link: <https://example.com/services>; rel="canonical"
X-Robots-Tag: noindex

---
title: Services
url: https://example.com/services
updated: 2026-10-01
---
# Services

JavaScript example

function prefersMarkdown(accept = '') {
  const weightOf = type => {
    const part = accept.split(',').map(p => p.trim()).find(p => p.split(';')[0] === type);
    if (!part) return 0;
    const q = /;\s*q=([\d.]+)/.exec(part);
    return q ? Number(q[1]) : 1;
  };
  return weightOf('text/markdown') > weightOf('text/html');
}

Python example

def prefers_markdown(accept: str = "") -> bool:
    def weight_of(media_type: str) -> float:
        for part in (p.strip() for p in accept.split(",")):
            name, *params = [x.strip() for x in part.split(";")]
            if name == media_type:
                q = next((x[2:] for x in params if x.startswith("q=")), "1")
                return float(q)
        return 0.0
    return weight_of("text/markdown") > weight_of("text/html")

How to check

curl -sI -H 'Accept: text/markdown' URL shows Content-Type: text/markdown, Vary: Accept and the canonical Link. A plain browser request to the same URL gets HTML with Vary: Accept.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/agents-as-users/ready-for-ai-agents/markdown-source-endpoints">Agent Wranglers agents as users playbook</a>. In code you write, add a comment next to it with the same link.