Agent WranglersSEO
Make every page easy to find and understand with robots.txt, sitemaps, clean addresses, redirects, rendering on the server, structured data and breadcrumbs.
- Required 2
- Recommended 8
- Optional 3
Required
Keep private pages out of search
RequiredLeave public pages open to search engines. Mark error, private, staging and utility responses
noindex, with a meta tag orX-Robots-Tagheader.Use one h1 and keep headings in order
RequiredGive every page exactly one
h1, and a heading outline that never skips a level. Never use headings just for their look.
Recommended
Tell crawlers where they may go
RecommendedServe a plain-text
/robots.txtat the site root. It should allow public content, keep private paths out and name the sitemap.List every public page in a sitemap
RecommendedPublish an XML sitemap, a list of your pages for search engines. List only canonical public URLs that answer 200, each with an honest last-changed date.
Keep addresses short and readable
RecommendedKeep addresses lowercase, hyphenated, descriptive and shallow. Leave out file endings and session parameters, and treat addresses as permanent.
Send the content in the HTML
RecommendedPut each page's content, metadata, links and structured data in the HTML the server sends. Don't leave them to JavaScript to build later.
Link every page from another
RecommendedLink every page from others with real
<a href>links and text that says where they go, so no page is left orphaned.Describe each page for machines
RecommendedDescribe each page with schema.org JSON-LD, a block of data search engines read. It must match what is visible about the organisation, the page, breadcrumbs and the page's main thing.
Show where the page sits
RecommendedShow a breadcrumb trail on inner pages, with the current page not linked. Mark it up as a matching
BreadcrumbList.Mark links you do not vouch for
RecommendedMark links in user content with
rel="ugc", paid links withrel="sponsored", and other links you don't vouch for withrel="nofollow".
Optional
Split sitemaps only when huge
OptionalSplit sitemaps with a sitemap index only when a site has very many URLs or several content types. Small sites need one file.
List key images in the sitemap
OptionalList important images in the sitemap with the image extension. This matters most when crawlers cannot reach them through links.
Tell search engines what changed
OptionalTell search engines that support IndexNow about new and changed pages. Prove the site is yours with a key file served from it.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers website playbook: https://agentwranglers.se/playbook/website/seo.