Crawling and indexing

Recommended

• Tell crawlers where they may go

  SEO-1

  Serve a plain-text /robots.txt at the root of every host. It allows public content, blocks only private paths and names the sitemap.

• List every public page in a sitemap

  SEO-2

  Publish an XML sitemap, a list of your pages for search engines. List only canonical public URLs that answer 200, each with an honest last-changed date.

• Send the content in the HTML

  SEO-6

  Put each page's content, metadata, links and structured data in the HTML the server sends. Don't leave them to JavaScript to build later.

• Serve phones the same content

  SEO-7

  Give phone visitors the same content, links, metadata and structured data as desktop visitors. Google indexes the phone version of every page.

• Keep private pages out of search

  SEO-9

  Leave public pages open to search engines. Mark error, private, utility and internal search-result responses noindex, with a meta tag or X-Robots-Tag header.

Optional

• Split sitemaps only when huge

  SEO-3

  Split the sitemap only when one file would pass 50,000 URLs or 50 MB uncompressed. List the parts in a sitemap index. Below those limits, keep one file.

• Tell search engines what changed

  SEO-19

  Tell search engines that support IndexNow, such as Bing, Yandex and Seznam, about new and changed pages. Prove the site is yours with a key file served from it.

Read this with

• Website playbook

  Robots.txt, sitemaps, keywords and links decide whether search engines find and rank the pages you build.