Keep crawlers off 5xx and 429 answers for more than a day

How to do it

Google slows its crawling when it gets 500, 503 or 429. It warns against doing so for longer than one or two days. A URL that keeps answering them for several days may drop from the index. So a rate limit that catches Googlebot, or a long maintenance window, can cost pages. Exempt verified search crawlers from limits meant for scrapers, as Read crawler visits in the server logs and verify each bot says. Run maintenance as Say when you'll be back says.

Examples

JavaScript example

export const crawlerErrorShare = lines => {
  const crawler = lines.filter(line => line.verifiedCrawler);
  return crawler.filter(line => line.status === 429 || line.status >= 500).length / Math.max(crawler.length, 1);
};

Python example

def crawler_error_share(lines):
    crawler = [line for line in lines if line.verified_crawler]
    errors = [line for line in crawler if line.status == 429 or line.status >= 500]
    return len(errors) / max(len(crawler), 1)

How to check

An alert reads the access log each hour and fires when over 1% of verified crawler requests got 429 or a 5xx. A test sends 100 requests in 10 seconds from a verified crawler address in staging, and none gets 429.

Sources

• Google crawling docs – Reduce the Google crawl rate

Related rules

• SEO-84 Read crawler visits in the server logs and verify each bot in SEO
• WEB-221 Say when you'll be back in Website