---
title: "Keep crawlers off 5xx and 429 answers for · Agent Wranglers"
url: https://agentwranglers.se/playbook/seo/crawling-and-indexing/server-errors-and-crawl-rate
id: SEO-85
level: Recommended
updated: 2026-10-22
build: a2a6bdfed830
tokens: 841
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Keep crawlers off 5xx and 429 answers for more than a day

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Crawling and indexing](https://agentwranglers.se/playbook/seo/crawling-and-indexing.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: SEO-85
- Level: Recommended
- Chapter: [Crawling and indexing](https://agentwranglers.se/playbook/seo/crawling-and-indexing.md)
- Rule: Answer crawlers with 500, 503 or 429 only during an outage or planned work of under 24 hours. Alert when more than 1% of crawler requests in an hour get one of them.
- Check: An alert reads the access log each hour and fires when over 1% of verified crawler requests got 429 or a 5xx. A test sends 100 requests in 10 seconds from a verified crawler address in staging, and none gets 429.
- How: Google slows its crawling when it gets 500, 503 or 429. It warns against doing so for longer than one or two days. A URL that keeps answering them for several days may drop from the index. So a rate limit that catches Googlebot, or a long maintenance window, can cost pages. Exempt verified search crawlers from limits meant for scrapers, as [Read crawler visits in the server logs and verify each bot](https://agentwranglers.se/playbook/seo/crawling-and-indexing/crawler-log-analysis.md) says. Run maintenance as [Say when you'll be back](https://agentwranglers.se/playbook/website/errors-and-outages/maintenance-pages.md) says.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JavaScript example

```js
export const crawlerErrorShare = lines => {
  const crawler = lines.filter(line => line.verifiedCrawler);
  return crawler.filter(line => line.status === 429 || line.status >= 500).length / Math.max(crawler.length, 1);
};
```

### Python example

```python
def crawler_error_share(lines):
    crawler = [line for line in lines if line.verified_crawler]
    errors = [line for line in crawler if line.status == 429 or line.status >= 500]
    return len(errors) / max(len(crawler), 1)
```

## Sources

- [Google crawling docs – Reduce the Google crawl rate](https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate)

## Related rules

- SEO-84 [Read crawler visits in the server logs and verify each bot](https://agentwranglers.se/playbook/seo/crawling-and-indexing/crawler-log-analysis.md) in SEO
- WEB-221 [Say when you'll be back](https://agentwranglers.se/playbook/website/errors-and-outages/maintenance-pages.md) in Website

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/seo/crawling-and-indexing/server-errors-and-crawl-rate">Agent Wranglers SEO playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Read crawler visits in the server logs and verify each bot](https://agentwranglers.se/playbook/seo/crawling-and-indexing/crawler-log-analysis.md)
- [Crawling and indexing](https://agentwranglers.se/playbook/seo/crawling-and-indexing.md)