---
title: "Read crawler visits in the server logs and · Agent Wranglers"
url: https://agentwranglers.se/playbook/seo/crawling-and-indexing/crawler-log-analysis
id: SEO-84
level: Recommended
updated: 2026-10-22
build: a2a6bdfed830
tokens: 1030
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Read crawler visits in the server logs and verify each bot

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Crawling and indexing](https://agentwranglers.se/playbook/seo/crawling-and-indexing.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: SEO-84
- Level: Recommended
- Chapter: [Crawling and indexing](https://agentwranglers.se/playbook/seo/crawling-and-indexing.md)
- Rule: Each month, count crawler requests per site section and status code from the server logs. Count a request as a search engine only after its IP address passes the engine's own check.
- Check: A monthly script reads one day of logs and lists the requests by verified bot, section and status. A request with a Googlebot user agent from an address that fails both checks is counted as unverified.
- How: Search Console shows a sample, while the logs show every request, such as crawl spent on filter pages or 404s. Anyone can send a Googlebot user agent, so verify each address. Run a reverse DNS lookup, check the name ends in `googlebot.com`, `google.com` or `googleusercontent.com`, then look the name up forward and match the address. Or compare the address with the IP range files Google, Bing and OpenAI publish. Log no full visitor IP addresses for people, as [Keep only what you need and delete it on time](https://agentwranglers.se/playbook/website/privacy/data-minimization.md) says. Keep the bot check to crawler lines. Fix what wastes crawling, as [Keep filter combinations out of the crawl](https://agentwranglers.se/playbook/seo/site-structure/faceted-navigation.md) says.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### Python example

```python
import socket

GOOGLE_HOSTS = (".googlebot.com", ".google.com", ".googleusercontent.com")

def is_google_crawler(ip: str) -> bool:
    try:
        host = socket.gethostbyaddr(ip)[0]
        return host.endswith(GOOGLE_HOSTS) and ip in socket.gethostbyname_ex(host)[2]
    except OSError:
        return False
```

### JavaScript example

```js
import { reverse, resolve4 } from 'node:dns/promises';

const GOOGLE_HOSTS = ['.googlebot.com', '.google.com', '.googleusercontent.com'];

export async function isGoogleCrawler(ip) {
  try {
    const [host] = await reverse(ip);
    return GOOGLE_HOSTS.some(suffix => host.endsWith(suffix)) && (await resolve4(host)).includes(ip);
  } catch {
    return false;
  }
}
```

## Sources

- [Google crawling docs – Verify requests from Google crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
- [OpenAI – Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)

## Related rules

- WEB-212 [Keep only what you need and delete it on time](https://agentwranglers.se/playbook/website/privacy/data-minimization.md) in Website
- SEO-15 [Keep filter combinations out of the crawl](https://agentwranglers.se/playbook/seo/site-structure/faceted-navigation.md) in SEO
- SEO-85 [Keep crawlers off 5xx and 429 answers for more than a day](https://agentwranglers.se/playbook/seo/crawling-and-indexing/server-errors-and-crawl-rate.md) in SEO

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/seo/crawling-and-indexing/crawler-log-analysis">Agent Wranglers SEO playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Tell search engines what changed](https://agentwranglers.se/playbook/seo/crawling-and-indexing/indexnow.md)
- Next: [Keep crawlers off 5xx and 429 answers for more than a day](https://agentwranglers.se/playbook/seo/crawling-and-indexing/server-errors-and-crawl-rate.md)
- [Crawling and indexing](https://agentwranglers.se/playbook/seo/crawling-and-indexing.md)