Read crawler visits in the server logs and verify each bot

How to do it

Search Console shows a sample, while the logs show every request, such as crawl spent on filter pages or 404s. Anyone can send a Googlebot user agent, so verify each address. Run a reverse DNS lookup, check the name ends in googlebot.com, google.com or googleusercontent.com, then look the name up forward and match the address. Or compare the address with the IP range files Google, Bing and OpenAI publish. Log no full visitor IP addresses for people, as Keep only what you need and delete it on time says. Keep the bot check to crawler lines. Fix what wastes crawling, as Keep filter combinations out of the crawl says.

Examples

Python example

import socket

GOOGLE_HOSTS = (".googlebot.com", ".google.com", ".googleusercontent.com")

def is_google_crawler(ip: str) -> bool:
    try:
        host = socket.gethostbyaddr(ip)[0]
        return host.endswith(GOOGLE_HOSTS) and ip in socket.gethostbyname_ex(host)[2]
    except OSError:
        return False

JavaScript example

import { reverse, resolve4 } from 'node:dns/promises';

const GOOGLE_HOSTS = ['.googlebot.com', '.google.com', '.googleusercontent.com'];

export async function isGoogleCrawler(ip) {
  try {
    const [host] = await reverse(ip);
    return GOOGLE_HOSTS.some(suffix => host.endsWith(suffix)) && (await resolve4(host)).includes(ip);
  } catch {
    return false;
  }
}

How to check

A monthly script reads one day of logs and lists the requests by verified bot, section and status. A request with a Googlebot user agent from an address that fails both checks is counted as unverified.

Sources

• Google crawling docs – Verify requests from Google crawlers
• OpenAI – Overview of OpenAI crawlers

Related rules

• WEB-212 Keep only what you need and delete it on time in Website
• SEO-15 Keep filter combinations out of the crawl in SEO
• SEO-85 Keep crawlers off 5xx and 429 answers for more than a day in SEO