Agent Wranglers

Hostile agents and bots Required

Verify who an agent is

Never trust the user agent string. Give special access to a bot or agent only after it proves who it is.

By Erik Bergstedt, Agent Wranglers. Updated .

How to do it

Check signed requests where the agent offers them, such as HTTP message signatures in the style of Web Bot Auth. For known crawlers, confirm the address with a reverse DNS lookup and a forward lookup that matches. Treat every other request as an ordinary visitor. To sign your own agent's requests, see Web Bot Auth.

Example

host 66.249.66.1          # crawl-66-249-66-1.googlebot.com
host crawl-66-249-66-1.googlebot.com   # must return 66.249.66.1

How to check

Send a request with a well-known crawler's user agent from a test machine. It gets no more access than a normal visitor.

For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers security testing playbook: https://agentwranglers.se/playbook/adversarial-testing/hostile-agents-and-bots/verify-agent-identity.