Sign your bot's requests

If you run a crawler or agent, sign its requests with Web Bot Auth. As a site, don't block unsigned bots by default.

By Erik Bergstedt, Agent Wranglers. Updated 20 October 2026.

How to do it

If you run a bot, sign each request with HTTP Message Signatures (RFC 9421), using tag="web-bot-auth". Send Signature-Agent as a dictionary that names your key directory, as in the example. Publish your public keys at /.well-known/http-message-signatures-directory on that domain, as application/http-message-signatures-directory+json. The keyid is the key's JWK SHA-256 thumbprint. As a site, verify signatures where you need to tell bots apart, as in verify who an agent is. Web Bot Auth is an IETF working group draft from 2026, so expect changes.

Example

GET /page HTTP/1.1
Host: example.com
Signature-Agent: sig1="https://bot.example.com"
Signature-Input: sig1=("@authority" "signature-agent";key="sig1");created=1760000000;expires=1760000300;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";tag="web-bot-auth"
Signature: sig1=:base64signature...:

How to check

A verifier, such as the open-source libraries linked from the draft, accepts your signed request. The key directory answers 200 over HTTPS.

For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers website playbook: https://agentwranglers.se/playbook/website/ready-for-ai-agents/web-bot-auth.