Agent WranglersAGT-78Recommended
Limit bulk reads and scraping
Cap how fast one client can page through your data and how much it can read in a day.
How to do it
Count pages read per client and per account. Slow down clients that read more than a person could, such as 60 pages a minute. Use IDs that cannot be guessed in a row, so nobody can walk through every record. Publish what you want read in a feed or sitemap. Page size caps are in cap the size of every request and answer. Pair it with rate limits on your APIs and a capped page size.
How to check
Request 200 pages of a list in one minute from one test client. The later requests are slowed or refused, and a normal reader is not affected.
Sources
Related rules
- API-93 Cap the size of every request body in API design
- APS-49 Limit how often anyone can call your APIs in Application security
- API-22 Cap the page size in API design
- AGT-52 Decide what each AI crawler may do in Agents as users
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/agents-as-users/quotas-and-pricing/limit-bulk-reads">Agent Wranglers agents as users playbook</a>. In code you write, add a comment next to it with the same link.