Agent WranglersHostile agents and bots Recommended
Limit bulk reads and scraping
Cap how fast one client can page through your data and how much it can read in a day.
How to do it
Count pages read per client and per account, and slow down clients that read far more than a person could. Use IDs that cannot be guessed in a row, so nobody can walk through every record. Publish what you want read in a feed or sitemap. Page size caps are in cap the size of every request and answer.
How to check
Request 200 pages of a list in one minute from one test client. The later requests are slowed or refused, and a normal reader is not affected.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers security testing playbook: https://agentwranglers.se/playbook/adversarial-testing/hostile-agents-and-bots/limit-bulk-reads.