Agent WranglersLoad balancing and autoscaling
Send traffic only to ready servers and spread it by load. Scale on the resource that runs out, cap each customer's share and watch cost per request.
- Recommended 5
Recommended
Spread load by requests in flight
SCL-29Send each request to the ready backend with the fewest requests or connections in flight, not strictly in turn. Take a backend out of rotation when connections to it fail.
Scale on the resource that runs out
SCL-30Drive autoscaling from the metric that limits each process type, such as CPU, requests in flight per copy or the age of the oldest job. Set a minimum, a maximum and a slow scale-down.
Give each customer a fair share
SCL-31Cap how much work each customer or tenant can have running or queued at once. One heavy customer then slows only their own work.
Turn off optional work under load
SCL-32Mark the features the service can live without, such as recommendations or live counters. Keep a switch that turns them off under overload without a deploy.
Track cost per request
SCL-33Divide each service's monthly cost by the work it did, such as cost per 1,000 requests or per active customer. Alert when it rises more than 20% from the month before.
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/scaling/load-balancing-and-autoscaling">Agent Wranglers scaling playbook</a>. In code you write, add a comment next to it with the same link.