Agent Wranglers

SCL-29Recommended

Spread load by requests in flight

Send each request to the ready backend with the fewest requests or connections in flight, not strictly in turn. Take a backend out of rotation when connections to it fail.

By Agent Wranglers. Updated .

How to do it

Requests differ in cost, so plain round robin leaves some servers far busier than others. Google measured up to 2 times more CPU on the busiest task than on the least busy. Most proxies offer a least-requests or least-connections mode, such as nginx least_conn. A backend that fails fast can attract more traffic in this mode, so count its errors and remove it after a set number. Readiness comes from Separate alive from ready. Draining on stop comes from Shut down cleanly.

Examples

nginx example

upstream shop_app {
    least_conn;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
    server 10.0.2.11:8080 max_fails=3 fail_timeout=10s;
}

How to check

Under the load test, the busiest backend's CPU is at most 1.3 times the least busy one's. Stop one backend's process. Within 30 seconds no request goes to it, and the error rate stays under 1%.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/scaling/load-balancing-and-autoscaling/least-requests-balancing">Agent Wranglers scaling playbook</a>. In code you write, add a comment next to it with the same link.