Turn work away before overload

How to do it

Under overload, a long queue makes every request slow until all of them time out, and nobody is served. Set the cap from a load test, near the number of requests in flight where response time starts to climb. Register the health check before the cap, so it still answers. Before turning requests away, switch off optional work, as Turn off optional work under load says. Per-client limits are a separate rule, rate limits. Clients wait and retry as in retry with backoff.

Examples

JavaScript example

const MAX_IN_FLIGHT = 200;
let inFlight = 0;

app.use((req, res, next) => {
  if (inFlight >= MAX_IN_FLIGHT) return res.set('Retry-After', '5').status(503).end();
  inFlight += 1;
  res.on('close', () => { inFlight -= 1; });
  next();
});

Python example

from starlette.responses import Response

MAX_IN_FLIGHT = 200
in_flight = 0

@app.middleware("http")
async def shed_load(request, call_next):
    global in_flight
    if in_flight >= MAX_IN_FLIGHT:
        return Response(status_code=503, headers={"Retry-After": "5"})
    in_flight += 1
    try:
        return await call_next(request)
    finally:
        in_flight -= 1

How to check

Load staging with twice its tested limit. Extra requests get 503 within 100 ms, accepted requests stay within their speed objective, and the service recovers without a restart.

Sources

• RFC 9110 section 15.6.4 503 Service Unavailable
• RFC 9110 section 10.2.3 Retry-After

Related rules

• SCL-32 Turn off optional work under load in Scaling
• APS-49 Limit how often anyone can call your APIs in Application security
• API-34 Retry with backoff and jitter in API design
• OPS-66 Cap retries across the whole service in Operations and observability
• OPS-74 Stop calling a failing service with a circuit breaker in Operations and observability
• SCL-15 Keep the app working when the cache is gone in Scaling
• EVT-18 Bound the backlog and push back on producers in Integration and events