Turn work away before overload

How to do it

Under overload, a long queue makes every request slow until all of them time out, and nobody is served. Set the cap from a load test, near the number of requests in flight where response time starts to climb. Register the health check before the cap, so it still answers. Per-client limits are a separate rule, rate limits. Clients wait and retry as in retry with backoff.

Example

const MAX_IN_FLIGHT = 200;
let inFlight = 0;

app.use((req, res, next) => {
  if (inFlight >= MAX_IN_FLIGHT) return res.set('Retry-After', '5').status(503).end();
  inFlight += 1;
  res.on('close', () => { inFlight -= 1; });
  next();
});

from starlette.responses import Response

MAX_IN_FLIGHT = 200
in_flight = 0

@app.middleware("http")
async def shed_load(request, call_next):
    global in_flight
    if in_flight >= MAX_IN_FLIGHT:
        return Response(status_code=503, headers={"Retry-After": "5"})
    in_flight += 1
    try:
        return await call_next(request)
    finally:
        in_flight -= 1

How to check

Load staging with twice its tested limit. Extra requests get 503 within 100 ms, accepted requests stay within their speed objective, and the service recovers without a restart.