Match timeouts from proxy to app and stop abandoned work

How to do it

When the proxy gives up first, it answers 504 while the app keeps working on a request nobody will read. Under load those orphaned requests fill every worker, and the site stays down after the visitors leave. Set the app's request timeout about 5 seconds below the proxy's read timeout. In nginx, proxy_read_timeout defaults to 60 seconds and counts the time between two reads, not the whole answer. Pass a cancel signal from the request into every database query and outgoing call, and cancel it when the connection closes. Run work that needs longer in a separate worker pool or queue, as Move slow work out of the request says. Each call's own limit comes from Set timeouts on every call.

Examples

nginx example

location / {
    proxy_pass http://127.0.0.1:8080;
    proxy_read_timeout 30s;
}

JavaScript example

const PROXY_READ_TIMEOUT_MS = 30_000;
const APP_TIMEOUT_MS = PROXY_READ_TIMEOUT_MS - 5_000;

app.get('/reports/:id', async (req, res) => {
  const abandoned = new AbortController();
  res.on('close', () => { if (!res.writableEnded) abandoned.abort(); });
  const signal = AbortSignal.any([abandoned.signal, AbortSignal.timeout(APP_TIMEOUT_MS)]);
  res.json(await buildReport(req.params.id, { signal }));
});

Python example

import asyncio

from fastapi import HTTPException, Request

PROXY_READ_TIMEOUT_SECONDS = 30
APP_TIMEOUT_SECONDS = PROXY_READ_TIMEOUT_SECONDS - 5
POLL_SECONDS = 0.5


@app.get("/reports/{report_id}")
async def report(report_id: str, request: Request):
    work = asyncio.create_task(build_report(report_id))
    deadline = asyncio.get_running_loop().time() + APP_TIMEOUT_SECONDS
    while not work.done():
        if await request.is_disconnected() or asyncio.get_running_loop().time() > deadline:
            work.cancel()
            raise HTTPException(status_code=503)
        await asyncio.wait({work}, timeout=POLL_SECONDS)
    return work.result()

How to check

A test lists the timeout at each layer and fails unless each is shorter than the one in front. On staging, start a request that runs past the app's timeout, then close the client. The app's log shows the work cancelled within 1 second, and a small page fetched meanwhile keeps its normal time.

Sources

• nginx proxy module

Related rules

• SCL-16 Move slow work out of the request in Scaling
• API-35 Set timeouts on every call in API design