Time out every model call

How to do it

Set the total limit from the feature's spec, such as 30 seconds for chat and 10 seconds for a classification. Set an idle limit for streams, such as 15 seconds without a chunk. Retry rate limits and server errors, such as 429, 500, 503 and 529, with backoff and jitter, at most twice. Honour a retry-after header. Never retry a 429 that says the quota or credit is used up, because waiting will not fix it. Never retry once text has streamed to the user, because the user would see it twice. Official SDKs retry on their own, twice by default in the OpenAI and Anthropic SDKs. Set their maxRetries or max_retries to 0 when the gateway retries, or the attempts multiply. The general rules are timeouts and retry with backoff.

Example

const RETRYABLE = new Set([429, 500, 502, 503, 504, 529]);
const MAX_RETRIES = 2;

export async function callWithLimits(send: (signal: AbortSignal) => Promise<Response>, totalMs: number) {
  const deadline = AbortSignal.timeout(totalMs);
  for (let attempt = 0; ; attempt++) {
    const response = await send(deadline);
    if (response.ok || !RETRYABLE.has(response.status) || attempt === MAX_RETRIES) return response;
    const waitSeconds = Number(response.headers.get('retry-after')) || 2 ** attempt + Math.random();
    await new Promise(resolve => setTimeout(resolve, waitSeconds * 1000));
  }
}

RETRYABLE = {429, 500, 502, 503, 504, 529}
MAX_RETRIES = 2
CONNECT_SECONDS, IDLE_SECONDS = 5.0, 15.0

async def call_with_limits(client: httpx.AsyncClient, request: httpx.Request, total_seconds: float) -> httpx.Response:
    async with asyncio.timeout(total_seconds):
        for attempt in range(MAX_RETRIES + 1):
            response = await client.send(request, timeout=httpx.Timeout(CONNECT_SECONDS, read=IDLE_SECONDS))
            if response.is_success or response.status_code not in RETRYABLE or attempt == MAX_RETRIES:
                return response
            await asyncio.sleep(float(response.headers.get("retry-after") or 2 ** attempt + random.random()))

How to check

A test with a fake provider that never answers returns a timeout error within the limit plus one second. A fake that answers 503 twice and then succeeds gives one answer after three calls, not more. A search finds every SDK client built with its own retries set to 0.