Queue model calls within the provider's rate limits

How to do it

Hosted model providers limit requests and tokens per minute for each account and model, and answer 429 above them. Every copy of the app shares those limits, so count in a shared store. Let interactive calls go first, and move work nobody waits for to a batch, as Send work nobody waits for as a batch says. Read the limits from the provider's response headers or console, and ask for a raise before a known peak. On your own GPUs, the server batches requests on the fly, so CPU and even GPU use say little about load. Scale on requests waiting in the server's queue, and cap the number running per GPU from a load test. Spending limits are in Price every model call and cap the spend.

Examples

JavaScript example

const TOKENS_PER_MINUTE = 400_000;

export async function reserveTokens(redis, model, tokens) {
  const key = `model-tokens:${model}:${Math.floor(Date.now() / 60_000)}`;
  const used = await redis.incrby(key, tokens);
  await redis.expire(key, 120);
  if (used <= TOKENS_PER_MINUTE) return true;
  await redis.decrby(key, tokens);
  return false;
}

Python example

import time

TOKENS_PER_MINUTE = 400_000

def reserve_tokens(redis, model: str, tokens: int) -> bool:
    key = f"model-tokens:{model}:{int(time.time() // 60)}"
    used = redis.incrby(key, tokens)
    redis.expire(key, 120)
    if used <= TOKENS_PER_MINUTE:
        return True
    redis.decrby(key, tokens)
    return False

How to check

Under the load test at twice the planned peak, fewer than 1% of model calls get a 429. Interactive calls keep their speed objective. Self-hosted inference has an autoscaling rule on waiting requests with a minimum and a maximum.

Sources

• OpenAI rate limits
• Claude API rate limits

Related rules

• AIF-45 Send work nobody waits for as a batch in AI features
• AIF-42 Price every model call and cap the spend in AI features