---
title: "Queue model calls within the provider's · Agent Wranglers"
url: https://agentwranglers.se/playbook/scaling/load-balancing-and-autoscaling/model-rate-limits
id: SCL-45
level: Recommended
applies: ai-features
updated: 2026-10-27
build: a2a6bdfed830
tokens: 1016
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Queue model calls within the provider's rate limits

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Load balancing and autoscaling](https://agentwranglers.se/playbook/scaling/load-balancing-and-autoscaling.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: SCL-45
- Level: Recommended
- Chapter: [Load balancing and autoscaling](https://agentwranglers.se/playbook/scaling/load-balancing-and-autoscaling.md)
- Rule: Count the requests and tokens you send to each model provider per minute, and hold calls in a queue before they pass the provider's limit. On your own inference servers, scale on waiting requests, not on CPU.
- Check: Under the load test at twice the planned peak, fewer than 1% of model calls get a 429. Interactive calls keep their speed objective. Self-hosted inference has an autoscaling rule on waiting requests with a minimum and a maximum.
- Applies when: The project uses AI models.
- How: Hosted model providers limit requests and tokens per minute for each account and model, and answer 429 above them. Every copy of the app shares those limits, so count in a shared store. Let interactive calls go first, and move work nobody waits for to a batch, as [Send work nobody waits for as a batch](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/batch-offline-work.md) says. Read the limits from the provider's response headers or console, and ask for a raise before a known peak. On your own GPUs, the server batches requests on the fly, so CPU and even GPU use say little about load. Scale on requests waiting in the server's queue, and cap the number running per GPU from a load test. Spending limits are in [Price every model call and cap the spend](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/cost-per-call.md).

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### JavaScript example

```js
const TOKENS_PER_MINUTE = 400_000;

export async function reserveTokens(redis, model, tokens) {
  const key = `model-tokens:${model}:${Math.floor(Date.now() / 60_000)}`;
  const used = await redis.incrby(key, tokens);
  await redis.expire(key, 120);
  if (used <= TOKENS_PER_MINUTE) return true;
  await redis.decrby(key, tokens);
  return false;
}
```

### Python example

```python
import time

TOKENS_PER_MINUTE = 400_000

def reserve_tokens(redis, model: str, tokens: int) -> bool:
    key = f"model-tokens:{model}:{int(time.time() // 60)}"
    used = redis.incrby(key, tokens)
    redis.expire(key, 120)
    if used <= TOKENS_PER_MINUTE:
        return True
    redis.decrby(key, tokens)
    return False
```

## Sources

- [OpenAI rate limits](https://developers.openai.com/api/docs/guides/rate-limits)
- [Claude API rate limits](https://platform.claude.com/docs/en/api/rate-limits)

## Related rules

- AIF-45 [Send work nobody waits for as a batch](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/batch-offline-work.md) in AI features
- AIF-42 [Price every model call and cap the spend](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/cost-per-call.md) in AI features

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/scaling/load-balancing-and-autoscaling/model-rate-limits">Agent Wranglers scaling playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Cap serverless concurrency and measure cold starts](https://agentwranglers.se/playbook/scaling/load-balancing-and-autoscaling/serverless-concurrency.md)
- [Load balancing and autoscaling](https://agentwranglers.se/playbook/scaling/load-balancing-and-autoscaling.md)