Give each customer a fair share

How to do it

Per-client rate limits count requests, but one export can cost more than a thousand page views. Count running work per tenant, such as jobs and open exports, in a shared store. Beyond the cap, queue the work or answer 429. The Google SRE book sets quotas per customer, and under overload it refuses only the customers over their quota. Larger plans can get larger caps.

Examples

JavaScript example

const MAX_RUNNING_EXPORTS = 3;

export async function startExport(redis, tenantId, start) {
  const key = `running-exports:${tenantId}`;
  if (await redis.incr(key) > MAX_RUNNING_EXPORTS) {
    await redis.decr(key);
    return { status: 429, retryAfterSeconds: 30 };
  }
  await redis.expire(key, 3600);
  return start({ onDone: () => redis.decr(key) });
}

Python example

MAX_RUNNING_EXPORTS = 3

def start_export(redis, tenant_id: str, start):
    key = f"running-exports:{tenant_id}"
    if redis.incr(key) > MAX_RUNNING_EXPORTS:
        redis.decr(key)
        return {"status": 429, "retry_after_seconds": 30}
    redis.expire(key, 3600)
    return start(on_done=lambda: redis.decr(key))

How to check

On staging, one tenant starts 100 exports at once. No more than its cap run together, and an export from another tenant starts within 30 seconds.

Sources

• Google SRE book, Handling Overload

Related rules

• Limit how often anyone can call your APIs in Website