Cap each customer's running and queued jobs

How to do it

Per-client rate limits count requests, but one export can cost more than a thousand page views. Count running work per tenant, such as jobs and open exports, in a shared store. Beyond the cap, queue the work or answer 429. The Google SRE book sets quotas per customer, and under overload it refuses only the customers over their quota. Larger plans can get larger caps. Database time per tenant is measured and capped in Find and contain noisy neighbours.

Examples

JavaScript example

const MAX_RUNNING_EXPORTS = 3;

export async function startExport(redis, tenantId, start) {
  const key = `running-exports:${tenantId}`;
  if (await redis.incr(key) > MAX_RUNNING_EXPORTS) {
    await redis.decr(key);
    return { status: 429, retryAfterSeconds: 30 };
  }
  await redis.expire(key, 3600);
  return start({ onDone: () => redis.decr(key) });
}

Python example

MAX_RUNNING_EXPORTS = 3

def start_export(redis, tenant_id: str, start):
    key = f"running-exports:{tenant_id}"
    if redis.incr(key) > MAX_RUNNING_EXPORTS:
        redis.decr(key)
        return {"status": 429, "retry_after_seconds": 30}
    redis.expire(key, 3600)
    return start(on_done=lambda: redis.decr(key))

How to check

On staging, one tenant starts 100 exports at once. No more than its cap run together, and an export from another tenant starts within 30 seconds.

Sources

• Google SRE book, Handling Overload

Related rules

• DAT-73 Find and contain noisy neighbours in Data and databases
• APS-49 Limit how often anyone can call your APIs in Application security
• AGT-24 Set quotas per agent grant in Agents as users
• USG-13 Alert when a customer costs more than they pay in Usage-based billing
• MED-27 Count each owner's storage and enforce a quota in Files and media