Bound the backlog and push back on producers

How to do it

A backlog that grows for hours fills the queue server's memory and delivers work nobody wants any more, such as an expired reset link. Set the limit from the delay users accept, which Alert on the age of the oldest waiting job measures. When full, answer the producer with 503 and Retry-After, as Turn work away before overload does. Or drop work that has lost its value, and count it in the metrics. AWS lists long backlogs and missing queue age metrics as anti-patterns. Queues inside a process follow Cap every collection that grows with use.

Examples

JavaScript example

const MAX_WAITING = 50_000;

export async function enqueueExport(queue, data) {
  if (await queue.getWaitingCount() >= MAX_WAITING) {
    throw new ServiceBusyError('export queue full', { retryAfterSeconds: 60 });
  }
  return queue.add('export', data);
}

Python example

MAX_WAITING = 50_000

def enqueue_export(queue, data):
    if queue.count >= MAX_WAITING:
        raise ServiceBusy("export queue full", retry_after_seconds=60)
    return queue.enqueue(run_export, data)

How to check

Stop the workers on staging and add jobs past the limit. New jobs are refused or slowed with a logged reason, and the queue server's memory stays below its limit. Every queue has a written maximum backlog.

Sources

• AWS Well-Architected, REL05-BP04 Fail fast and limit queues
• Google SRE book, Addressing Cascading Failures

Related rules

• Alert on the age of the oldest waiting job in Operations
• Turn work away before overload in Operations
• Cap every collection that grows with use in Architecture