Replay dead letters with a written procedure

How to do it

A dead letter queue that nobody empties is a slow data loss. Group the failed messages by error first, because one cause usually explains most of them. Fix the code or the data, deploy, then replay. Replay at a capped rate, such as 50 messages a second, so a backlog does not swamp the consumer. Amazon SQS has a redrive task with a maximum rate, and job libraries have a retry command for failed jobs. Replayed messages may be days old, so the consumer must skip what is already done, as Process each message once says. Drop a message only with a written reason, and log who dropped it. Alert when a dead letter queue holds any message for more than 1 day.

Examples

Shell example

# Amazon SQS: move dead letters back to their source queue at 50 messages a second
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:eu-north-1:111122223333:order-emails-dlq \
  --max-number-of-messages-per-second 50

JavaScript example

const REPLAY_PER_SECOND = 50;

export async function replayFailed(queue, { reason, operator }) {
  const failed = await queue.getFailed();
  for (const [index, job] of failed.entries()) {
    await job.retry();
    if ((index + 1) % REPLAY_PER_SECOND === 0) await new Promise(done => setTimeout(done, 1000));
  }
  log.info({ event: 'dead_letters.replayed', queue: queue.name, count: failed.length, reason, operator });
}

Python example

import time

REPLAY_PER_SECOND = 50

def replay_failed(registry, queue, reason: str, operator: str) -> None:
    job_ids = registry.get_job_ids()
    for index, job_id in enumerate(job_ids, start=1):
        registry.requeue(job_id)
        if index % REPLAY_PER_SECOND == 0:
            time.sleep(1)
    log.info("dead_letters.replayed", queue=queue.name, count=len(job_ids), reason=reason, operator=operator)

How to check

Each dead letter queue has a procedure in the runbook with a named owner. On staging, send 100 messages that fail, fix the cause and run the procedure. All 100 are processed once, the queue is empty, and the log names who replayed them.

Sources

• Amazon SQS dead-letter queue redrive

Related rules

• EVT-11 Process each message once in Integration and events