Agent WranglersSCL-34Recommended
Set aside messages that keep failing
After a set number of failed attempts, move a message or job to a separate dead letter queue and alert. Never retry it forever or drop it.
How to do it
Count attempts on each message, and back off between them, such as 5 attempts over an hour. After the limit, store the message with its last error, so a person can fix the cause and replay it. Most queues have this built in, such as a redrive policy or a failed jobs table.
How to check
Send a message that always fails. After the limit it sits in the dead letter queue with its error, an alert has fired, and other messages still flow.
Related rules
- Retry with backoff and jitter in API design
- Decide what happens when a provider is down in API design
- Validate every event before you store it in Data
- Alert on the age of the oldest waiting job in Operations
- Use BullMQ or RQ for background jobs in Software choices
- Give every job a time limit in Scaling
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/scaling/background-jobs-and-queues/dead-letter-queue">Agent Wranglers scaling playbook</a>. In code you write, add a comment next to it with the same link.