Agent WranglersEVT-23Recommended
Set aside messages that keep failing
After a set number of failed attempts, move a message or job to a separate dead letter queue and alert. Never retry it forever or drop it.
How to do it
Count attempts on each message, and back off between them, such as 5 attempts over an hour. After the limit, store the message with its last error, so a person can fix the cause and replay it. Most queues have this built in, such as a redrive policy or a failed jobs table.
How to check
Send a message that always fails. After the limit it sits in the dead letter queue with its error, an alert has fired, and other messages still flow.
Sources
Related rules
- API-34 Retry with backoff and jitter in API design
- API-78 Decide what happens when a provider is down in API design
- OPS-67 Alert on the age of the oldest waiting job in Operations and observability
- STK-35 Use BullMQ or RQ for background jobs in Software choices
- DEN-39 Commit stream offsets in the same transaction as the data in Data engineering
- EVT-17 Give every job a time limit in Integration and events
- ANA-2 Validate every event before you store it in Product analytics and experiments
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/events/background-jobs-and-queues/dead-letter-queue">Agent Wranglers integration and events playbook</a>. In code you write, add a comment next to it with the same link.