Agent WranglersTransactions and concurrency Recommended
Set aside messages that keep failing
After a set number of failed attempts, move a message or job to a separate dead letter queue and alert. Never retry it forever or drop it.
How to do it
Count attempts on each message, and back off between them, such as 5 attempts over an hour. After the limit, store the message with its last error, so a person can fix the cause and replay it. Most queues have this built in, such as a redrive policy or a failed jobs table.
How to check
Send a message that always fails. After the limit it sits in the dead letter queue with its error, an alert has fired, and other messages still flow.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers data playbook: https://agentwranglers.se/playbook/data/transactions-and-concurrency/dead-letter-queue.