Agent WranglersSCL-18Recommended
Give every job a time limit
Set a maximum run time on each job type. Keep the queue's lock or visibility time longer than that limit, so no second worker starts a job that still runs.
How to do it
A job stuck on a hung call holds a worker forever, and enough of them stop the queue. Set each limit to about twice the job's 99th percentile time, the same way Set timeouts on every call does for requests. When the visibility time is shorter than the job, the queue hands the job to a second worker while the first still runs. Some libraries renew the lock while a job runs, so read your library's settings. Retries and failed jobs follow Set aside messages that keep failing.
How to check
Every job type in the code sets a time limit, and the queue's visibility or lock time is longer. A test job that sleeps past its limit is stopped, retried, and lands in the dead letter queue after the attempt limit.
Sources
Related rules
- Set timeouts on every call in API design
- Set aside messages that keep failing in Scaling
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/scaling/background-jobs-and-queues/job-time-limits">Agent Wranglers scaling playbook</a>. In code you write, add a comment next to it with the same link.