---
title: "Background jobs and queues · Agent Wranglers"
url: https://agentwranglers.se/playbook/scaling/background-jobs-and-queues
updated: 2026-10-21
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Background jobs and queues

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Scaling](https://agentwranglers.se/playbook/scaling.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers). Updated 2026-10-21.

> Move slow work out of requests into queues. Make every job safe to run twice, bound the backlog, and keep urgent work ahead of bulk work.

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/scaling/background-jobs-and-queues">Agent Wranglers scaling playbook</a>`. In code you write, add a comment next to it with the same link.

## Required

- SCL-17 [Make every job safe to run twice](https://agentwranglers.se/playbook/scaling/background-jobs-and-queues/idempotent-jobs): Write each job so that running it twice has the same effect as running it once. Queues deliver at least once, and a worker can crash after the work but before it marks the job done.
- SCL-19 [Bound the backlog and push back on producers](https://agentwranglers.se/playbook/scaling/background-jobs-and-queues/bounded-backlog): Set a maximum backlog for every queue, as a count or an age. When it is reached, slow or refuse new work instead of letting the backlog grow without end.

## Recommended

- SCL-16 [Move slow work out of the request](https://agentwranglers.se/playbook/scaling/background-jobs-and-queues/slow-work-off-requests): Run work the visitor does not need in the answer as a background job. Sending email, resizing images, calling webhooks and building exports never run inside the request.
- SCL-18 [Give every job a time limit](https://agentwranglers.se/playbook/scaling/background-jobs-and-queues/job-time-limits): Set a maximum run time on each job type. Keep the queue's lock or visibility time longer than that limit, so no second worker starts a job that still runs.
- SCL-20 [Keep urgent jobs in their own queue](https://agentwranglers.se/playbook/scaling/background-jobs-and-queues/urgent-queue): Put jobs a person waits for, such as sign-in codes and password resets, in a separate queue with its own workers. Bulk work, such as exports and newsletters, never delays them.
- SCL-21 [Put record IDs in jobs instead of copies](https://agentwranglers.se/playbook/scaling/background-jobs-and-queues/ids-in-jobs): Store the record's ID and the few inputs the job needs in the job. The worker reads the current record when it runs.
- SCL-22 [Spread scheduled work across the hour](https://agentwranglers.se/playbook/scaling/background-jobs-and-queues/spread-scheduled-jobs): Give each scheduled job its own start minute instead of the full hour. Split jobs that touch every customer into batches with gaps between them.
- SCL-34 [Set aside messages that keep failing](https://agentwranglers.se/playbook/scaling/background-jobs-and-queues/dead-letter-queue): After a set number of failed attempts, move a message or job to a separate dead letter queue and alert. Never retry it forever or drop it.
- SCL-38 [Run each scheduled job once at a time](https://agentwranglers.se/playbook/scaling/background-jobs-and-queues/single-job-runs): Take a lock before each scheduled job starts, so a second copy skips its run. Jobs that charge, send or delete must also be safe to run twice.

- Previous: [Caching](https://agentwranglers.se/playbook/scaling/caching)
- Next: [Growing the database](https://agentwranglers.se/playbook/scaling/growing-the-database)
- [All chapters](https://agentwranglers.se/playbook/scaling)