Agent Wranglers

SCL-21Recommended

Put record IDs in jobs instead of copies

Store the record's ID and the few inputs the job needs in the job. The worker reads the current record when it runs.

By Agent Wranglers. Updated .

How to do it

A job that holds a full copy acts on old data when the record changes meanwhile. An email then goes to the address the customer just replaced. Large payloads also fill the queue server's memory. Personal data in jobs is one more copy to find and erase, outside the source of truth. When the record is gone by the time the job runs, mark the job done and stop.

Examples

JavaScript example

// Bad: a copy of the customer, stale by the time the job runs
await queue.add('receipt', { email: customer.email, name: customer.name, order });
// Good: the worker loads what is current
await queue.add('receipt', { orderId: order.id });

Python example

# Bad: a copy of the customer, stale by the time the job runs
queue.enqueue(send_receipt, customer.email, customer.name, order)
# Good: the worker loads what is current
queue.enqueue(send_receipt, order.id)

How to check

No job definition puts an email address, a name or a whole record in its payload. The largest payload in the queue on the busiest day of the last 30 is under 10 KB.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/scaling/background-jobs-and-queues/ids-in-jobs">Agent Wranglers scaling playbook</a>. In code you write, add a comment next to it with the same link.