Make every job safe to run twice

How to do it

Give each job a key from its purpose, such as invoice:2026-10:cust_481. Record the key with a unique constraint in the same transaction as the work, and skip the job when the key exists. This is Make writes safe to repeat applied to jobs. Pass the key on to other services as their idempotency key, as Create each charge with an idempotency key does. On a stop signal, a worker returns its current job to the queue, so the rerun must be safe.

Examples

JavaScript example

export async function sendInvoice(job) {
  const key = `invoice:${job.data.month}:${job.data.customerId}`;
  await db.tx(async tx => {
    const claimed = await tx.result('INSERT INTO done_jobs (key) VALUES ($1) ON CONFLICT DO NOTHING', [key]);
    if (claimed.rowCount === 0) return;
    const invoice = await createInvoice(tx, job.data);
    await tx.none('INSERT INTO outbox (kind, ref) VALUES ($1, $2)', ['invoice-email', invoice.id]);
  });
}

Python example

def send_invoice(month: str, customer_id: str) -> None:
    key = f"invoice:{month}:{customer_id}"
    with conn.transaction():
        claimed = conn.execute(
            "INSERT INTO done_jobs (key) VALUES (%s) ON CONFLICT DO NOTHING", (key,)
        ).rowcount
        if claimed == 0:
            return
        invoice = create_invoice(conn, month, customer_id)
        conn.execute("INSERT INTO outbox (kind, ref) VALUES (%s, %s)", ("invoice-email", invoice.id))

How to check

Run each job type twice in a test with the same input. The second run changes 0 rows and sends nothing. Kill a worker after the side effect and before it acknowledges the job, and let it retry. Exactly one effect exists.

Sources

• AWS Well-Architected, REL04-BP04 Make mutating operations idempotent
• The Twelve-Factor App, Disposability

Related rules

• Make writes safe to repeat in Data
• Create each charge with an idempotency key in Payments