Process each message once

How to do it

A worker can crash after the work but before it acknowledges the message, and a sender retries when an answer is slow. Take the key from the message ID, or from the job's purpose, such as invoice:2026-10:cust_481. Record it under a unique constraint in the same transaction as the work, and skip the message when the key exists. When several consumers share one store, key each row on the consumer's name and the message ID. A consumer that writes to another store keeps its stream position there, as Commit stream offsets in the same transaction as the data shows. Pass the key on to other services as their idempotency key, as Send the same idempotency key when you retry a write says. Keep processed keys at least as long as the sender may still retry, such as the 24 hours or more of a webhook sender. Webhook receivers answer 2xx within a few seconds and queue the work, so the sender does not time out and retry. HTTP clients that retry a POST use idempotency keys instead.

Examples

JavaScript example

export async function sendInvoice(job) {
  const key = `invoice:${job.data.month}:${job.data.customerId}`;
  await db.tx(async tx => {
    const claimed = await tx.result('INSERT INTO processed_messages (key) VALUES ($1) ON CONFLICT DO NOTHING', [key]);
    if (claimed.rowCount === 0) return;
    const invoice = await createInvoice(tx, job.data);
    await tx.none('INSERT INTO outbox (kind, ref) VALUES ($1, $2)', ['invoice-email', invoice.id]);
  });
}

Python example

def handle_event(event: dict) -> None:
    with conn.transaction():
        claimed = conn.execute(
            "INSERT INTO processed_messages (key) VALUES (%s) ON CONFLICT DO NOTHING", (event["id"],)
        ).rowcount
        if claimed == 0:
            return
        apply_event(conn, event)

How to check

Run each job type twice in a test with the same input. The second run changes 0 rows and sends nothing. Deliver the same webhook event twice, and the receiver acts once and answers 2xx both times. Kill a worker after the side effect and before it acknowledges, and let it retry. Exactly one effect exists.

Sources

• Microsoft Azure Architecture Center, Idempotent Consumer pattern
• The Twelve-Factor App, Disposability

Related rules

• ARC-91 Run a change across services as a saga with undo steps in Clean code and architecture
• API-32 Accept idempotency keys on POST that creates or charges in API design
• DAT-68 Run every background job inside its tenant in Data and databases
• PAY-18 Let provider events move the order in Payments and billing
• MOB-5 Verify every store purchase on your server in Mobile apps
• DEN-7 Make every load safe to rerun in Data engineering
• ECM-26 Buy each shipping label once in E-commerce and fulfilment
• ECM-27 Update delivery status from carrier events in E-commerce and fulfilment
• EVT-2 Wrap every event in one envelope in Integration and events
• DEN-39 Commit stream offsets in the same transaction as the data in Data engineering
• API-76 Send the same idempotency key when you retry a write in API design
• EVT-14 Keep events long enough to replay them in Integration and events
• EVT-15 Undo failed multi-step work with compensating steps in Integration and events
• EVT-24 Replay dead letters with a written procedure in Integration and events
• EVT-28 Store each incoming webhook before you act on it in Integration and events