Integration and events playbook

An integration and events playbook for any system where one part tells another that something happened. That covers message brokers such as Kafka, RabbitMQ or Amazon SQS, and job queues inside one app. It also covers change data capture and webhooks to and from other companies.

It starts with contracts, because an event without a written schema breaks its readers the day it changes. Then it covers publishing only what was committed, consuming each message once whatever the order, and background jobs that never run twice or wait forever. The last chapter covers webhooks you send and receive.

Messages arrive at least once, sometimes late and sometimes out of order. Every rule here assumes that, and each has a check you can run on staging by sending, delaying or repeating a message.

HTTP requests that retry follow the API design playbook, and warehouse loads follow the data engineering playbook. Email and text providers' delivery events are webhooks too, so the email and notifications playbooks link here.

Chapters

1. Contracts and schemas

   Describe every channel in a contract and wrap each event in one envelope. Let a registry refuse changes that break the services that read them.

   • Required 1
   • Recommended 4

2. Publishing events

   Publish only what was committed, through an outbox or the database log. Pick partition keys for the order you need, and share events between servers.

   • Required 1
   • Recommended 4

3. Consuming events

   Process each message once, never trust arrival order, alert when consumers fall behind, keep the log long enough to replay, and undo failed multi-step work.

   • Required 1
   • Recommended 4

4. Background jobs and queues

   Move slow work out of requests into queues. Limit each job's time, bound the backlog, keep urgent work first, and set aside and replay what keeps failing.

   • Required 1
   • Recommended 8

5. Webhooks

   Sign every webhook, send only to public addresses and retry until accepted. Store what arrives before acting, and let customers test their endpoints.

   • Required 2
   • Recommended 3

All 29 rules on one page

This is a lot to check. An Expert can check it for you.

Related playbooks

• Reliability

  In the API design playbook. Clients that retry an HTTP request send an idempotency key, the request-side twin of a consumer that runs once.

• Transactions and concurrency

  In the Data and databases playbook. Each consumer records its work and its processed key in one database transaction.

• Alerts and on-call

  In the Operations and observability playbook. Queue age and consumer lag only help when an alert wakes someone.

• Payment events and reconciliation

  In the Payments and billing playbook. Payment webhooks arrive late, twice or never. Reconcile them against the provider every hour.