Choose the partition key for the order you need

How to do it

Kafka and similar logs hash the key to pick a partition, so all events with one key arrive in the order they were written. Events without a key spread over partitions and lose their order. Pick the narrowest key that still covers what must stay in order. An account ID keeps every event of a large customer on one partition, and one busy key then slows that whole partition. Set enough partitions for the consumers you plan, because adding partitions later moves keys to other partitions and breaks order for a while. Queues that are not logs, such as Amazon SQS FIFO queues, use a message group ID the same way. Consumers still guard against old events, as Don't rely on event order says.

Examples

JavaScript example

await producer.send({
  topic: 'shop.orders',
  messages: [{ key: order.id, value: JSON.stringify(envelope('com.example.shop.order.paid.v1', { orderId: order.id })) }],
});

Python example

producer.produce(
    "shop.orders",
    key=order.id,
    value=json.dumps(envelope("com.example.shop.order.paid.v1", {"orderId": order.id}, SCHEMA_URL)),
)

How to check

A unit test publishes three events for one order through the producer code, and all three carry the order ID as their key. On staging, a consumer reads 1,000 events for 10 orders and finds each order's events in write order.

Sources

• Apache Kafka design, the producer and partitions

Related rules

• EVT-12 Don't rely on event order in Integration and events
• EVT-13 Alert when a consumer falls behind in Integration and events