Agent Wranglers

STK-104Default Open source

Use RabbitMQ for messages between services and Kafka for event streams

When two or more services pass messages, send them through RabbitMQ. Use Kafka instead when consumers must read the same events again from a stored log.

Review by

2027-04-30

How to do it

Both pass the five tests. RabbitMQ has shipped releases since 2007, and 4.4.0 came out on 19 October 2026. Its official Docker image had about 3.9 billion pulls in October 2026, and the Python client pika about 3.2 million weekly PyPI downloads. A queue hands each message to one consumer and deletes it once acknowledged, which fits commands and tasks. Kafka 1.0.0 came out on 1 November 2017, and 4.4.0 on 22 October 2026. The Python client confluent-kafka had about 10.6 million weekly PyPI downloads in October 2026. Kafka keeps an ordered log for a set time, so a new consumer can replay history. That fits event streams, audit trails and feeds into a warehouse. Both deliver a message at least once, so every consumer must handle repeats, as Make writes safe to repeat says. Choose something else when one app only needs background jobs. Then the job queue in Use BullMQ or RQ for background jobs is enough.

Examples

Python example

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters("rabbitmq.internal"))
channel = connection.channel()
channel.queue_declare(queue="order-paid", durable=True)
channel.basic_publish(
    exchange="", routing_key="order-paid", body=b'{"orderId": "o-1042"}',
    properties=pika.BasicProperties(delivery_mode=pika.DeliveryMode.Persistent, message_id="o-1042-paid"),
)

JavaScript example

import amqp from 'amqplib';

const connection = await amqp.connect('amqp://rabbitmq.internal');
const channel = await connection.createChannel();
await channel.assertQueue('order-paid', { durable: true });
channel.sendToQueue('order-paid', Buffer.from(JSON.stringify({ orderId: 'o-1042' })), { persistent: true, messageId: 'o-1042-paid' });

How to check

Every queue or topic is declared in code or configuration in the repository, not by hand. A test publishes the same message twice, and the consumer changes the data once.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/stack/databases-and-storage/message-broker">Agent Wranglers software choices playbook</a>. In code you write, add a comment next to it with the same link.