Kafka is a partitioned, replayable log suited to high-throughput event streaming and multiple independent consumers; RabbitMQ is a broker with rich routing and per-message acknowledgement suited to work queues and task distribution. Either way, resilience depends on idempotent consumers, explicit ordering decisions and a deliberate dead-letter strategy.
Kafka or RabbitMQ: choose by access pattern
The two systems differ in what they fundamentally are. Kafka is a partitioned, append-only log: consumers read at their own pace, retain their own position, and can replay history. That makes it excellent for event streaming, for feeding several independent consumers from one stream, and for rebuilding derived state.
RabbitMQ is a message broker: it routes messages to queues and removes them once acknowledged. That makes it excellent for task distribution, work queues, request-reply, complex routing topologies and scenarios where per-message acknowledgement and priority matter.
A common and sound pattern is to use both. Kafka carries the domain event stream; a consumer translates relevant events into RabbitMQ work items for slow, retryable tasks such as document generation or third-party calls.
Delivery guarantees and idempotency
Exactly-once delivery across a network boundary is not achievable in the general case; exactly-once processing is achievable, and only by making consumers idempotent. Every practical system therefore runs at-least-once delivery with deduplication on the consumer side.
Deduplication requires a stable event identity and a durable record of what has already been processed. The usual implementation is a unique event identifier stored in the same transaction as the business change, so the write and the deduplication record commit or fail together.
This is why the transactional outbox pattern matters. Writing to the database and publishing to a broker as two separate operations creates a window where one succeeds and the other does not. An outbox table committed with the business change, then drained by a publisher, makes the pair atomic.
- Assume at-least-once delivery and deduplicate on a stable event identifier.
- Commit the deduplication record in the same transaction as the business change.
- Use a transactional outbox instead of dual-writing to database and broker.
- Make replay safe, because replay will eventually be necessary.
Ordering and partitioning
Kafka guarantees order only within a partition. Choose the partition key so that events which must be ordered together land together, typically the aggregate identifier such as an order or account number. Choosing a random or round-robin key maximises throughput and destroys ordering.
More partitions increase parallelism but also increase rebalance time and reduce the effectiveness of per-key ordering. Partition counts should be set with the expected consumer parallelism in mind and increased only with care, because increasing partitions changes key-to-partition assignment.
When strict global ordering is genuinely required, the honest answer is usually that the aggregate needs to be single-threaded, either by routing all its events to one partition or by using a serialised command handler in front.
| Failure | Handling | Notes |
|---|---|---|
| Transient downstream error | Bounded retry with backoff | Retry budget must be finite |
| Permanent processing error | Dead-letter topic or queue | Include payload, headers, error and offset |
| Poison message blocking a partition | Skip with alert after N attempts | Never block the partition indefinitely |
| Duplicate delivery | Idempotent consumer | Deduplicate on event identifier |
| Consumer lag growth | Alert on lag, scale consumers | Lag is the primary health metric |
| Broker unavailable | Producer buffer plus backpressure | Fail writes fast rather than corrupting order |
Operating event-driven systems
The primary health metric is consumer lag, not error rate. A system can be error-free and still be hours behind, which means derived state is stale and the business is making decisions on old data. Alert on lag growing over time, not just on a static threshold.
Schema evolution needs the same discipline as API contracts. Register schemas, evolve them compatibly, and reject incompatible writes at the producer rather than letting a consumer fail in production. Test the consumer against the previous schema version in continuous integration.
Finally, treat topic retention as a product decision. Retention determines how far back you can replay, which determines how quickly you can rebuild a projection or recover from a bad deployment.
- Alert on consumer lag trend, not only absolute lag.
- Register and compatibly evolve event schemas.
- Decide retention from your replay and recovery requirements.
- Rehearse replay and dead-letter recovery before you need them.
Frequently asked questions
Should I use Kafka or RabbitMQ for microservices?
Use Kafka when several consumers need the same stream, when you need replay, or when throughput is high. Use RabbitMQ when you need task distribution, complex routing, per-message acknowledgement and priority. Many architectures sensibly use both.
How do I achieve exactly-once processing?
You achieve exactly-once processing, not exactly-once delivery, by making consumers idempotent: give every event a stable identifier and record processed identifiers in the same transaction as the business change.
What causes message reordering?
Reordering happens when related events are spread across partitions or when multiple consumers process the same aggregate concurrently. Fix it by choosing a partition key derived from the aggregate identifier and by serialising per-aggregate processing.
References
Engineer your next strategic platform
Estimate your project investment and delivery timeline instantly, or schedule an architectural discovery call with our engineering leadership.