Kafka vs RabbitMQ vs Amazon SQS
Asynchronous messaging decouples services in time — producers send without waiting for consumers. Kafka is a distributed event log for high-throughput streams and replay. RabbitMQ is a broker with rich exchange→queue routing. Amazon SQS is a fully managed AWS queue with almost no broker ops.
Wrong fit hurts: Kafka for a tiny task queue is heavy; SQS when you need multi-day replay falls short; RabbitMQ when you need log-style fan-out across many independent readers fights the model. This guide is broker choice. For AMQP shapes inside RabbitMQ (work queue vs fanout), see RabbitMQ Messaging Patterns.
Place the broker inside a complete event-driven architecture, then use the microservices design patterns guide to handle outbox, saga, and read-model boundaries.
Apache Kafka
Kafka stores messages in append-only topic partitions — an ordered, durable log. Producers write to topics; consumer groups read at their own pace. Unlike a classic queue where one consumer takes a message away, independent groups can each replay the same stream from retained offsets.
Best for event sourcing backbones, analytics, audit trails, and “new service catches up from history.” Trade-off: clusters, partitions, and consumer lag are real ops work — overkill for a single email-job queue.
Quick reference
- Architecture: partitioned commit log; offset-based consumers.
- Best for: streaming, analytics, audit, multi-group replay.
- Retention lets new consumers read history (hours/days — you configure it).
- Trade-off: operational complexity vs simple task queues.
- When not: one worker pool draining jobs with no replay need — RabbitMQ/SQS.
Remember this
Kafka when you need an ordered, replayable event log at scale.
RabbitMQ
RabbitMQ is a traditional message broker. Producers publish to exchanges; exchanges route to queues (direct, topic, fanout, headers). Consumers pull from queues. After ack, the message is gone from that queue — no built-in multi-day log replay like Kafka.
Best for task distribution, work queues, and complex routing. A tuned Kafka log often sustains higher sustained publish throughput on the same hardware class for append-heavy workloads, but that gap depends on batching, persistence, and consumer design — measure your path. RabbitMQ's setup and routing flexibility are friendlier for many microservice workloads. Pattern detail (competing consumers vs fanout) is in the RabbitMQ patterns guide.
Quick reference
- Architecture: exchange routes → queue; competing consumers share work.
- Best for: tasks, request-reply, flexible routing.
- DLX handles failed messages when you wire it.
- Trade-off: consumed messages are not a durable multi-group log.
- When not: many independent readers need the same historical stream — Kafka.
Remember this
RabbitMQ for flexible routing and task queues — not Kafka-style replay.
Amazon SQS
Amazon SQS is a fully managed queue: create a queue, send messages, consumers poll (or Lambda triggers). AWS handles scaling and patching. Standard queues: at-least-once delivery, best-effort ordering, very high throughput. FIFO queues: ordering within a MessageGroupId, and AWS’s exactly-once processing means duplicate sends with the same deduplication id are suppressed inside a 5-minute deduplication window — not a magical “never think about duplicates” guarantee for your business logic.
Visibility timeout hides an in-flight message; if the worker dies before DeleteMessage, the message reappears. Dead-letter queues catch repeated failures. Throughput quotas for FIFO are lower than Standard and depend on region, batching, and whether high-throughput FIFO mode is enabled — check current AWS quota docs rather than memorizing a single TPS number. Trade-offs: AWS lock-in and no Kafka-style multi-day replay after delete.
Quick reference
- Architecture: managed poll-based queues (Standard or FIFO).
- Best for: AWS-native workers, Lambda triggers, simple async jobs.
- FIFO: order per MessageGroupId; dedup id / content-based dedup within ~5 minutes.
- Still design idempotent consumers — receive can retry after visibility expiry.
- FIFO throughput is quota-limited (region/config); Standard is the high-volume default.
- When not: cross-cloud or long retention replay — Kafka (or self-hosted broker).
Remember this
SQS when you want managed AWS queues — qualify FIFO claims and stay idempotent.
Which Broker Should You Choose?
Decide on replay, routing, throughput shape, and who runs the cluster. Need an event log and multiple consumer groups reading history? Kafka. Need flexible AMQP routing and work queues you operate yourself? RabbitMQ. On AWS and want zero broker VMs? SQS.
Many teams use Kafka as the event backbone and SQS/Rabbit for per-service work queues. Start with the simplest option that preserves the property you refuse to lose — then migrate on evidence (lag, routing pain, replay need), not fashion.
Quick reference
- Event log + replay + multi-group fans → Kafka.
- Task queues + exchange routing, self-hosted → RabbitMQ.
- AWS-native, serverless workers, minimal ops → SQS.
- All three: assume at-least-once to the handler unless you proved otherwise.
- Wire DLQ/DLX and idempotent consumers before launch.
Remember this
Kafka for streams; RabbitMQ for routing; SQS for managed AWS queues.
Key takeaway
Kafka, RabbitMQ, and SQS solve different jobs. Kafka is a replayable log. RabbitMQ routes and distributes work. SQS removes broker ops on AWS — with FIFO ordering/dedup rules that still require idempotent handlers.
Practice (20 min): For an OrderPlaced event, write one sentence naming (1) which broker you pick and (2) one property you refuse to lose (replay, routing key, or zero ops). Then paste the matching consume sketch above and add a deliberate failure (throw before ack/delete) to prove the message retries instead of vanishing.
Related Articles
Explore this topic