Why queues
A message queue lets one service hand off work without waiting for it. The producer writes a message and returns; consumers process it at their own pace. This gives you:
- Faster responses: send the email, resize the image or update the search index after replying to the user.
- Load levelling: a spike piles up in the queue instead of crushing the database.
- Decoupling: producers don't need to know who consumes, and consumers can be down briefly without losing work.
- Fan-out: one event (order placed) can drive many independent consumers (email, analytics, inventory).
Queues and logs
| Message queue | Event log | |
|---|---|---|
| Examples | RabbitMQ, SQS | Kafka, Kinesis, Pulsar |
| After consumption | Message is deleted | Message stays (retention period); consumers track their offset |
| Multiple consumers | Competing consumers share the work | Each consumer group reads everything independently |
| Replay | No | Yes: rewind the offset |
| Ordering | Usually best effort | Ordered within a partition |
Delivery guarantees
- At most once: may lose messages, never duplicates (acknowledge before processing).
- At least once: never loses, may duplicate (acknowledge after processing). The usual choice.
- Exactly once: hard end to end; in practice it's at-least-once delivery plus idempotent processing.
Idempotency means processing the same message twice has the same effect as once. Give each message a unique id and record processed ids (or use upserts, or conditional updates like "set status = PAID where status = PENDING").
void handle(PaymentEvent e) {// insert fails if this event id was already processedif (!processed.insertIfAbsent(e.id())) return;ledger.credit(e.account(), e.amount());}
Messages that fail repeatedly should go to a dead-letter queue after a few retries, so one bad message doesn't block the rest.
Ordering and back-pressure
Global ordering limits you to one consumer. Most systems need order only per entity, so they partition by a key (user id, order id): all events for one key land in one partition and are processed in order, while different keys run in parallel.
If consumers fall behind, the queue grows. Watch consumer lag, autoscale consumers, and decide what happens at the limit: block producers, shed low-priority work, or drop stale messages.
Uploading a photo with async processing
The upload API saves the original file to object storage.
See it animated
Interview questions
Try answering out loud first, then open the model answer.
Q1When would you add a message queue?beginner
When work doesn't need to finish before replying (emails, thumbnails, analytics), when traffic is bursty and the downstream can't absorb peaks, or when several independent services need to react to the same event. It trades immediacy for resilience and throughput.
Q2Kafka vs RabbitMQ?intermediate
RabbitMQ is a broker that routes messages to queues and deletes them once acknowledged: good for task queues and flexible routing. Kafka is a partitioned, replicated append-only log that keeps messages for a retention period: very high throughput, replay, and many independent consumer groups. Use Kafka for event streams, RabbitMQ (or SQS) for work queues.
Q3How do you get exactly-once processing?advanced
Accept at-least-once delivery and make the consumer idempotent: deduplicate by message id, use upserts or conditional writes, or commit the consumer offset in the same transaction as the side effect (the transactional outbox/inbox patterns).
Q4How do you keep events for the same order in sequence?intermediate
Partition by order id so all its events go to the same partition, which one consumer reads in order. Different orders still process in parallel across partitions.
Q5What is the transactional outbox pattern?advanced
To update the database and publish an event atomically, write the event to an outbox table in the same transaction as the business change. A separate relay reads the outbox (or the DB's change log) and publishes to the broker, retrying until it succeeds. This avoids the dual-write problem where one succeeds and the other fails.
Quick check
0 of 3 answered