GamifiedGamified challengesPracticeRaceLeaderboardPricing

Learn data structures by watching them move, then doing it yourself. Press M anytime to toggle sound.

A product of Amypo Technologies, Coimbatore.

Learn

  • All topics
  • Learning path
  • How it works
  • About dotprep

Practice

  • Practice arena
  • Daily challenge
  • Gamified challenges
  • Race mode
  • Leaderboard

Interview prep

  • Java interview
  • Coding questions
  • Company questions
  • System design
  • Jobs & careers

You

  • Dashboard
  • Settings
  • Pricing
  • Create account

© 2026 dotprep

Content last updated 20 November 2026

GamifiedGamified challengesPracticeRaceLeaderboardPricing
System designMessage queues and async processing

Message queues and async processing

Decoupling with queues and logs, delivery guarantees, idempotency, ordering and back-pressure.

intermediate 12 min read 5 interview questions

On this page

1/6
  1. 01Why queues
  2. 02Queues and logs
  3. 03Delivery guarantees
  4. 04Ordering and back-pressure
  5. 05Interview questions
  6. 06Quick check
Back to top

Why queues

A message queue lets one service hand off work without waiting for it. The producer writes a message and returns; consumers process it at their own pace. This gives you:

  • Faster responses: send the email, resize the image or update the search index after replying to the user.
  • Load levelling: a spike piles up in the queue instead of crushing the database.
  • Decoupling: producers don't need to know who consumes, and consumers can be down briefly without losing work.
  • Fan-out: one event (order placed) can drive many independent consumers (email, analytics, inventory).

Queues and logs

Message queueEvent log
ExamplesRabbitMQ, SQSKafka, Kinesis, Pulsar
After consumptionMessage is deletedMessage stays (retention period); consumers track their offset
Multiple consumersCompeting consumers share the workEach consumer group reads everything independently
ReplayNoYes: rewind the offset
OrderingUsually best effortOrdered within a partition

Delivery guarantees

  • At most once: may lose messages, never duplicates (acknowledge before processing).
  • At least once: never loses, may duplicate (acknowledge after processing). The usual choice.
  • Exactly once: hard end to end; in practice it's at-least-once delivery plus idempotent processing.

Idempotency means processing the same message twice has the same effect as once. Give each message a unique id and record processed ids (or use upserts, or conditional updates like "set status = PAID where status = PENDING").

java
void handle(PaymentEvent e) {
// insert fails if this event id was already processed
if (!processed.insertIfAbsent(e.id())) return;
ledger.credit(e.account(), e.amount());
}

Messages that fail repeatedly should go to a dead-letter queue after a few retries, so one bad message doesn't block the rest.

Ordering and back-pressure

Global ordering limits you to one consumer. Most systems need order only per entity, so they partition by a key (user id, order id): all events for one key land in one partition and are processed in order, while different keys run in parallel.

If consumers fall behind, the queue grows. Watch consumer lag, autoscale consumers, and decide what happens at the limit: block producers, shed low-priority work, or drop stale messages.

Uploading a photo with async processing

1/5
ClientUpload APIObject storageQueuephoto.uploadedThumbnail workerFeed workerNotify worker

The upload API saves the original file to object storage.

See it animated

QueueCircular queuePriority queue

Interview questions

Try answering out loud first, then open the model answer.

Q1When would you add a message queue?beginner

When work doesn't need to finish before replying (emails, thumbnails, analytics), when traffic is bursty and the downstream can't absorb peaks, or when several independent services need to react to the same event. It trades immediacy for resilience and throughput.

Q2Kafka vs RabbitMQ?intermediate

RabbitMQ is a broker that routes messages to queues and deletes them once acknowledged: good for task queues and flexible routing. Kafka is a partitioned, replicated append-only log that keeps messages for a retention period: very high throughput, replay, and many independent consumer groups. Use Kafka for event streams, RabbitMQ (or SQS) for work queues.

Q3How do you get exactly-once processing?advanced

Accept at-least-once delivery and make the consumer idempotent: deduplicate by message id, use upserts or conditional writes, or commit the consumer offset in the same transaction as the side effect (the transactional outbox/inbox patterns).

Q4How do you keep events for the same order in sequence?intermediate

Partition by order id so all its events go to the same partition, which one consumer reads in order. Different orders still process in parallel across partitions.

Q5What is the transactional outbox pattern?advanced

To update the database and publish an event atomically, write the event to an outbox table in the same transaction as the business change. A separate relay reads the outbox (or the DB's change log) and publishes to the broker, retrying until it succeeds. This avoids the dual-write problem where one succeeds and the other fails.

Quick check

Question 1

1.The usual practical delivery guarantee is…

Question 2

2.Which lets a new consumer replay last week's events?

Question 3

3.To keep per-user ordering with parallelism, partition by…

0 of 3 answered

CAP theorem and consistencyAPI design and rate limiting
Log inStart free