Message Queues and Asynchronous Processing

In synchronous processing the caller waits for a result. In asynchronous processing the work is handed off and the caller continues immediately.

A message queue is a buffer that holds tasks (messages) between a producer that creates work and a consumer that processes it later.

A producer puts messages onto the queue; one or more consumers pull them off and do the work. They run independently.

Queues decouple components so a slow or down consumer does not block the producer. Work waits safely in the queue until it can be handled.

When a user uploads a video, the server enqueues a transcoding job and responds instantly. A worker processes the video in the background.

Queues absorb bursts. If 10,000 requests arrive at once, they line up in the queue and consumers drain them at a sustainable rate.

Popular tools include RabbitMQ, Apache Kafka, and Amazon SQS. Kafka is also used for high-throughput event streaming.

Many queues guarantee a message is delivered at least once, so consumers must be idempotent to safely handle duplicates.

A consumer acknowledges a message after processing. If it crashes first, the queue re-delivers the message so no work is lost.

Messages that repeatedly fail are moved to a dead letter queue for later inspection, instead of clogging the main queue forever.

In point-to-point, each message goes to one consumer. In publish/subscribe, a message is broadcast to many subscribers at once.

Test your understanding of message queues.

You learned about message queues:

• They decouple producers from consumers
• Enable async processing and smooth load spikes
• At-least-once delivery needs idempotent consumers
• Acknowledgements, dead letter queues, and pub/sub patterns add reliability