Work Queues, Prefetch, and Competing Consumers
Distribute load across consumers fairly with prefetch limits and the competing-consumers pattern.
Why Work Queues?
A work queue (sometimes called a task queue) lets your Node.js backend push time-consuming jobs to a queue instead of running them inside the HTTP request. Think image resizing, sending emails, or generating PDFs.
- The producer publishes a small message describing the job.
- One or more consumers (worker processes) pull jobs and execute them in the background.
- The HTTP response returns immediately, keeping your API fast.
RabbitMQ stores the messages durably until a worker is free to process them.
A Minimal Producer
The producer connects to RabbitMQ, declares a durable queue, and sends a task message. Each message is just a buffer of bytes — here we send a plain string describing the job.
Notice { durable: true } on the queue and { persistent: true } on the message. Together they let tasks survive a broker restart.
const amqp = require('amqplib');
async function send(task) {
const conn = await amqp.connect('amqp://localhost');
const ch = await conn.createChannel();
const queue = 'tasks';
await ch.assertQueue(queue, { durable: true });
ch.sendToQueue(queue, Buffer.from(task), { persistent: true });
console.log('Sent: %s', task);
await ch.close();
await conn.close();
}
send('resize-image:42');All lessons in this course
- Producers, Consumers, and the AMQP Model
- Exchange Types: Direct, Topic, Fanout, and Headers
- Acknowledgements, Dead-Letter Queues, and Retries
- Work Queues, Prefetch, and Competing Consumers