Introduction to Consumer Groups

Redis Streams are powerful for logging and event sourcing. But what if multiple applications or instances need to process the same stream data, without duplicating work?

This is where Consumer Groups come in. They are a core feature of Redis Streams designed for reliable, shared message processing.

Imagine two services, Service A and Service B, both needing to process every new order from an orders_stream.

If both services simply use XREAD, they would both receive and process all messages, leading to duplicated effort or inconsistent states.

• How do they know which messages are already processed?
• How do they share the workload?

Consumer Groups provide a way for multiple clients (consumers) to jointly read from the same Redis Stream.

They ensure that:

• Each message is delivered to only one consumer within the group.
• The state (which messages are pending, which are acknowledged) is managed by Redis.
• Consumers can recover and resume processing from where they left off.

Let's define the core parts of a Consumer Group:

• Stream: The Redis Stream holding the messages.
• Consumer Group: A logical group of consumers processing a stream.
• Consumer: An individual client instance within a group.
• Pending Entry List (PEL): A list maintained by Redis for each consumer, tracking messages that have been delivered but not yet acknowledged.

Before consumers can join, a Consumer Group must be created on a specific stream. We use the XGROUP CREATE command.

The MKSTREAM option automatically creates the stream if it doesn't exist.

Let's add a few messages to our my_stream so we have something to read. Remember, XADD is how you add entries to a stream.

Consumers read from a group using the XREADGROUP command. You specify the group name, the consumer name, and the stream ID to start reading from.

The special ID > means "read only new, unread messages that have not been delivered to any other consumer in this group yet."

After a consumer processes a message, it must acknowledge it using XACK. This tells Redis the message is handled and can be removed from the consumer's Pending Entry List (PEL).

This is crucial for reliability and prevents messages from being re-delivered if a consumer crashes.

Acknowledging messages is vital for two main reasons:

• Reliability: If a consumer fails before acknowledging, the message remains in its PEL and can be claimed by another consumer later.
• Resource Management: Unacknowledged messages stay in the PEL, consuming memory. Acknowledging keeps the PEL clean and efficient.

Which Redis command is used to create a new Consumer Group for a stream?

In this lesson, you learned about Redis Consumer Groups and how they enable multiple consumers to reliably process messages from a single stream, ensuring each message is handled only once.

You now understand the roles of groups, consumers, and the Pending Entry List (PEL), and how to use XGROUP CREATE, XREADGROUP, and XACK.

Next, we'll dive deeper into implementing consumer group logic and handling pending messages!