Consumer Groups and Rebalancing

A consumer group lets multiple consumer instances share the work of reading a topic. Kafka assigns each partition to exactly one consumer in the group.

Consumers join a group by sharing a group.id. Kafka tracks the offset position per group, so each group reads the topic independently.

The number of partitions sets the maximum parallelism. With four partitions, at most four consumers in a group can work at once.

If a group has more consumers than partitions, the extras sit idle. Scale partitions to scale consumption beyond the current limit.

A rebalance reassigns partitions when group membership changes.

• A consumer joins or leaves
• A consumer is considered dead
• Partitions are added

During a rebalance, consumers stop processing briefly while partitions are reassigned. Frequent rebalances hurt throughput.

Consumers send heartbeats. Missing them past session.timeout.ms marks a consumer dead and triggers a rebalance.

If processing between polls exceeds max.poll.interval.ms, the consumer is evicted. Keep per-message work bounded or increase the limit.

Offsets record progress per partition per group. Spring Kafka can auto-commit or commit manually after successful processing for at-least-once delivery.

The cooperative rebalance protocol reassigns only the partitions that must move, avoiding a full stop-the-world reassignment.

A @KafkaListener joins the configured group automatically; running multiple app instances scales the group with no code change.

Reason about parallelism.

You learned that a consumer group shares a topic by assigning each partition to one consumer, so partition count caps parallelism. Rebalances reassign partitions when membership changes and briefly pause processing; heartbeats and poll intervals govern liveness. Cooperative rebalancing and manual offset commits keep Spring Kafka consumers efficient and reliable.