Konkurensi dan Pengelolaan Thread

When consuming messages from Kafka, processing them sequentially might not be fast enough. Concurrency allows your application to process multiple messages in parallel, significantly increasing throughput.

This is crucial for high-volume topics where messages arrive rapidly and need quick processing.

By default, a @KafkaListener method in Spring Boot will create one consumer thread per listener container. This single thread is responsible for fetching and processing messages from the partitions assigned to it.

While simple, this setup doesn't always leverage multi-core processors for parallel message processing from a single topic effectively.

Spring for Apache Kafka provides a powerful concurrency property that allows you to specify the number of consumer threads to run for a given listener.

• Each concurrent consumer runs in its own thread.
• These threads collectively manage the partitions assigned to the consumer group.

This is key to scaling your consumer horizontally within a single application instance.

You can set the concurrency level directly on the @KafkaListener annotation or globally via configuration properties.

For example, @KafkaListener(topics = "my-topic", groupId = "my-group", concurrency = "3") will start 3 consumer threads.

Alternatively, you can set it in application.properties for all listeners using a specific container factory.

Let's see how to configure a listener with concurrency. This listener will process messages from "my-topic" using 3 concurrent threads.

Spring Kafka uses ConcurrentKafkaListenerContainerFactory to create and manage the underlying consumer threads. When you set concurrency, this factory creates that many KafkaMessageListenerContainer instances.

Each container manages one consumer instance, which in turn is assigned a subset of topic partitions.

For more fine-grained control, you can customize the thread pool used by the container factory. This involves creating your own ConcurrentKafkaListenerContainerFactory bean.

You can set properties like max.poll.records and max.poll.interval.ms to optimize how many messages each consumer fetches and how often it commits offsets.

The effective concurrency for a consumer group is limited by the number of partitions in the topic. Kafka guarantees that messages within a single partition are processed in order.

• If you have 5 partitions and set concurrency = 10, only 5 threads will be active (one per partition).
• It's best practice to set concurrency to be less than or equal to the number of topic partitions.

While concurrency boosts throughput, keep these in mind:

• Order: Messages within a single partition are ordered, but overall order across partitions is NOT guaranteed.
• Resource Usage: More threads mean more memory and CPU. Monitor your application's resources.
• Rebalancing: High concurrency can lead to more frequent consumer rebalances if not managed well.

In this lesson, we explored how to manage consumer concurrency in Spring Boot Kafka applications. We learned about the concurrency property, how it relates to topic partitions, and key considerations for using it effectively.

By configuring concurrency, you can significantly optimize your application's throughput and resource utilization for high-volume event processing.