오프셋 수동 커밋
메시지 처리 보장 범위를 정밀하게 제어하고 데이터 손실이나 중복을 방지하도록 오프셋 커밋을 수동으로 제어합니다.
오프셋 수동 커밋은(는) CoddyKit의 무료 Advanced Spring Boot 4: Event-Driven Architecture (Kafka) 강의입니다. 이것은 4개 중 1번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 Advanced Spring Boot 4: Event-Driven Architecture (Kafka) 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. Advanced Spring Boot 4: Event-Driven Architecture (Kafka) 강의에는 총 4개의 강의가 포함되어 있습니다.
이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.
Why Manual Commits?
In Kafka, an offset marks the last message a consumer group has successfully processed from a topic partition. Committing an offset tells Kafka: "I've handled messages up to this point."
By default, Spring Kafka uses auto-commit, where offsets are committed periodically in the background. While convenient, this can sometimes lead to data loss or duplication if your application crashes mid-processing.
Manual offset committing gives you precise control, allowing you to decide exactly when an offset is marked as processed. This is crucial for ensuring message processing guarantees.
Auto-Commit: A Quick Look
With auto-commit, Kafka automatically commits offsets at a set interval (e.g., every 5 seconds). This means:
- Messages are processed.
- Offsets are committed later by Kafka.
If your application processes a message but crashes *before* Kafka's auto-commit interval passes, that message's offset might not be committed. When the application restarts, it will re-read and re-process that message, leading to potential duplicates (at-least-once processing).
Switching to Manual Mode
To take control of offset management, you need to disable auto-commit in your Spring Boot application's Kafka configuration. This is typically done by setting the AckMode.
The AckMode determines when a consumer acknowledges messages. For manual control, we'll use MANUAL_IMMEDIATE or MANUAL.
Here's how you might configure it in application.properties:
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.listener.ack-mode=MANUAL_IMMEDIATEThe Acknowledgment Object
When ack-mode is set to a manual option, your @KafkaListener method can receive an additional parameter: the Acknowledgment object.
This object is your direct interface to signal to Kafka that you have successfully processed a message (or a batch of messages) and its offset can now be committed.
You'll call its acknowledge() method when you're ready.
Basic Manual Commit Example
Let's see a simple example where we manually commit the offset after processing each message. Notice the Acknowledgment acknowledgment parameter.
Run this code, then stop and restart. You'll see messages are not re-processed if committed.
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
@SpringBootApplication
public class ManualCommitApp implements CommandLineRunner {
@Autowired
private KafkaTemplate<String, String> kafkaTemplate;
public static void main(String[] args) {
SpringApplication.run(ManualCommitApp.class, args);
}
@Override
public void run(String... args) throws Exception {
System.out.println("Sending message...");
kafkaTemplate.send("my-topic", "Hello CoddyKit!");
System.out.println("Message sent.");
}
@KafkaListener(topics = "my-topic", groupId = "manual-group")
public void listen(String message, Acknowledgment acknowledgment) {
System.out.println("Received: " + message);
// Simulate processing
try {
Thread.sleep(500);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
System.out.println("Processed: " + message + ", committing offset.");
acknowledgment.acknowledge(); // Manual commit
}
}Configuring for the Example
For the previous example to work, you'd need a src/main/resources/application.properties file with Kafka broker details and the manual ack mode:
spring.kafka.bootstrap-servers=localhost:9092(or your Kafka broker)spring.kafka.consumer.enable-auto-commit=falsespring.kafka.listener.ack-mode=MANUAL_IMMEDIATEspring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializerspring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializerspring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializerspring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
Remember to have a local Kafka running!
When to Acknowledge?
The core principle for manual committing is: commit only after your business logic has successfully completed.
- After each message: As shown in the previous example. Good for low throughput or critical messages.
- After a batch: Process multiple messages, then commit once for the entire batch. This is more efficient for high throughput.
- After external interactions: If you write data to a database, commit the offset *only after* the database transaction is successful.
Choosing the right strategy depends on your application's requirements for performance and data consistency.
Batch Processing & Manual Commit
When your listener consumes a batch of messages (e.g., List<String>), you should commit the offset only after *all* messages in that batch have been successfully processed. The Acknowledgment object still works for the entire batch.
This is often combined with AckMode.BATCH, though MANUAL_IMMEDIATE also works.
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import java.util.List;
@SpringBootApplication
public class BatchManualCommitApp implements CommandLineRunner {
@Autowired
private KafkaTemplate<String, String> kafkaTemplate;
public static void main(String[] args) {
SpringApplication.run(BatchManualCommitApp.class, args);
}
@Override
public void run(String... args) throws Exception {
System.out.println("Sending 3 messages...");
kafkaTemplate.send("my-batch-topic", "Batch Msg 1");
kafkaTemplate.send("my-batch-topic", "Batch Msg 2");
kafkaTemplate.send("my-batch-topic", "Batch Msg 3");
System.out.println("Messages sent.");
}
// Ensure spring.kafka.listener.ack-mode=MANUAL_IMMEDIATE in properties
@KafkaListener(topics = "my-batch-topic", groupId = "batch-manual-group")
public void listenBatch(List<String> messages, Acknowledgment acknowledgment) {
System.out.println("Received batch of " + messages.size() + " messages.");
for (String msg : messages) {
System.out.println(" Processing: " + msg);
// Simulate processing each message in the batch
try {
Thread.sleep(100);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
System.out.println("Finished processing batch. Committing offset.");
acknowledgment.acknowledge(); // Commit once for the entire batch
}
}Handling Errors & Reprocessing
What happens if an error occurs during message processing *before* acknowledgment.acknowledge() is called?
Since the offset was not committed, Kafka considers the message (or batch) as not processed. Upon restart or rebalance, the consumer will re-fetch and re-process those messages. This is the foundation of at-least-once processing semantics.
While this guarantees no data loss, it requires your message processing logic to be idempotent, meaning processing the same message multiple times has the same effect as processing it once.
Trade-offs & Best Practices
Manual offset committing offers control but comes with considerations:
- Overhead: Committing too frequently can add network and Kafka broker overhead.
- Reprocessing Scope: Committing too infrequently means more messages might be reprocessed if a failure occurs.
- Idempotency: Always design your consumers to be idempotent when using manual commits to handle potential duplicates gracefully.
- Error Handling: Combine manual commits with robust exception handling (e.g., retries, Dead Letter Topics) to manage failures effectively.
Quick Check: Manual Commits
You are using manual offset committing in your Spring Kafka consumer. If an error occurs while processing a message, and the acknowledgment.acknowledge() method is NOT called for that message, what will happen?
Recap: Manual Offset Control
We've explored manual offset committing, a powerful technique for precise control over message processing in Spring Kafka.
- It disables auto-commit, giving you control.
- You use the
Acknowledgmentobject to explicitly commit offsets. - Committing should happen only after successful business logic execution.
- It enables at-least-once processing, but requires idempotent consumers.
This fine-grained control is vital for building robust and reliable event-driven applications.
자주 묻는 질문
“오프셋 수동 커밋” 강의는 무료인가요?
네 — “오프셋 수동 커밋” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 Advanced Spring Boot 4: Event-Driven Architecture (Kafka) 강의 전체를 잠금 해제할 수 있습니다. Advanced Spring Boot 4: Event-Driven Architecture (Kafka) 강의에는 총 4개의 강의가 포함되어 있습니다.
“오프셋 수동 커밋”에서 뭘 배우나요?
메시지 처리 보장 범위를 정밀하게 제어하고 데이터 손실이나 중복을 방지하도록 오프셋 커밋을 수동으로 제어합니다. 브라우저에서 직접 실행하는 실습 코드로 Advanced Spring Boot 4: Event-Driven Architecture (Kafka)을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.
Advanced Spring Boot 4: Event-Driven Architecture (Kafka)을(를) 시작하는 데 경험이 필요한가요?
사전 경험은 필요하지 않습니다. CoddyKit의 Advanced Spring Boot 4: Event-Driven Architecture (Kafka)은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 1번째 강의입니다.
“오프셋 수동 커밋” 강의는 얼마나 걸리나요?
대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.
이 Advanced Spring Boot 4: Event-Driven Architecture (Kafka) 강의에서 코드를 작성하고 실행할 수 있나요?
네. 모든 Advanced Spring Boot 4: Event-Driven Architecture (Kafka) 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.