Apache Kafka & Stream Processing Fundamentals · 课时

理解分区和偏移量

掌握分区对可扩展性和并行处理的重要性，以及偏移量如何跟踪消费者进度

理解分区和偏移量 是 CoddyKit 上的免费 Apache Kafka & Stream Processing Fundamentals 课时。这是第 3 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Apache Kafka & Stream Processing Fundamentals 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Apache Kafka & Stream Processing Fundamentals 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

What are Kafka Partitions?

Imagine a Kafka topic as a category for messages. To handle lots of messages efficiently, Kafka divides a topic into smaller, ordered segments called partitions.

Think of each partition as its own mini-log. Messages are appended to the end of a partition in the order they arrive. Once written, messages in a partition are immutable.

Partitions: Ordered & Immutable

It's crucial to understand that while messages within a single partition are strictly ordered, there's no guaranteed order across different partitions of the same topic.

Ordered: Messages in one partition always have a clear sequence.
Immutable: Once a message is written to a partition, it cannot be changed.
Append-only: New messages are always added to the end.

Scalability Through Partitions

Partitions are the backbone of Kafka's scalability and parallelism. Here's why they matter:

Parallel Processing: Multiple consumers can read from different partitions of the same topic simultaneously.
Distributed Storage: Partitions can be spread across different Kafka brokers (servers) in a cluster. This allows topics to handle more data than a single server could.

How Messages Are Assigned

When a producer sends a message, Kafka needs to decide which partition it should go into. This is called partitioning strategy:

With a Key: If a message includes a key (e.g., a user ID), Kafka uses a hash of that key to consistently assign it to the same partition. This ensures all messages for a specific key are processed in order.
Without a Key: If no key is provided, Kafka typically uses a round-robin approach, distributing messages evenly across all partitions.

Producer with Message Keys

This Java example shows how a producer sends messages to a topic, explicitly providing a key. Messages with the same key will end up in the same partition.

import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;

public class KeyedProducer {
  public static void main(String[] args) {
    Properties props = new Properties();
    props.put("bootstrap.servers", "localhost:9092");
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

    try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
      String topic = "my_keyed_topic";
      for (int i = 0; i < 4; i++) {
        String key = "user-" + (i % 2); // user-0, user-1, user-0, user-1
        String value = "Message " + i + " for " + key;
        producer.send(new ProducerRecord<>(topic, key, value));
        System.out.println("Sent: Key=" + key + ", Value=" + value);
      }
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

Introducing Message Offsets

Every message within a Kafka partition has a unique, sequential identifier called an offset. Think of it as an index number for messages within that specific partition.

The first message in a partition has offset 0.
The next message has offset 1, and so on.
Offsets are local to each partition.

Offsets for Consumer Progress

Offsets are critical for consumers to track their progress. A consumer keeps a record of the offset of the last message it successfully processed in each partition.

This allows consumers to:

Resume processing exactly where they left off if they stop or crash.
Know which messages they still need to read.

Committing Offsets

After processing messages, consumers need to inform Kafka about their progress by committing their offsets. This means saving the current offset to a special Kafka topic (__consumer_offsets).

Committing can be:

Automatic: Kafka commits offsets periodically in the background.
Manual: The application explicitly tells Kafka when to commit offsets, offering more control over processing guarantees.

Check Your Understanding

Let's test your knowledge about Kafka partitions and offsets.

Recap: Partitions & Offsets

Today, we explored two core Kafka concepts:

Partitions: These segments divide a topic, enabling parallel processing, distributed storage, and ordered messages within each partition.
Offsets: These sequential IDs track the position of messages within a partition, allowing consumers to precisely manage their progress and resume reliably.

Understanding these concepts is key to building scalable and robust Kafka applications!

常见问题解答

「理解分区和偏移量」课时是免费的吗？

是的 — 「理解分区和偏移量」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Apache Kafka & Stream Processing Fundamentals 课程的其余内容，请升级到 CoddyKit PRO。 Apache Kafka & Stream Processing Fundamentals 课程共包含 4 节课。

「理解分区和偏移量」这节课中我会学到什么？

掌握分区对可扩展性和并行处理的重要性，以及偏移量如何跟踪消费者进度你通过在浏览器中直接运行的动手代码来练习 Apache Kafka & Stream Processing Fundamentals，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Apache Kafka & Stream Processing Fundamentals 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Apache Kafka & Stream Processing Fundamentals 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 3 节课，共 4 节。

「理解分区和偏移量」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Apache Kafka & Stream Processing Fundamentals 课中编写并运行代码吗？

能。每节 Apache Kafka & Stream Processing Fundamentals 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Apache Kafka & Stream Processing Fundamentals