Apache Kafka & Stream Processing Fundamentals · 课时

变更数据捕获（CDC）

利用 Kafka 实现变更数据捕获，实时传输数据库变更，满足各种应用场景的需求

变更数据捕获（CDC） 是 CoddyKit 上的免费 Apache Kafka & Stream Processing Fundamentals 课时。这是第 2 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Apache Kafka & Stream Processing Fundamentals 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Apache Kafka & Stream Processing Fundamentals 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

What is Change Data Capture?

Imagine needing to know every time a record in your database is updated, inserted, or deleted, in real-time. This is where Change Data Capture (CDC) comes in!

CDC is a software design pattern used to track and capture changes made to data in a database. It focuses on identifying and capturing only the data that has changed, rather than performing full scans.

Real-Time Data with CDC & Kafka

Combining CDC with Kafka unlocks powerful capabilities for real-time data processing and integration:

Real-time Analytics: Update dashboards and reports instantly.
Data Synchronization: Keep multiple databases or data stores consistent.
Event Sourcing: Reconstruct the full history of changes for auditing or debugging.
Microservices: Enable services to react to changes in other services' data without direct database access.

Log-Based CDC Explained

The most common and efficient CDC method, especially with Kafka, is log-based CDC. Databases like PostgreSQL, MySQL, and SQL Server maintain a transaction log (or write-ahead log - WAL).

This log records every change made to the database. Log-based CDC tools read these logs directly, without impacting the database's performance, to extract changes.

CDC Flow to Kafka

A typical CDC architecture with Kafka involves:

Source Database: The database where changes originate.
CDC Connector/Tool: Reads the database's transaction log.
Kafka Connect: A framework for connecting Kafka with other systems.
Kafka Topic: Where the captured change events are published.
Consumers: Applications that read and process the change events from Kafka.

Debezium: Open-Source CDC

Debezium is a popular open-source distributed platform for Change Data Capture. It provides a set of Kafka Connect connectors that monitor specific database systems.

When changes occur in your database, Debezium streams these changes as events to Kafka topics. It supports various databases like PostgreSQL, MySQL, MongoDB, and SQL Server.

Debezium PostgreSQL Connector

To set up Debezium, you'd typically configure a connector via Kafka Connect's REST API. Here's a simplified example of a Debezium PostgreSQL connector configuration:

{
  "name": "pg-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "localhost",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "password",
    "database.dbname": "mydb",
    "topic.prefix": "dbserver1",
    "table.include.list": "public.customers"
  }
}

This config tells Debezium to monitor the mydb database on localhost:5432 and send changes from the public.customers table to Kafka topics prefixed with dbserver1.

What a CDC Event Looks Like

When Debezium captures a change, it publishes an event to Kafka. This event usually contains structured information:

before: The state of the record before the change (for updates/deletes).
after: The state of the record after the change (for inserts/updates).
op: The operation type (c for create, u for update, d for delete, r for read/snapshot).
source: Metadata about the database, table, and transaction.

These events are often serialized as JSON or Avro.

Processing Change Events

A Kafka consumer application can read these CDC events and react to them. For instance, you might update a search index, invalidate a cache, or trigger another microservice.

Here's a basic Java consumer example illustrating how you might parse a Debezium event:

public class CdcConsumer {
  public static void main(String[] args) {
    // This is a simplified example.
    // In reality, use KafkaConsumer and JSON/Avro parsing.

    String jsonEvent = "{ \"payload\": { \"op\": \"c\", \"after\": { \"id\": 1, \"name\": \"Alice\" } } }";
    
    // Imagine parsing jsonEvent here
    String operationType = getOperationFromJson(jsonEvent, "op");
    String newName = getOperationFromJson(jsonEvent, "name");

    if ("c".equals(operationType)) {
      System.out.println("New customer created: " + newName);
    } else if ("u".equals(operationType)) {
      System.out.println("Customer updated: " + newName);
    }
  }

  // Placeholder for JSON parsing logic
  private static String getOperationFromJson(String json, String key) {
    if (key.equals("op")) return "c"; // Simulate 'c' operation
    if (key.equals("name")) return "Alice"; // Simulate 'name'
    return null;
  }
}

Practical CDC Use Cases

CDC with Kafka is incredibly versatile. Some common use cases include:

Data Warehousing: Populate data warehouses with only changed data for efficient ETL (Extract, Transform, Load).
Cache Invalidation: Automatically clear or update caches when underlying data changes.
Audit Logs: Maintain a complete, immutable history of all data changes for compliance.
Search Indexing: Keep search indexes (e.g., Elasticsearch) up-to-date with real-time database changes.

CDC Knowledge Check

Which of the following is a primary benefit of using log-based Change Data Capture (CDC) with Kafka?

CDC with Kafka: A Powerful Pattern

We've explored how Change Data Capture (CDC) is a powerful pattern for streaming database changes in real-time to Kafka. Using tools like Debezium, you can efficiently capture inserts, updates, and deletes from your databases.

This enables a wide array of real-time use cases, from data synchronization and analytics to event sourcing and microservice communication. Understanding CDC is key to building responsive, data-driven applications with Kafka.

常见问题解答

「变更数据捕获（CDC）」课时是免费的吗？

是的 — 「变更数据捕获（CDC）」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Apache Kafka & Stream Processing Fundamentals 课程的其余内容，请升级到 CoddyKit PRO。 Apache Kafka & Stream Processing Fundamentals 课程共包含 4 节课。

「变更数据捕获（CDC）」这节课中我会学到什么？

利用 Kafka 实现变更数据捕获，实时传输数据库变更，满足各种应用场景的需求你通过在浏览器中直接运行的动手代码来练习 Apache Kafka & Stream Processing Fundamentals，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Apache Kafka & Stream Processing Fundamentals 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Apache Kafka & Stream Processing Fundamentals 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 2 节课，共 4 节。

「变更数据捕获（CDC）」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Apache Kafka & Stream Processing Fundamentals 课中编写并运行代码吗？

能。每节 Apache Kafka & Stream Processing Fundamentals 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Apache Kafka & Stream Processing Fundamentals