실시간 데이터 피드를 위한 변경 데이터 캡처

Change Data Capture (CDC) watches a database for inserts, updates, and deletes and emits each change as an event.

Instead of polling tables, downstream systems subscribe to the change stream and react in real time.

Polling a table every few seconds is wasteful and laggy. It misses fast intermediate states and hammers the database.

CDC captures every committed change exactly once, with low latency and no extra query load.

Most CDC tools tail the database's write-ahead log (WAL in Postgres, binlog in MySQL). The log already records every committed change, so reading it is cheap and accurate.

A CDC event typically carries the operation type, the table, and the before/after row state.

CDC events are usually published to a stream like Kafka or Redpanda. Each table often maps to its own topic, letting many consumers fan out from one source of truth.

A consumer reads CDC events and pushes updates to clients over WebSocket or SSE. The dashboard reflects database changes within milliseconds, without the UI polling at all.

CDC is ideal for cache invalidation. When a row changes, the event tells your cache exactly what to refresh or evict, so stale data never lingers.

CDC pipelines usually guarantee at-least-once delivery, so duplicates can occur on retry. Make consumers idempotent, using the primary key plus a log offset to dedupe.

When a new consumer starts, it needs the current state, not just future changes. CDC tools take an initial snapshot of existing rows, then switch to streaming the log.

• Debezium for Postgres, MySQL, MongoDB into Kafka.
• Postgres logical replication with custom consumers.
• Managed options in many cloud data platforms.

When a table's schema changes, downstream consumers must adapt. Use a schema registry and versioned events so a new column does not break existing readers.

Test your understanding of CDC.

CDC turns database mutations into a real-time event stream by reading the transaction log. Pair it with a message queue to power live dashboards, fresh caches, and reactive services. Make consumers idempotent and handle snapshots plus schema evolution.