The Commit Log: How Kafka Stores Data

At its core Kafka is a distributed, append-only commit log. Producers append to the end and data is never modified in place — that's the secret to its speed and durability.

Every record is written sequentially to the end of a partition's log. Sequential disk writes are far faster than random ones, sustaining high throughput on cheap disks.

Each record gets a monotonically increasing offset — its position in the partition. Offsets are unique within a partition and never reused.

Unlike a queue, consuming a record does not delete it. Many consumers read the same log independently, each tracking its own offset.

A partition log is split into segment files on disk. Only the active segment is written to; older segments are immutable and can be deleted or compacted.

Kafka keeps data for a configurable period regardless of whether it's consumed. The default retention is 7 days.

You can also cap a partition by size. Whichever limit hits first — time or bytes — triggers deletion of the oldest segments.

Instead of deleting by age, compaction keeps only the latest record per key — perfect for changelog topics where you just want each key's current value.

Kafka serves reads straight from the OS page cache to the network using zero-copy, skipping extra memory copies. Another big reason consumption is so fast.

Records are durable because they're written to disk and replicated to other brokers. Even if the OS cache is lost, replicas preserve the data.

The commit log explains Kafka's character: sequential writes for speed, offsets for replayable reads, retention and compaction for storage, replication for durability.

Test your understanding of the commit log.

You learned Kafka's storage: an append-only commit log with stable offsets, reads that don't delete, retention plus compaction, and replication for durability.