キャパシティプランニング：パーティションとレプリケーション

Partition count and replication factor are hard to change after launch and directly shape throughput, parallelism, and durability. Good capacity planning avoids painful migrations later.

A consumer group can have at most as many active consumers as there are partitions. More partitions allow more parallel consumers and higher throughput.

A common heuristic: divide target throughput by per-partition throughput.

• Measure max producer throughput per partition (p).
• Measure max consumer throughput per partition (c).
• partitions = max(target/p, target/c).

More partitions are not free:

• More open file handles and memory on brokers.
• Longer leader-election and rebalance times.
• Higher end-to-end latency.

Plan headroom, but do not wildly over-provision.

The replication factor is how many copies of each partition exist across brokers. A factor of 3 is standard for production durability.

With min.insync.replicas=2 and acks=all, a write succeeds only when at least two replicas confirm. This tolerates one broker failure without data loss.

Specify partitions and replication when creating a topic so it starts correctly sized.

You can increase partitions later but never decrease them, and adding partitions breaks key-based ordering. Pick a count with growth headroom from the start.

Disk need = throughput x retention x replication factor. A topic at 10 MB/s with 7-day retention and RF 3 needs roughly 17 TB. Plan storage accordingly.

Ensure partitions and replicas are evenly distributed so no broker becomes a hotspot. Rack-awareness places replicas across failure domains.

Capacity planning balances parallelism, durability, latency, and cost. Size partitions from throughput, use RF 3 with min ISR 2, and plan disk from retention.

Test your understanding of capacity planning.

You learned capacity planning.

• Partition count drives consumer parallelism and throughput.
• Too many partitions add latency and broker overhead.
• RF 3 with min ISR 2 tolerates one broker failure.
• Plan disk from throughput x retention x replication.