大規模環境におけるバックプレッシャーとフロー制御

Backpressure is feedback that slows a fast producer when a downstream stage can't keep up.

Without it, queues grow unbounded, memory fills, and the whole pipeline collapses under load.

Kafka itself acts as a giant durable buffer between producers and consumers.

• Producers write at their pace; consumers read at theirs.
• The log absorbs bursts up to the retention limit.

But the buffer is finite — you still need flow control.

Kafka consumers pull via poll(). They fetch only what they can handle, giving natural backpressure on the read side.

If processing is slow, you simply poll less — but you must respect timeouts.

If a poll returns more records than you can process within max.poll.interval.ms, the broker assumes the consumer is dead and triggers a rebalance.

When a downstream sink is overwhelmed, pause consumption on specific partitions and resume when capacity returns.

On the write side, buffer.memory plus max.block.ms create backpressure: when the buffer is full, send() blocks instead of dropping data, slowing the producer to match broker throughput.

If you hand records to worker threads, use bounded queues.

An unbounded queue hides backpressure until you run out of memory; a bounded one blocks the poll loop, propagating pressure back to Kafka.

Backpressure buys time, but sustained overload needs more capacity:

• Add consumers up to the partition count.
• Increase partitions for more parallelism.
• Pre-provision for known peak load.

Sometimes correctness allows dropping or downsampling under extreme load — load shedding.

Route non-critical events to a lower-priority topic or sample metrics. Only do this where lost data is acceptable.

Signals that backpressure is active or failing:

• Rising consumer lag.
• Growing producer record-queue-time.
• Frequent rebalances.
• Memory pressure on workers.

Build for stability:

• Always bound your buffers and queues.
• Let slow stages slow upstream stages, not crash them.
• Combine backpressure (short-term) with scaling (long-term).
• Shed load only where acceptable.

Test your understanding of backpressure.

You learned backpressure and flow control at scale.

• Kafka's durable log and pull-based consumers provide natural buffering.
• Pause/resume, bounded queues, and buffer.memory propagate pressure safely.
• Scale out for sustained load; shed load only when acceptable.
• Monitor lag, queue time, and rebalances.