バッチ処理・圧縮・lingerの調整

Sending one record per network call is slow. Kafka producers batch records per partition before sending, amortizing overhead and dramatically raising throughput.

batch.size sets the maximum size (in bytes) of a single batch per partition.

• Larger batches = better throughput, more memory.
• A batch is sent when it fills or when linger expires.

linger.ms tells the producer to wait a short time for more records before sending, even if the batch isn't full.

• 0 (default) = send as soon as possible, lowest latency.
• A few ms = bigger batches, higher throughput.

Think of it as fill or wait:

• The batch ships when it reaches batch.size.
• Or when linger.ms elapses, whichever comes first.

Raising both increases batch sizes at the cost of latency.

compression.type compresses each batch before it goes over the wire and to disk.

Bigger batches compress better, so compression pairs naturally with linger tuning.

Common codecs and their trade-offs:

• lz4 — fast, good ratio, popular default.
• snappy — very fast, lower ratio.
• zstd — best ratio, slightly more CPU.
• gzip — high ratio, higher CPU.

Kafka stores batches compressed and serves them compressed; consumers decompress.

This means compression saves network, disk, and replication bandwidth — not just the producer link.

buffer.memory caps total memory for unsent records.

If batches accumulate faster than they ship, the producer blocks (up to max.block.ms) — a sign brokers can't keep up or batching is too aggressive.

A typical throughput-oriented configuration:

There is no free lunch:

• Lower linger.ms and smaller batches = lower latency.
• Higher values + compression = higher throughput, more latency.

Tune to your SLA, then measure.

Watch producer JMX metrics to confirm tuning works:

• batch-size-avg — are batches actually larger?
• compression-rate-avg — how well does data compress?
• record-queue-time-avg — added latency from linger.

Test your understanding of batching and linger.

You learned producer batching and compression tuning.

• batch.size + linger.ms control batch size and the latency trade-off.
• Compression (lz4/zstd) saves network, disk, and replication bandwidth.
• buffer.memory bounds in-flight data.
• Verify with batch-size-avg and compression-rate-avg metrics.