Batching, Compression & Linger Tuning
Learn how producer batching, linger.ms, and compression interact to trade latency for throughput and reduce network and disk usage.
Batching, Compression & Linger Tuning is a free Apache Kafka & Stream Processing Fundamentals lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Apache Kafka & Stream Processing Fundamentals learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Throughput Lever
Sending one record per network call is slow. Kafka producers batch records per partition before sending, amortizing overhead and dramatically raising throughput.
batch.size
batch.size sets the maximum size (in bytes) of a single batch per partition.
- Larger batches = better throughput, more memory.
- A batch is sent when it fills or when linger expires.
// 64 KB batches
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 65536);linger.ms
linger.ms tells the producer to wait a short time for more records before sending, even if the batch isn't full.
0(default) = send as soon as possible, lowest latency.- A few ms = bigger batches, higher throughput.
props.put(ProducerConfig.LINGER_MS_CONFIG, 10);How They Interact
Think of it as fill or wait:
- The batch ships when it reaches
batch.size. - Or when
linger.mselapses, whichever comes first.
Raising both increases batch sizes at the cost of latency.
Compression Basics
compression.type compresses each batch before it goes over the wire and to disk.
Bigger batches compress better, so compression pairs naturally with linger tuning.
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");Choosing a Codec
Common codecs and their trade-offs:
- lz4 — fast, good ratio, popular default.
- snappy — very fast, lower ratio.
- zstd — best ratio, slightly more CPU.
- gzip — high ratio, higher CPU.
End-to-End Compression
Kafka stores batches compressed and serves them compressed; consumers decompress.
This means compression saves network, disk, and replication bandwidth — not just the producer link.
buffer.memory
buffer.memory caps total memory for unsent records.
If batches accumulate faster than they ship, the producer blocks (up to max.block.ms) — a sign brokers can't keep up or batching is too aggressive.
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 33554432);A High-Throughput Profile
A typical throughput-oriented configuration:
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 131072);
props.put(ProducerConfig.LINGER_MS_CONFIG, 20);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
props.put(ProducerConfig.ACKS_CONFIG, "all");Latency vs. Throughput
There is no free lunch:
- Lower
linger.msand smaller batches = lower latency. - Higher values + compression = higher throughput, more latency.
Tune to your SLA, then measure.
Measuring the Effect
Watch producer JMX metrics to confirm tuning works:
batch-size-avg— are batches actually larger?compression-rate-avg— how well does data compress?record-queue-time-avg— added latency from linger.
Quick Check
Test your understanding of batching and linger.
Recap
You learned producer batching and compression tuning.
batch.size+linger.mscontrol batch size and the latency trade-off.- Compression (lz4/zstd) saves network, disk, and replication bandwidth.
buffer.memorybounds in-flight data.- Verify with batch-size-avg and compression-rate-avg metrics.
Frequently asked questions
Is the “Batching, Compression & Linger Tuning” lesson free?
Yes — the full text of “Batching, Compression & Linger Tuning” is free to read here on the web, and the Apache Kafka & Stream Processing Fundamentals course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Apache Kafka & Stream Processing Fundamentals course, upgrade to CoddyKit PRO.
What will I learn in “Batching, Compression & Linger Tuning”?
Learn how producer batching, linger.ms, and compression interact to trade latency for throughput and reduce network and disk usage. You practise Apache Kafka & Stream Processing Fundamentals with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Apache Kafka & Stream Processing Fundamentals?
No prior experience is required. Apache Kafka & Stream Processing Fundamentals on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Batching, Compression & Linger Tuning” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Apache Kafka & Stream Processing Fundamentals lesson?
Yes. Every Apache Kafka & Stream Processing Fundamentals lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Producer & Consumer Performance
- Broker Configuration & Tuning
- Disk I/O & Network Optimization
- Batching, Compression & Linger Tuning