Apache Kafka & Stream Processing Fundamentals · Lesson

Designing for High Throughput

Learn architectural considerations and best practices for building Kafka-based systems that handle massive data volumes.

Designing for High Throughput is a free Apache Kafka & Stream Processing Fundamentals lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Apache Kafka & Stream Processing Fundamentals learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What is High Throughput?

In Kafka, high throughput means your system can efficiently process a massive volume of data per second or minute. It's crucial for applications like real-time analytics, IoT data ingestion, and log aggregation where data arrives continuously at high rates.

Designing for high throughput ensures your Kafka cluster and applications can handle peak loads without performance degradation, data loss, or significant delays.

Key Factors for Throughput

Achieving high throughput in Kafka involves optimizing several interconnected components. Think of it as a chain – the weakest link limits the overall speed.

Producers: How efficiently they send data.
Brokers: How quickly they store and serve data.
Consumers: How fast they read and process data.
Infrastructure: Network bandwidth and disk I/O speed.

Producer Optimization: Batching

Sending messages one by one is inefficient. Producers can batch messages, sending multiple records in a single request. This reduces network overhead and improves throughput.

batch.size: The maximum size in bytes of a single batch.
linger.ms: The maximum time a producer waits before sending a batch, even if it's not full.

Adjusting these balances latency (how quickly a message is sent) and throughput (how many messages are sent over time).

Batching Producer Example

Here's how to configure a producer for batching. Notice the batch.size and linger.ms settings.

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;

public class HighThroughputProducer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        // High throughput settings
        props.put("batch.size", 65536); // 64 KB batch size
        props.put("linger.ms", 10);    // Wait up to 10 ms for more records
        props.put("compression.type", "snappy"); // Compress batches
        props.put("acks", "1");         // Acks=1 for good balance

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            for (int i = 0; i < 1000; i++) {
                String message = "Hello Kafka Throughput - " + i;
                producer.send(new ProducerRecord<>("throughput-topic", "key-" + i, message));
            }
            System.out.println("1000 messages sent to throughput-topic.");
        }
    }
}

Producer Compression & ACKs

Beyond batching, two other producer settings greatly influence throughput:

compression.type: Compressing data (e.g., gzip, snappy, lz4) reduces network bandwidth usage and disk space. This allows more data to be sent and stored, boosting effective throughput.
acks: Controls the durability guarantee. acks=0 (fire-and-forget) offers the highest throughput but lowest durability. acks=1 (leader acknowledges) is a good balance. acks=all (all in-sync replicas acknowledge) provides the highest durability but lowest throughput.

Broker Scaling: Partitions & Disks

Kafka brokers are the backbone. Their throughput depends heavily on:

Partitions: Each topic partition is an ordered log. More partitions allow for greater parallelism in both writing and reading data across brokers and consumers. Distribute partitions evenly across brokers.
Disk I/O: Kafka is disk-intensive. Using fast SSDs and configuring RAID 0 or RAID 10 for data directories significantly improves write and read speeds, which is critical for high throughput.

Broker Network & CPU

Don't overlook the underlying hardware for your brokers:

Network: High-speed network interfaces (e.g., 10 Gigabit Ethernet) and sufficient bandwidth are paramount. Brokers constantly move data between themselves (replication) and with clients.
CPU: While Kafka is optimized for sequential disk I/O, CPU can become a bottleneck, especially with heavy data compression/decompression, SSL encryption, or complex ACLs. Ensure adequate CPU cores.

Consumer Optimization: Batch Fetching

Consumers also benefit from batching. Instead of polling for one record at a time, consumers fetch a batch of records.

max.poll.records: The maximum number of records returned in a single call to poll(). Processing more records per poll reduces the overhead of repeated calls.
fetch.min.bytes: The minimum amount of data (in bytes) that the consumer will wait to receive from a broker before returning.
fetch.max.wait.ms: The maximum amount of time (in ms) the consumer will wait for fetch.min.bytes to be satisfied.

Batching Consumer Example

A consumer configured for batch fetching will retrieve more records per poll() call, improving processing efficiency.

import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class HighThroughputConsumer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("group.id", "throughput-group");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        // High throughput settings
        props.put("max.poll.records", 500); // Fetch up to 500 records at once
        props.put("fetch.min.bytes", 1048576); // Wait for 1MB of data
        props.put("fetch.max.wait.ms", 500); // Or wait 500ms

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(Collections.singletonList("throughput-topic"));

            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                if (!records.isEmpty()) {
                    System.out.println("Received " + records.count() + " records.");
                    // Process records (e.g., in a thread pool for parallelism)
                    records.forEach(record -> {
                        // System.out.println("Processing record: " + record.value());
                    });
                    consumer.commitSync();
                }
            }
        }
    }
}

Consumer Parallel Processing

While max.poll.records helps with fetching, the actual processing of messages can be a bottleneck. To maximize consumer throughput:

Internal Thread Pool: Implement a thread pool within your consumer application. When poll() returns a batch of records, submit these records to the thread pool for parallel processing.
Careful Offset Management: If processing in parallel, ensure you commit offsets only after all messages in a batch (or a specific subset) have been successfully processed. Otherwise, you risk data loss or reprocessing.

Monitoring Throughput Metrics

To identify throughput bottlenecks, continuous monitoring is essential. Key metrics to watch:

Producer: Request rate, byte rate, request latency.
Consumer: Fetch rate, byte rate, consumer lag (most critical for identifying processing bottlenecks).
Broker: Disk I/O (read/write), network I/O, CPU utilization, memory usage.
Network: Bandwidth utilization, packet loss.

Tools like JMX, Prometheus/Grafana, or Confluent Control Center can help visualize these metrics.

Throughput Optimization Challenge

You've learned about various strategies to boost Kafka system throughput. Which of the following actions would generally help increase the overall throughput of a Kafka-based data pipeline?

Designing for Throughput Recap

Congratulations! You've explored key strategies for designing Kafka-based systems to handle massive data volumes with high throughput.

Optimize producers with batching, compression, and appropriate `acks` settings.
Scale brokers by using sufficient partitions, fast disks, and ample network/CPU resources.
Tune consumers with batch fetching and implement internal parallelism for processing.
Monitor critical metrics to identify and address bottlenecks proactively.

Mastering these techniques is vital for building robust and performant real-time data platforms.

Frequently asked questions

Is the “Designing for High Throughput” lesson free?

Yes — the full text of “Designing for High Throughput” is free to read here on the web, and the Apache Kafka & Stream Processing Fundamentals course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Apache Kafka & Stream Processing Fundamentals course, upgrade to CoddyKit PRO.

What will I learn in “Designing for High Throughput”?

Learn architectural considerations and best practices for building Kafka-based systems that handle massive data volumes. You practise Apache Kafka & Stream Processing Fundamentals with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Apache Kafka & Stream Processing Fundamentals?

No prior experience is required. Apache Kafka & Stream Processing Fundamentals on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Designing for High Throughput” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Apache Kafka & Stream Processing Fundamentals lesson?

Yes. Every Apache Kafka & Stream Processing Fundamentals lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to Apache Kafka & Stream Processing Fundamentals