Apache Kafka & Stream Processing Fundamentals · Leçon

Concevoir pour un débit élevé

Apprenez les considérations architecturales et les bonnes pratiques pour créer des systèmes fondés sur Kafka capables de gérer d’importants volumes de données.

Concevoir pour un débit élevé est une leçon Apache Kafka & Stream Processing Fundamentals gratuite sur CoddyKit. Ceci est la leçon 1 sur 4. Tu peux lire la leçon complète ci-dessous gratuitement — puis la pratiquer en direct dans le navigateur avec un éditeur de code intégré et un tuteur IA 24/7. Elle fait partie du parcours d'apprentissage Apache Kafka & Stream Processing Fundamentals, et ta progression se synchronise sur le web et l'application CoddyKit. Le cours Apache Kafka & Stream Processing Fundamentals comprend 4 leçons au total.

Certaines parties de cette leçon n'ont pas encore été traduites et s'affichent en anglais.

What is High Throughput?

In Kafka, high throughput means your system can efficiently process a massive volume of data per second or minute. It's crucial for applications like real-time analytics, IoT data ingestion, and log aggregation where data arrives continuously at high rates.

Designing for high throughput ensures your Kafka cluster and applications can handle peak loads without performance degradation, data loss, or significant delays.

Key Factors for Throughput

Achieving high throughput in Kafka involves optimizing several interconnected components. Think of it as a chain – the weakest link limits the overall speed.

Producers: How efficiently they send data.
Brokers: How quickly they store and serve data.
Consumers: How fast they read and process data.
Infrastructure: Network bandwidth and disk I/O speed.

Producer Optimization: Batching

Sending messages one by one is inefficient. Producers can batch messages, sending multiple records in a single request. This reduces network overhead and improves throughput.

batch.size: The maximum size in bytes of a single batch.
linger.ms: The maximum time a producer waits before sending a batch, even if it's not full.

Adjusting these balances latency (how quickly a message is sent) and throughput (how many messages are sent over time).

Batching Producer Example

Here's how to configure a producer for batching. Notice the batch.size and linger.ms settings.

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;

public class HighThroughputProducer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        // High throughput settings
        props.put("batch.size", 65536); // 64 KB batch size
        props.put("linger.ms", 10);    // Wait up to 10 ms for more records
        props.put("compression.type", "snappy"); // Compress batches
        props.put("acks", "1");         // Acks=1 for good balance

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            for (int i = 0; i < 1000; i++) {
                String message = "Hello Kafka Throughput - " + i;
                producer.send(new ProducerRecord<>("throughput-topic", "key-" + i, message));
            }
            System.out.println("1000 messages sent to throughput-topic.");
        }
    }
}

Producer Compression & ACKs

Beyond batching, two other producer settings greatly influence throughput:

compression.type: Compressing data (e.g., gzip, snappy, lz4) reduces network bandwidth usage and disk space. This allows more data to be sent and stored, boosting effective throughput.
acks: Controls the durability guarantee. acks=0 (fire-and-forget) offers the highest throughput but lowest durability. acks=1 (leader acknowledges) is a good balance. acks=all (all in-sync replicas acknowledge) provides the highest durability but lowest throughput.

Broker Scaling: Partitions & Disks

Kafka brokers are the backbone. Their throughput depends heavily on:

Partitions: Each topic partition is an ordered log. More partitions allow for greater parallelism in both writing and reading data across brokers and consumers. Distribute partitions evenly across brokers.
Disk I/O: Kafka is disk-intensive. Using fast SSDs and configuring RAID 0 or RAID 10 for data directories significantly improves write and read speeds, which is critical for high throughput.

Broker Network & CPU

Don't overlook the underlying hardware for your brokers:

Network: High-speed network interfaces (e.g., 10 Gigabit Ethernet) and sufficient bandwidth are paramount. Brokers constantly move data between themselves (replication) and with clients.
CPU: While Kafka is optimized for sequential disk I/O, CPU can become a bottleneck, especially with heavy data compression/decompression, SSL encryption, or complex ACLs. Ensure adequate CPU cores.

Consumer Optimization: Batch Fetching

Consumers also benefit from batching. Instead of polling for one record at a time, consumers fetch a batch of records.

max.poll.records: The maximum number of records returned in a single call to poll(). Processing more records per poll reduces the overhead of repeated calls.
fetch.min.bytes: The minimum amount of data (in bytes) that the consumer will wait to receive from a broker before returning.
fetch.max.wait.ms: The maximum amount of time (in ms) the consumer will wait for fetch.min.bytes to be satisfied.

Batching Consumer Example

A consumer configured for batch fetching will retrieve more records per poll() call, improving processing efficiency.

import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class HighThroughputConsumer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("group.id", "throughput-group");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        // High throughput settings
        props.put("max.poll.records", 500); // Fetch up to 500 records at once
        props.put("fetch.min.bytes", 1048576); // Wait for 1MB of data
        props.put("fetch.max.wait.ms", 500); // Or wait 500ms

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(Collections.singletonList("throughput-topic"));

            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                if (!records.isEmpty()) {
                    System.out.println("Received " + records.count() + " records.");
                    // Process records (e.g., in a thread pool for parallelism)
                    records.forEach(record -> {
                        // System.out.println("Processing record: " + record.value());
                    });
                    consumer.commitSync();
                }
            }
        }
    }
}

Consumer Parallel Processing

While max.poll.records helps with fetching, the actual processing of messages can be a bottleneck. To maximize consumer throughput:

Internal Thread Pool: Implement a thread pool within your consumer application. When poll() returns a batch of records, submit these records to the thread pool for parallel processing.
Careful Offset Management: If processing in parallel, ensure you commit offsets only after all messages in a batch (or a specific subset) have been successfully processed. Otherwise, you risk data loss or reprocessing.

Monitoring Throughput Metrics

To identify throughput bottlenecks, continuous monitoring is essential. Key metrics to watch:

Producer: Request rate, byte rate, request latency.
Consumer: Fetch rate, byte rate, consumer lag (most critical for identifying processing bottlenecks).
Broker: Disk I/O (read/write), network I/O, CPU utilization, memory usage.
Network: Bandwidth utilization, packet loss.

Tools like JMX, Prometheus/Grafana, or Confluent Control Center can help visualize these metrics.

Throughput Optimization Challenge

You've learned about various strategies to boost Kafka system throughput. Which of the following actions would generally help increase the overall throughput of a Kafka-based data pipeline?

Designing for Throughput Recap

Congratulations! You've explored key strategies for designing Kafka-based systems to handle massive data volumes with high throughput.

Optimize producers with batching, compression, and appropriate `acks` settings.
Scale brokers by using sufficient partitions, fast disks, and ample network/CPU resources.
Tune consumers with batch fetching and implement internal parallelism for processing.
Monitor critical metrics to identify and address bottlenecks proactively.

Mastering these techniques is vital for building robust and performant real-time data platforms.

Questions Fréquemment Posées

La leçon « Concevoir pour un débit élevé » est-elle gratuite ?

Oui — le texte complet de « Concevoir pour un débit élevé » est gratuit à lire ici sur le web. Pour la pratiquer de manière interactive (un éditeur de code intégré et un tuteur IA 24/7) et déverrouiller le reste du cours Apache Kafka & Stream Processing Fundamentals, passe à CoddyKit PRO. Le cours Apache Kafka & Stream Processing Fundamentals comprend 4 leçons au total.

Qu'est-ce que j'apprendrai dans « Concevoir pour un débit élevé » ?

Apprenez les considérations architecturales et les bonnes pratiques pour créer des systèmes fondés sur Kafka capables de gérer d’importants volumes de données. Tu pratiques Apache Kafka & Stream Processing Fundamentals avec du code pratique que tu exécutes directement dans le navigateur, et un tuteur IA 24/7 répond à tes questions au fur et à mesure que tu avances dans la leçon.

Dois-je avoir de l'expérience pour commencer Apache Kafka & Stream Processing Fundamentals ?

Aucune expérience préalable n'est requise. Apache Kafka & Stream Processing Fundamentals sur CoddyKit est structuré pour les débutants jusqu'aux apprenants avancés, donc tu peux commencer ici ou depuis le début et avancer à ton rythme. Ceci est la leçon 1 sur 4.

Combien de temps prend la leçon « Concevoir pour un débit élevé » ?

La plupart des leçons CoddyKit prennent environ 5–10 minutes. Chacune est courte et interactive, tu progresses régulièrement et tu repiques exactement où tu t'es arrêté sur le web et l'app.

Peux-tu écrire et exécuter du code dans cette leçon Apache Kafka & Stream Processing Fundamentals ?

Oui. Chaque leçon Apache Kafka & Stream Processing Fundamentals inclut un éditeur de code intégré, tu écris et exécutes du vrai code directement dans ton navigateur et tu reçois des retours IA instantanés — aucune configuration locale requise.

Toutes les leçons de ce cours

Concevoir pour un débit élevé
Reprise après sinistre et réplication géographique
Tendances futures du traitement par flux
Contrôle de la pression et régulation des flux à grande échelle

← Retour à Apache Kafka & Stream Processing Fundamentals