Replication & Fault Tolerance

Imagine you have important data. If it's stored in only one place, and that place fails, your data is gone!

In distributed systems like Kafka, ensuring data isn't lost and is always available is crucial. This is where replication comes in.

Replication in Kafka means making multiple copies of your data (messages) and storing them on different Kafka brokers (servers).

This simple idea provides two massive benefits:

• Fault Tolerance: If one broker fails, copies exist elsewhere.
• High Availability: Data remains accessible even during outages.

The replication factor determines how many copies of each partition Kafka will maintain across your cluster.

• A replication factor of 1 means only one copy (no fault tolerance).
• A factor of 3 (e.g., 3 copies) is common for production.

Each copy is called a replica.

For each partition, one replica is designated as the leader. The other replicas are followers.

• The leader handles all read and write requests for that partition.
• Followers passively replicate data from the leader to stay synchronized.

This design simplifies client interactions, as they only communicate with the leader.

Kafka keeps track of which followers are fully caught up with the leader. This group is called the In-Sync Replica (ISR) set.

The ISR set includes the leader and all followers that have replicated all messages from the leader and are not too far behind.

This is key for guaranteeing data durability.

Producers can configure how many acknowledgements (acks) they wait for before considering a message 'successfully sent'. This impacts durability and performance.

• acks=0: No acknowledgements. Fastest, but data loss possible.
• acks=1: Leader acknowledges receipt. Better, but data loss if leader fails before followers sync.
• acks=all (or -1): All ISRs must acknowledge. Slowest, but strongest durability guarantee.

Followers continuously send fetch requests to the leader to get new messages. They then write these messages to their own local log.

This constant syncing ensures that if the leader fails, a follower from the ISR can quickly become the new leader without data loss.

What happens if the broker hosting a partition's leader fails?

1. The leader becomes unavailable.
2. Kafka automatically initiates a leader election.
3. One of the other brokers in the ISR set is chosen as the new leader.

This process is fast and transparent to clients, ensuring high availability.

If a broker hosting a follower replica fails, it's less critical:

• The follower is temporarily removed from the ISR set.
• The leader continues to serve requests.
• Once the failed broker recovers, it will catch up with the leader and rejoin the ISR.

The system remains fully operational throughout.

You set the replication factor when you create a topic. Here's an example using Kafka's command-line tools:

kafka-topics.sh --create --topic my_replicated_topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 3

This creates 'my_replicated_topic' with 3 partitions, each having 3 copies across brokers.

Consider a Kafka topic partition with a replication factor of 3. Currently, the leader and one follower are in sync, but another follower is temporarily down. What is the state of the In-Sync Replica (ISR) set?

We've covered the critical role of replication in Kafka for data safety and high availability.

• Replication Factor: Number of data copies.
• Leaders & Followers: Roles for handling requests and syncing data.
• In-Sync Replicas (ISR): The crucial set of fully synchronized replicas.
• Acknowledgements (acks): Producer setting to control durability.
• Fault Tolerance: How Kafka handles broker failures gracefully.

Understanding these concepts is key to building robust Kafka systems!