Disaster Recovery & Geo-Replication

In the world of real-time data, ensuring continuous operation is paramount. What happens if an entire datacenter hosting your Kafka cluster goes offline?

• Disaster Recovery (DR): A plan to recover from major outages and resume critical business functions.
• Geo-Replication: Replicating data across geographically distant locations to protect against regional disasters.
• The goal is to minimize data loss (Recovery Point Objective - RPO) and downtime (Recovery Time Objective - RTO).

You've learned that Kafka topics are replicated across multiple brokers within a single Kafka cluster. This built-in replication protects against individual broker failures.

However, if an entire datacenter experiences a catastrophic event (e.g., power outage, network failure, natural disaster), this internal replication won't protect your data or services. All replicas in that datacenter would be lost.

To truly achieve disaster recovery, you need to replicate data beyond the boundaries of a single Kafka cluster. This means having a separate, operational Kafka cluster in a different physical location or region.

This setup allows your applications to seamlessly switch over to the secondary cluster if the primary one becomes unavailable, ensuring continuous data flow and service availability even during major outages.

Kafka MirrorMaker 2 (MM2) is the industry-standard tool for replicating data between distinct Kafka clusters. It's built on Kafka Connect, providing a robust and fault-tolerant replication framework.

MM2 can replicate not just topics, but also consumer groups, ACLs (Access Control Lists), and configurations from a source cluster to a target cluster, making it ideal for geo-replication scenarios.

MM2 operates as a Kafka Connect cluster itself. It leverages Kafka Connect's framework to manage replication tasks:

• Source Connector: Reads messages from topics in the source Kafka cluster.
• Sink Connector: Writes those messages to topics in the target Kafka cluster.
• MM2 also replicates internal topics (like consumer offsets, heartbeats, and checkpoints) to ensure proper state transfer during failover.

MM2 is configured using properties files, much like a regular Kafka Connect worker. You define the source and target Kafka clusters and specify which topics should be replicated.

You can use regular expressions to match topic names, allowing for flexible replication of many topics with minimal configuration. This simplifies management of evolving topic sets.

Here's a simplified example of a mirror-maker.properties file. This defines two clusters and enables replication from 'primary' to 'secondary' for specific topics:

This is a widely adopted disaster recovery pattern:

• Primary (Active) Cluster: Handles all application reads and writes under normal operation.
• Secondary (Passive) Cluster: Continuously replicates data from the primary and is on standby, not actively serving client requests.

If the primary fails, applications are reconfigured (e.g., via DNS changes) to point to the secondary. This pattern is simpler to manage but requires a clear failover process.

In an active-active setup, both Kafka clusters can simultaneously handle reads and writes. This can be for different applications or even the same applications if designed carefully.

This pattern offers higher availability and potentially lower latency for geographically dispersed users. However, it introduces significant complexity, especially in handling data conflicts and ensuring strict consistency across clusters.

Executing a failover (switching operations to the secondary cluster) and subsequent failback (returning to the primary) requires careful planning and automation:

• DNS Updates: Directing producers and consumers to the new active cluster.
• Consumer Group Management: Ensuring consumers resume processing from correct offsets on the new cluster.
• Data Sync: During failback, ensuring any data written to the secondary during the outage is synced back to the primary before switching.

You've learned about Kafka MirrorMaker 2's role in geo-replication. Consider its architecture and capabilities:

This lesson covered the critical importance of disaster recovery and geo-replication for Kafka:

• Intra-cluster replication protects against broker failure, but not datacenter failure.
• Kafka MirrorMaker 2 (MM2) is the primary tool for robust cross-cluster replication.
• MM2 runs on Kafka Connect and can replicate topics, consumer groups, and more.
• Common DR patterns include Active-Passive (simpler failover) and Active-Active (higher availability, more complex).
• Successful failover and failback require meticulous planning and automation.