How would you design Kafka disaster recovery across geographically separate data centers?
Instruction: Distinguish local partition replication from cross-cluster mirroring. Explain topology, recovery objectives, offset translation, writer fencing and tested failover.
Updated
Official answer
For geographically distant sites, start with independent regional Kafka clusters and an explicit cross-cluster replication flow, rather than assuming a single stretched cluster is the right design. Within each region, distribute brokers, partition replicas and controllers across suitable local failure domains.
A replication factor of three does not by itself place copies in three data centers. Intra-cluster replication and MirrorMaker 2 between independent clusters are different mechanisms, with different latency and failure boundaries.
A representative active/passive design mirrors selected primary topics to a standby. Configure topic naming, group checkpoint/offset translation, authentication, authorization and required schema/configuration dependencies. Bidirectional replication is needed only for a deliberate multi-writer design; it adds loop, conflict and ownership considerations.
Mirroring is asynchronous. Define recovery point and recovery time objectives, measure replication age and checkpoint freshness, and budget standby capacity. A primary can fail before its newest records reach the standby, so zero loss must not be assumed.
The failover runbook should fence the old writer, establish the surviving data boundary, select translated resume positions, redirect clients and reconcile repeats or missing work. Test failback and avoid simultaneous authoritative writers. Back up or separately manage dependencies that record mirroring does not preserve.
Rehearse regional isolation and recovery under realistic traffic. Report observed recovery results and remaining gaps rather than promising a seamless transition from architecture alone.