Explain how to manage a global Kafka cluster across multiple regions.

Instruction: Discuss considerations and strategies for managing Kafka in a geographically distributed setup.

Context: This question is aimed at assessing the candidate's knowledge of Kafka's capabilities and limitations in a global context, including replication, latency, and data sovereignty.

Updated

Official answer

Distinguish a single cluster stretched across regions from multiple regional clusters. A stretched cluster exposes replication and metadata quorum decisions to WAN latency and partitions. It requires an explicit supported failure/latency design; a larger replication factor alone does not solve regional availability.

A common starting point is independent regional clusters with local clients and selected asynchronous replication between them. MirrorMaker 2 can copy topics and maintain checkpoint information, but this is not synchronous partition replication or a global transaction. Measure replication delay and define recovery-point and recovery-time objectives.

Choose authoritative writers for each entity or define conflict resolution. Bidirectional copying does not resolve simultaneous conflicting updates. Before promoting a destination, establish the recoverable data/offset point, fence obsolete writers, route clients, verify credentials and schemas, and document replay/duplicate handling. Test failback as well as failover.

Keep regional replication factors, rack placement and KRaft controller quorums resilient within the intended failure domains. Ensure the recovery region can handle the promoted workload and backlog catch-up.

Classify which datasets may cross each boundary and enforce selection and access policy across source, destination, backups and derived stores. Topic names or geo-replication do not prove compliance with data-location requirements. Validate policies with the responsible organization and retain operational evidence.

Datacenter deployment guidance, cross-cluster mirroring

Related Questions