Explain the role of ISR (In-Sync Replicas) in Kafka's replication mechanism.
Instruction: Discuss the concept of ISR in Kafka, how it affects data consistency and availability, and how it is managed during broker failures.
Updated
Official answer
The in-sync replica set, or ISR, tracks replicas that meet Kafka's liveness and catch-up requirements for a partition. It includes the leader. Followers can temporarily lag behind the log end while remaining in sync; ISR membership does not mean every replica is byte-for-byte at the newest appended offset at every instant.
The leader handles writes and followers replicate its log. Depending on configuration, consumers can fetch from a suitable follower, so not every read must go through the leader. A failed or persistently lagging replica can leave the ISR; it must recover and catch up before rejoining.
Producer acks=all and min.insync.replicas work together. For example, with replication factor 3 and a minimum ISR of 2, writes require the configured replication condition rather than succeeding on a lone replica. This protects durability at the cost of write availability when replicas are unavailable. Acknowledgement and consumer offset commits are different concepts.
Leader election rules depend on version/features. The conventional clean-election path chooses an in-sync replica. Kafka's Eligible Leader Replicas feature can also track certain out-of-ISR replicas as safe election candidates; it is enabled by default on new 4.1 clusters. That is distinct from enabling unclean election of an unsafe replica, which can lose data. Do not describe every out-of-ISR election as necessarily unsafe or claim every failure is lossless.
Monitor ISR shrink/expand events, under-replicated and under-minimum-ISR partitions, replication lag and broker failures. Investigate disk/network pressure and replica placement before weakening durability settings to make alerts disappear.
Sources
Kafka replication design; Eligible Leader Replicas; topic minimum ISR.