How would you handle rebalancing in Kafka to minimize consumer downtime?

Official answer

First identify what triggers the churn: rolling deployments, crashes, missed polling bounds, subscription changes or expanding partitions. Lag alone does not automatically trigger a rebalance, although the underlying slow processing may cause a timeout.

Choose a supported protocol/assignor deliberately. Cooperative classic assignment reduces unnecessary revocation, and the newer consumer protocol uses incremental coordinator assignment. These can reduce disruption, but transferred partitions still need safe handoff and possibly state restoration.

Bound processing time and queues, keep consumer API calls on the owner thread, and poll within the timing contract. If workers run separately, disable automatic commits and track the completed prefix per partition. Stop dispatching revoked work and prevent stale workers from committing or causing uncontrolled effects after ownership changes.

During graceful shutdown, coordinate outstanding work while ownership is still valid, commit only completed progress when permitted, and close the consumer within a bounded deadline. Do not leave the group first and assume it is safe to finish arbitrary writes afterward. Lost ownership can already mean a replacement is processing the partition.

A stable unique group.instance.id can reduce reassignment during brief restarts, but duplicate active identities are invalid and static membership can delay failover. Heartbeat/session configuration differs between classic and consumer protocols; tune the deployed mode, and test long processing, crashes and rolling restarts.

Measure moved partitions, rebalance/restoration duration, errors and completion age before and after the change. Aim for the required recovery behavior, not a promise that rebalancing or downtime disappears.

Rebalance protocols, ownership callbacks, consumer threading and shutdown