Explain the concept of backpressure in Kafka and how it can be managed.
Instruction: Discuss what backpressure is, how it can occur in Kafka, and the techniques to manage it effectively.
Updated
Official answer
Backpressure is the response to a downstream stage that cannot keep up. Kafka buffers retained records, so a slow consumer normally creates lag rather than immediately slowing the producer. That buffer is bounded by storage and retention; it does not eliminate overload.
Measure arrival and completion rates, oldest unprocessed event age, lag by partition, processing latency, queue depth, broker disk and throttling. For an illustrative steady workload of 1,200 incoming records/s and 1,000 completed records/s, backlog grows by 200 records/s, or 720,000 in an hour. Actual bytes and retention risk depend on record sizes and topic policy.
Use bounded worker queues. If processing is delegated, the owning consumer thread can pause assigned partitions and continue polling to maintain membership, then resume when capacity returns. KafkaConsumer is not generally thread-safe. Disable automatic commits in this design and commit the next offset after the contiguous completed prefix of delivered records; never skip unfinished work just because a later worker finished. Offsets need not be consecutive. Handle revocation so stale workers cannot advance the new owner's progress.
Scale consumers only where partition parallelism and downstream capacity permit. Apply intentional producer/application admission control or broker quotas when necessary. batch.size and linger.ms influence batching, not a guaranteed records-per-second limit. Increasing producer buffers merely postpones exhaustion if the sustained imbalance remains.
Test overload, downstream outage and recovery. Alert before retention overtakes the consumer, and define an explicit rejection, durable spill or recovery policy for excess work.