Troubleshooting Kafka consumer lag issues.

Official answer

Treat lag as a backlog investigation, with correctness and retention risk visible alongside speed. Record the affected groups/partitions, when growth began, ingress rate, current/committed positions and downstream completion age. An offset gap is not a duration, and a fetched record may still be unprocessed.

Check whether the backlog is growing, stable or draining. For a roughly steady workload, a backlog of 100,000 records with ingress 1,000 records/s and completed processing 1,250 records/s takes about 400 seconds to drain. This estimate assumes comparable records, constant rates and no new failures; if processing is no faster than ingress, adding time alone will not catch up.

Use the shape of the problem to choose the next check:

• One partition lags: inspect key skew, unusually expensive records and its assigned consumer/downstream shard.
• Most partitions lag: inspect shared database/API capacity, consumer resources, broker throttling and a production-rate change.
• Progress repeatedly stops: inspect errors, poison-record retry loops, rebalances, poll intervals and open transactions affecting readable data.
• Fetch lag is low but business results are late: inspect worker queues, commits and sink completion.

Scale within partition and downstream limits, fix the measured processing bottleneck, or tune batching with a defined latency budget. Broker replica reassignment and consumer-group assignment are different operations; neither automatically corrects a hot key.

Do not reset offsets to the end merely to make lag disappear: that can skip required work. If retention has already removed needed records, use an explicit recovery/replay source and reconcile the missing interval. Verify correctness, sustained drain rate and recovery margin after the change.

Sources

Kafka lag metric definitions; KafkaConsumer positions and processing.