How does Kafka's consumer lag monitoring work, and how can it be used to detect issues in real-time data processing pipelines?

Official answer

Define which positions a lag metric compares. A common consumer-group view uses the partition's log-end offset minus the group's committed next offset. For example, end offset 120 and committed offset 100 gives an offset gap of 20. It does not mean the last processed record has offset 100.

A consumer's current fetch position, committed position and the application's completed-work position can differ. Kafka's client records-lag-max metric is based on the current offset, not the committed offset. A consumer can fetch quickly while an internal worker queue or downstream database is far behind. Offset gaps also are not always an exact count of useful pending records because compaction, transactions and filtered records affect the log.

Collect per-partition and per-group lag through the appropriate admin/group tooling and client metrics. Track the maximum and distribution, not just a total that hides one hot partition. A missing committed offset or missing metric is an unknown state to investigate, not proof of zero lag.

Combine lag with arrival/processing rates, record age, processing/commit latency, rebalances, retry/dead-letter counts and downstream completion. For read_committed consumers, open transactions can also constrain what is currently readable; label the endpoint used by the metric.

Alert on sustained growth, stalled progress, unacceptable event age and approaching retention limits. Calibrate thresholds to the workload and recovery objective. A temporary backlog during a burst may be expected; zero fetch lag does not establish correct processing or low end-to-end latency.

Sources

Kafka consumer monitoring metric definitions; KafkaConsumer position and committed offset.