How can consumer lag be monitored in Kafka?
Instruction: Describe methods for monitoring consumer lag within a Kafka application.
Updated
Official answer
Monitor several progress boundaries. Group committed lag compares a saved resume offset with an applicable log-end boundary; client fetch lag uses the consumer's current position. Neither alone proves that downstream work completed.
For an ad-hoc group view, use kafka-consumer-groups.sh with the broker bootstrap endpoint, group and --describe options plus a secured command configuration. Inspect each partition, assignments and missing checkpoints; an unknown offset is not zero lag. Use a regularly collected exporter or supported monitoring system for history and alerts.
Kafka's consumer-fetch-manager metrics expose records-lag-max at client scope, and partition-labelled lag metrics also exist. These use current rather than committed offsets. Aggregate across relevant clients deliberately instead of calling one client's maximum a whole-group completion metric.
If the end position is 120 and committed position is 100, the offset distance is 20. That is not twenty seconds and may not equal twenty visible business records. Measure age of the oldest unfinished event, downstream completion checkpoints, throughput and error rates alongside offset distance. For read_committed workloads, distinguish transaction visibility limits from the raw log end.
Alert on sustained failure to meet a freshness/recovery objective and on approaching the retention boundary. Include stale/missing telemetry and partition skew. Confluent Control Center is a Confluent product, not Apache Kafka's built-in monitoring tool; select tooling according to deployment needs and verify what its lag metric means.
Kafka monitoring metric scopes, group administration, consumer semantics