Implementing custom metrics for Kafka monitoring.
Instruction: Explain how you would extend Kafka's monitoring capabilities with custom metrics and what metrics you would consider important.
Updated
Official answer
Start with a metric contract: name, unit, labels, collection point and interpretation. Reuse existing broker/client metrics before inventing an overlapping measurement. Kafka's built-in client lag metrics and an application's completed-work progress are different signals.
For a partition, if E is the chosen end position and C is the group's committed next offset, E minus C is an offset-distance backlog. If E is 105 and C is 100, the distance is 5. It is not necessarily five user records: compaction, transactions and offset gaps matter. A consumer's current position can be ahead of successfully completed work, and read_committed visibility uses the last stable offset. Record which end and progress definitions are used. Missing commits, resets or inconsistent samples need explicit states rather than fabricated zero lag.
Add application counters for successes, retries, rejected/deserialization failures and duplicate suppression; histograms for processing/deserialization duration; gauges for in-flight work and oldest pending-event age. Use a monotonic clock for local durations and account for clock skew in cross-host event age. Observe errors as well as successful timings.
Expose application metrics through the application's metrics system or registered JMX instruments. Built-in broker JMX cannot automatically observe a business transaction in a consumer. Use a secured exporter/collector, bounded labels and explicit sampling; never label by event ID or include sensitive payloads. Define actionable alerts on sustained symptoms and validate that instrumentation does not dominate processing cost.
Kafka monitoring semantics, consumer position and committed offsets