What is the significance of the 'offset' in Kafka?
Instruction: Explain what an offset is in the context of Kafka and why it is important.
Updated
Official answer
An offset identifies a position within one partition. It allows a consumer to seek, checkpoint progress and replay available records; it does not by itself prove successful business processing.
The current consumer position is where it will fetch next. A committed group offset is the saved resume position used after restart or reassignment. If record 7 is the last completed record and the next relevant position is 8, commit 8 and resume at 8—not at a further position obtained by adding one to the committed value.
Offsets visible to an application need not be consecutive because of compaction or transaction/control records. For asynchronous processing, advance only past the completed prefix of delivered work; a later completed record must not cause earlier unfinished work to be skipped.
Committing before an effect can skip it after a crash; committing after an effect can replay it. Use idempotent effects, deduplication or an appropriate atomic transaction when required. The offset checkpoint alone does not eliminate loss and duplication.
Log order is partition-local and is different from event-time order or worker completion order. Retention can make a saved offset unavailable, so define reset/recovery policy and monitor the replay horizon. Track committed progress and actual completion age separately from consumer position and offset-distance metrics.
Consumer position, commits and replay, consumer reset/isolation policy