How can you explain Kafka's use of the term 'offset'?

Instruction: Define the term 'offset' in the context of Apache Kafka and its importance.

Context: This question tests the candidate's understanding of offsets in Kafka, which are crucial for tracking the position of consumers within a partition and ensuring that messages are processed in order.

Updated

Official answer

An offset identifies a position in one Kafka partition. It is not a topic-wide sequence number: partition 0 offset 10 and partition 1 offset 10 are different positions.

Appended offsets increase, but the records visible to a consumer need not have consecutive offsets. Compaction can remove records, and transaction/control records affect what a consumer sees. An offset gap alone does not prove data loss.

Distinguish three concepts: a record's offset; the consumer's current position, meaning where it will fetch next; and its committed offset, meaning the saved resume position for its group. A commit should describe the next position after the work being acknowledged, not simply repeat the last processed record's offset. With delivered offsets 10 and 12 completed and the next delivered record at 15 still unfinished, commit 15 rather than advancing past that unfinished record.

The log's start/end and a read_committed consumer's visibility limit are separate boundaries. Required data may disappear under retention, and an open transaction can constrain transactional visibility. Seeking or resetting offsets must therefore be an explicit recovery decision.

Offsets do not prove an external database update finished, do not remove records, and are not elapsed time. Use them with an application completion contract and the topic's replay horizon.

Consumer offsets, commits and isolation, cleanup policy

Related Questions