Explain how Kafka's Exactly Once Semantics (EOS) works and discuss its impact on system performance.

Official answer

Kafka's exactly-once mechanisms support a specific result: a committed Kafka read-process-write transaction can make its output records and input consumer-group progress visible atomically. The application may execute code again after a failure; arbitrary external effects are not automatically included.

Producer idempotence uses producer identity and per-partition sequence numbers to deduplicate supported retries. A transactional producer also uses a stable transactional.id for its logical producer instance. Each concurrently active logical instance needs its own identity; reusing an identity for a replacement lets Kafka fence an older instance. The transaction coordinator tracks transaction state and completion across the participating partitions.

A consumer-transform-producer workflow begins a transaction, writes outputs, sends the next input offsets with the consumer group's metadata, and commits. On failure it follows the API's abort/retry/fatal-error contract. Downstream consumers use read_committed to exclude aborted transactional records. Kafka Streams can manage the corresponding outputs, state updates and progress with its exactly-once processing setting.

A database write, payment or email outside that transaction still needs an idempotency key, outbox or another explicit coordination protocol. Application resubmission is not automatically the same as a producer retry.

Measure transaction commit latency, throughput, abort/retry rate and resource use with equivalent durability settings. Larger transactions can amortize coordination but delay visibility and increase recovery work; an open transaction can hold back read_committed progress on its partition. There is no universal percentage overhead or promise that enabling EOS alone meets a business latency target.

Transactional producer API, Streams processing settings, consumer isolation