What are the benefits of log compaction in Kafka?

Official answer

Compaction can reduce retained data and replay work for a topic representing current keyed state. When a key is updated repeatedly, older values can be removed while the retained log still supports reconstructing its latest value, subject to deletion/tombstone and retention rules.

The benefit depends on the workload. Replacing a large number of updates to a bounded key set can save substantial space; a stream of unique keys offers much less opportunity. Measure retained bytes, cleaner work and rebuild duration rather than promising a fixed reduction.

A rebuilding consumer still replays a log and applies later values and tombstones. Compaction is asynchronous, so multiple versions can remain. It does not add a point-lookup API, make reads instantaneous or create an atomic snapshot across partitions.

It also removes historical information. For example, retaining only a customer's current address cannot answer what the address was at every earlier order. Preserve a separately governed event history or historical store when auditing or time-travel queries are required. A compacted topic alone cannot provide them.

Replication and acknowledgements provide Kafka's configured durability behavior; compaction is not an additional guarantee against all data loss. Cleaning consumes CPU, I/O and temporary disk space. Choose it for replaceable keyed state, ensure consumers handle tombstones and recovery windows, and test a full state rebuild under realistic lag.

Sources

Kafka compaction design and guarantees; cleanup policy configuration.