Explain how Kafka's log compaction feature works and its use cases.

Official answer

Log compaction removes superseded records for the same key within a partition, preserving enough keyed state for reconstruction. It runs asynchronously: several versions of a key can remain visible until eligible segments are cleaned. It does not turn the log into an immediately deduplicated table.

Set the topic's cleanup.policy=compact; log.cleanup.policy is the broker default property. Cleaner eligibility is influenced by the dirty ratio and minimum/maximum compaction lag settings. min.cleanable.dirty.ratio concerns the proportion of dirty, not-yet-cleaned log data, not a proportion of records that can never be compacted. Eligibility does not promise an exact completion time under arbitrary cleaner load.

A non-null key with a null value is a tombstone. It removes prior state for that key, and the tombstone can itself be cleaned after its retention period. Consumers rebuilding state must handle tombstones and keep within the documented delete.retention.ms scan bound. Compaction preserves the ordering and offsets of retained records but leaves offset gaps.

For example, keyed values A=1, B=4, A=2 can eventually be reduced to B=4, A=2; a consumer may still encounter all three before cleaning. A later tombstone for A means A is absent after replay. This is useful for current configuration, change-data-capture state and eligible changelog restoration. It does not preserve every historical update for an audit trail.

compact,delete combines compaction with time/size deletion, so old segments can disappear even when they contain the last value for a key. Match the policy to the rebuild requirement and monitor cleaner progress, disk headroom and consumer recovery time.

Sources

Kafka log compaction design; topic cleanup and compaction settings.