What mechanisms does Kafka provide for data retention?

Official answer

Kafka offers two main cleanup policies, configured per topic with broker defaults:

• cleanup.policy=delete removes eligible old log segments using time (retention.ms) or size (retention.bytes) limits. The byte limit applies per partition. A size limit can shorten the effective time available for replay.
• cleanup.policy=compact removes superseded records for the same key within a partition. Cleanup is asynchronous, and tombstones represent deletions. This supports rebuilding current keyed state; it does not retain all transaction history.
• cleanup.policy=compact,delete applies both. Even the last retained value for a key can disappear when its segment expires under deletion rules.

For example, a four-partition topic with a 1 GiB per-partition retention limit has a rough 4 GiB logical retention budget, before replicas, indexes, segment granularity, temporary cleaning space and other overhead. This is not an exact disk-capacity ceiling or a guaranteed time window.

A consumer commit does not cause topic records to be deleted. Conversely, retention can remove unread data while a consumer is offline. Size the replay window from peak ingestion, expected outage duration and catch-up capacity, then observe the oldest available offsets and timestamps.

Use separate policies for a full event history and a current-state changelog. Monitor bytes, segment/cleaner progress and consumer recovery margin. Compaction cannot provide transaction-history completeness, and user-engagement metrics are not a substitute for measuring actual ingestion and retained storage.

Sources

Kafka topic cleanup and retention settings; Kafka compaction semantics.