How can you ensure data retention in Kafka meets both technical and regulatory requirements?
Instruction: Discuss the strategies and configurations in Kafka for managing data retention to comply with both system performance and legal compliance requirements.
Updated
Official answer
Translate the applicable obligations into a documented data lifecycle with the responsible privacy/legal and business owners. Do not assume a named privacy law supplies one universal retention period. Classify the data, purpose, required minimum/maximum retention, deletion requests and any applicable holds before choosing Kafka settings.
Configure topic-level cleanup.policy, retention.ms and retention.bytes to support that policy. Size retention is per partition and can remove data before the time window; neither setting is an exact per-record erasure deadline. Segment eligibility, cleanup scheduling and storage copies matter. Compaction removes superseded keyed values asynchronously and is not a complete-history archive or immediate deletion mechanism.
Trace all copies: replicated partitions, mirrored clusters, remote/tiered storage if enabled, snapshots/backups, consumer state, sink systems, exports and logs. A tombstone in one topic does not prove erasure everywhere. Minimize sensitive fields and separate data with incompatible lifecycle requirements. Design a verified deletion workflow and durable evidence of its outcome without logging the sensitive payload unnecessarily.
Keep enough replay history for the agreed outage/recovery window, and check that the size cap and peak ingestion rate can support it. If a separate archive is required, give it its own access, retention and deletion controls; indefinite copying is not a default compliance solution.
Test full replay, expiry, deletion propagation and restoration from backups under the agreed policy. Monitor oldest available offsets/timestamps, consumer recovery margins, cleanup progress and storage growth. Configuration inspection and disk-use metrics alone do not establish compliance; reconcile observed behavior with the approved obligations and record exceptions for resolution.
Sources
Kafka topic retention controls; ICO: storage limitation and documented retention.