How would you optimize Kafka's storage for SSDs?

Official answer

Use SSDs to address a measured storage bottleneck, not as a reason to raise every concurrency setting. Kafka appends partition logs sequentially, but many partitions, compaction, recovery and cache misses create a mixed workload. Device and filesystem behavior matter.

Measure disk latency and utilization, throughput, queue depth, free space, page-cache pressure, compaction backlog and broker request latency. Check device endurance and power-loss behavior. Sequential writes do not prove low physical write amplification on every SSD.

Segment size and roll time affect file counts, retention/compaction granularity and recovery. A segment roll is not the same as an fsync, so increasing segment size is not a universal reduction in durable-write operations. Benchmark the workload and retain sufficient space for cleaning and replica movement.

For compacted topics, delete.retention.ms controls how long tombstones remain available to readers. Reducing it to “save SSD deletes” can cause a slow rebuilding consumer to miss deletions and retain stale keys. Set it from the supported scan/recovery contract, not device wear alone.

The broker setting num.recovery.threads.per.data.dir controls startup recovery and shutdown flushing concurrency. More threads can help only until CPU or storage saturates; verify startup time and impact on other work. Preserve the chosen acknowledgement/replication durability policy during experiments. Monitor endurance, perform recovery tests and keep a measured rollback baseline.

Hardware and OS, topic retention and segments, broker recovery threads