Optimizing Kafka for low-latency applications.

Official answer

Define the latency target and measurement boundary first: producer acknowledgement, Kafka fetch, or completed business processing. Measure p50/p95/p99 and error rates under representative steady load, bursts and failures. Fix the dominant stage rather than applying a universal low-latency configuration.

Producer batching trades waiting against network/CPU efficiency. Lower linger.ms can reduce intentional batching delay, but smaller batches may increase contention and worsen tail latency under load. Measure batch size, queue time, compression cost and request latency together. A batching delay is not the only reason a record waits.

On consumers, tune fetch.min.bytes and fetch.max.wait.ms according to throughput and latency needs. Keep processing queues bounded and profile slow dependencies; frequent fetches do not help when application processing is the bottleneck. Transaction commit cadence and read_committed visibility can also affect observed delay.

Inspect broker network/request queues, CPU, disk and replication metrics before increasing thread counts. More threads can add contention and do not guarantee faster I/O. Keep clients near the appropriate cluster while retaining the required zone/rack separation for resilience. Rack-aware replica placement protects failure domains; it does not by itself eliminate WAN traffic.

Preserve the agreed replication, minimum ISR and acknowledgement contract. acks=all concerns in-sync replicas, not every configured replica regardless of state. Reducing durability to obtain a lower number changes the service guarantee and must be evaluated as a different design.

Compare controlled runs after each change, including throughput, resource use, loss/duplicate behavior and recovery. Avoid claiming a latency improvement without measurements from the actual workload.

Sources

Kafka producer batching and acknowledgements; consumer fetch settings; broker/client monitoring.