How can you increase the throughput of a Kafka producer?

Official answer

Start with a representative benchmark and identify whether the limit is serialization, CPU, network, broker I/O, partition skew or acknowledgement latency. Measure successful records/s together with tail latency, errors and resource use.

Useful producer changes include:

• Batch records for the same partition. Increase batch.size only where traffic can fill useful batches; many mostly empty partition buffers can waste memory.
• Tune linger.ms to allow batching within the latency budget. It is not a strict bound on total delivery latency, because queues, throttling, retries and broker response time also matter.
• Benchmark supported compression codecs and levels using actual payloads. Fewer bytes can improve a network-limited workload while extra compression CPU can hurt a CPU-limited one.
• Send asynchronously and handle completion errors. Avoid flushing or waiting on every record if the application can safely keep bounded work in flight. A KafkaProducer is thread-safe and can be shared rather than recreated for every message.
• Size buffer.memory and max.block.ms deliberately. A larger buffer absorbs a burst; it cannot sustain an indefinitely higher input rate than the cluster accepts.
• Use enough balanced partitions for useful parallelism, while accounting for key skew, batching efficiency and ordering requirements.

Keep the required acknowledgements and compatible idempotence settings. Disabling durability checks may improve a benchmark while changing the reliability contract. With idempotence disabled, retries and multiple in-flight requests can reorder batches.

Retest under peak load and broker failure. Select the smallest configuration change that improves the measured bottleneck without violating latency, memory or durability requirements.

Producer settings, producer lifecycle and concurrency