How to manage and optimize Kafka's resources in a cloud-native environment?

Official answer

Size Kafka from measured ingress/egress, retention, replication, partition skew and recovery targets. Track broker disk capacity and I/O, network, CPU, request latency, replica health and consumer lag. A CPU-only autoscaling rule can miss disk exhaustion or a single hot partition.

Treat brokers as stateful services. Use stable identities and durable storage, spread replicas across appropriate failure domains, and preserve controller quorum. Adding a broker does not automatically move every existing partition onto it; plan replica reassignment and throttle recovery traffic so it does not overwhelm normal work. Drain and reassign data before removing capacity.

Scale consumer applications separately from brokers. Ordinary consumer groups cannot use more active consumers for one topic than its partitions, and a slow downstream service may remain the bottleneck. Adding partitions can change key routing; Kafka does not support reducing a topic's partition count in place.

For Kubernetes, use a Kafka-aware operator and deliberate storage, disruption and placement policies rather than treating broker replicas as interchangeable stateless pods. For managed services, verify the exact supported scaling operation: Amazon MSK's storage autoscaling expands storage and does not automatically shrink it or imply that broker count scales with consumer lag.

Optimize compression, retention, replica placement and utilization without silently weakening required durability. Include inter-zone/region traffic, storage I/O, retained data and recovery headroom in the cost model. Interruption-prone capacity requires an explicit failure analysis and recovery test.

Validate changes under equivalent traffic, broker loss and catch-up load. Report measured cost per successful workload together with latency and recovery results, rather than an assumed percentage saving.

Partition and replica operations, KRaft deployment, MSK storage autoscaling