Discuss the role of Kafka's Quotas in cluster resource management.
Instruction: Explain what Quotas are in Kafka, and how they can be used to manage cluster resources and maintain a balanced load among clients.
Updated
Official answer
Kafka quotas limit resource use by a configured group of clients. They help control noisy neighbors, but do not reserve dedicated hardware or guarantee a latency target.
Producer and consumer bandwidth quotas are byte-rate limits. Request quotas measure the percentage of time spent on broker network and request-handler threads; they are not simply a number of requests per second. A few expensive requests can consume more of this budget than many cheap requests.
Quotas can be configured for authenticated users, client IDs, or user/client-ID combinations, with documented override precedence. A client ID is supplied by the client, so it is not an authentication boundary. Use principal-based policy when tenants must not evade limits by changing client IDs. Supported quota overrides can be changed dynamically without restarting every broker.
Enforcement is per broker. For example, a 10 MiB/s quota on each of three brokers is not a single 10 MiB/s cluster-wide cap: aggregate throughput depends on which brokers the client uses. Brokers throttle excess traffic, increasing delay; clients must handle the resulting backpressure and timeouts.
Monitor throttle time, byte rates, request utilization, errors and each tenant's latency/lag. Manage topic creation, partition counts, retention and disk capacity separately. Increase a quota only when spare capacity and the service objectives justify it.