Design a multi-tenant Kafka architecture.
Instruction: Describe how you would architect a Kafka cluster to support multiple tenants, ensuring isolation, security, and fair resource allocation.
Updated
Official answer
Separate the tenant's data namespace, identity and resource policy. Dedicated topics and consumer-group names are useful organization, but naming alone does not enforce isolation.
Authenticate each tenant/service, use TLS and grant explicit least-privilege ACLs for allowed topics, groups and transactional IDs. Restrict topic creation and administration, and test that one tenant cannot read, publish, join another group or impersonate another principal. A client ID is not a trusted identity. Shared topics do not provide row-level tenant filtering through Kafka ACLs.
Apply principal or user/client quotas for producer/fetch bytes and request-thread utilization. Request quotas are not simply requests per second, and quotas apply per broker. Bound topic/partition growth and retention separately; byte-rate throttles do not prevent a tenant from accumulating excessive retained storage.
Use rack-aware replica placement to survive failure domains. This does not assign exclusive machines or resources to each tenant, and shared brokers/controllers still create shared operational risk. Tenants requiring stronger administrative, performance or failure isolation may need dedicated clusters and independent infrastructure.
Automate provisioning, schema/retention policy, credential rotation and offboarding. Measure per-tenant errors, throttle time, storage, lag and service objectives with bounded metric-label cardinality. Test noisy-neighbor load, denied operations and failure/recovery before claiming isolation.
Related Questions
-
medium
-
medium
-
hard
-
hard