Design a multi-tenant Kafka architecture.

Instruction: Describe how you would architect a Kafka cluster to support multiple tenants, ensuring isolation, security, and fair resource allocation.

Context: This question evaluates the candidate's ability to design complex Kafka deployments that can securely and efficiently support multiple distinct users or applications within the same infrastructure.

Updated

Official answer

Separate the tenant's data namespace, identity and resource policy. Dedicated topics and consumer-group names are useful organization, but naming alone does not enforce isolation.

Authenticate each tenant/service, use TLS and grant explicit least-privilege ACLs for allowed topics, groups and transactional IDs. Restrict topic creation and administration, and test that one tenant cannot read, publish, join another group or impersonate another principal. A client ID is not a trusted identity. Shared topics do not provide row-level tenant filtering through Kafka ACLs.

Apply principal or user/client quotas for producer/fetch bytes and request-thread utilization. Request quotas are not simply requests per second, and quotas apply per broker. Bound topic/partition growth and retention separately; byte-rate throttles do not prevent a tenant from accumulating excessive retained storage.

Use rack-aware replica placement to survive failure domains. This does not assign exclusive machines or resources to each tenant, and shared brokers/controllers still create shared operational risk. Tenants requiring stronger administrative, performance or failure isolation may need dedicated clusters and independent infrastructure.

Automate provisioning, schema/retention policy, credential rotation and offboarding. Measure per-tenant errors, throttle time, storage, lag and service objectives with bounded metric-label cardinality. Test noisy-neighbor load, denied operations and failure/recovery before claiming isolation.

Multi-tenancy, ACLs, quotas

Related Questions