Explain how you would implement end-to-end encryption in a Kafka messaging system, including key management and rotation strategies.
Instruction: Distinguish client-side end-to-end encryption from Kafka TLS and storage encryption. Explain key access, rotation, replay and failure handling.
Updated
Official answer
End-to-end encryption means the producing application encrypts sensitive data and only authorized consuming applications can decrypt it. TLS protects network connections, but a broker terminates those connections and can otherwise see plaintext records. Disk encryption addresses a different threat, such as stolen storage.
Use a maintained authenticated-encryption library in the producer and consumer. A representative envelope contains ciphertext, an authentication tag, the required nonce and a versioned encrypted data key. Protect the data key with a key-encryption key managed by KMS or an HSM. Follow the algorithm's nonce requirements; authenticate relevant context, such as the schema version and tenant identifier, as associated data. Do not invent a Kafka broker message formatter to perform end-to-end encryption.
Give producers only the required encryption/key-generation permissions and consumers only the required decryption permissions. Brokers should not receive application decryption keys. Keep TLS with certificate/hostname verification and Kafka authorization in place. Mutual TLS requires client-certificate authentication to be configured; ordinary TLS does not automatically authenticate both parties.
For rotation, write new records with the new key version while allowing authorized consumers to read old versions. Retain old decryption capability for required topic retention, replays, mirrors and backups; revoke or destroy it only under an explicit recovery and deletion policy. Rotating a wrapping key and re-encrypting historical payloads are different operations.
Test old/new versions, denied key access, altered ciphertext/context, replay and key-service outages before rollout. Encrypting values still exposes unencrypted keys, headers, topic names and traffic patterns. It also limits broker-side inspection and downstream processing until authorized decryption. Availability and compliance require separate evidence.