Discuss the use of Kafka in edge computing scenarios.

Instruction: Explain how Kafka can be employed in edge computing architectures, including the challenges and benefits.

Context: This question is designed to assess the candidate's understanding of modern computing paradigms and their ability to apply Kafka in edge computing scenarios, highlighting its role in handling large volumes of distributed data.

Updated

Official answer

Start by deciding whether the edge site needs a Kafka cluster or only a durable gateway that forwards data to a regional cluster. A small sensor is not automatically a suitable Kafka broker. Budget CPU, memory, disks, power, operations and a viable failure model before deploying a local cluster.

A site cluster can decouple local ingestion and processing from WAN availability. Local consumers can react while the link is down, and selected events can later be mirrored to a regional cluster. This is asynchronous forwarding, not a guarantee that disconnected regions remain mutually consistent. Safety-critical control also needs its own timing and failure analysis.

Size an outage buffer from retained byte rate and outage duration, plus replication, indexes and headroom. At 2 MiB/s for six hours, the unreplicated payload alone is about 42.2 GiB. After reconnection, catch-up requires usable forwarding capacity greater than ongoing arrival rate; otherwise the backlog never drains.

Preserve event IDs and source timestamps, define duplicate/late-event handling, and specify what happens when storage fills. Consider clock drift, intermittent credentials, remote upgrade/rollback and physical access. Filter or aggregate only when the application permits losing that detail, and apply data-location policy before replication.

Measure local latency, disk headroom, oldest unforwarded event age and recovery time. There is no standard Apache Kafka “lightweight edition” that removes these constraints.

Hardware and filesystem considerations, cross-cluster replication

Related Questions